The Data Lake is Real and Open for Business

The market and folks on blog sites are talking about Big Data, internet of things, and how these factors are redefining business and IT. Sometimes it can feel like an academic discussion or a discussion amongst futurists. While there is a significant amount of hype, let me tell you about a real world example of Big Data in action and how this example could be a precursor to widespread adoption of data lakes.

I meet with customers regularly and I have been asked the same question from many of them, “How do I store, analyze and reason over large amounts of data?” One customer that asked me this question had a particularly thorny problem. They needed to analyze and reason over a vast amount of data to respond to increasing litigation requests and regulatory pressure. They had been trying for a few years to solve this daunting problem and none of the solutions could be delivered in a straightforward way.

Enter the data lake. Everyone has their own definition, but I think the one that is most meaningful is the data lake as a central repository where all data (internal and external) is loaded “as is” and is available for analytics. Through combined efforts of EMC Engineering and the EMC Managed Services team we took on the challenge of building a data lake to solve this problem. The data lake we built enables the customer to store, analyze and reason with over 1 billion records. In fact the solution is so efficient records that would have taken weeks or months to search can now be searched and presented for compliance within 4 hours.

This is only one of many big data use cases that I am seeing my customers ask about. In fact I believe the changing demands of business will make data lakes pervasive in the redefined IT environment. And that is a great thing! Data lakes hold the promise of enabling organizations to solve their most difficult and valuable problems.