Hadoop – Not The Answer
Hadoop et al – not the answer, but they help to ask the questions
Few in the industry haven’t heard about Hadoop, that data management platform that lends itself to distributed environments. Rather than storing data in one big bucket, Hadoop enables data to be managed, stored and queried across multiple little buckets – that is, exactly the kind of architecture used by cloud service and application providers.
To say that Hadoop, together with its programming framework MapReduce, have taken the IT world by storm would be an understatement. While the underlying platform is open source, several major vendors including IBM, EMC and Microsoft have embraced and extended the platform as part of their own offerings.
As distributed data management becomes mainstream, it is taking a number of overlapping paths. In the first, the Hadoop platform is improved, optimised, built upon. We are seeing the emergence of proprietary alternatives such as Greenplum’s Pivotal HD, announced a few weeks ago, and an update to Google’s BigQuery service this month (LINK: https://developers.google.com/bigquery/).
Whatever the relative merits of proprietary vs open source (LINK: ) or pay-as-you-go, service-based solutions, the drive is towards making the data easier to access using, in general, SQL-like commands such as SELECT and JOIN. Perhaps Dr Codd was right after all (http://www.stanford.edu/dept/itss/docs/oracle/10g/server.101/b10759/intro001.htm).
More importantly, a simpler data access mechanism implies lowering the hurdles to finding big things out of the big data pile. We are faced with mountains of information, much of which is growing faster than we know what to do with. Part of the challenge is human – the effort required to construct queries exceeds the amount of time available.
This will not be the end of the story however. Waiting in the wings are algorithms developed over decades, that are anxiously waiting to be unleashed on all that data. Autonomy’s famously Bayesian (LINK: http://www.director.co.uk/MAGAZINE/2011/5_May/mike-lynch_64_09.html) approaches were first conceived in the eighteenth century, and other models and mechanisms have existed for decades.
What we have lacked has been sufficient power to conduct large-scale analysis at reasonable cost – note that tasks that took 2 months and millions of pounds to run ten years ago, now take about 2 weeks and about 20 K’s worth of processing. You don’t need to be a computer scientist to extrapolate that one.
The ability to write simpler queries is a start, but it still puts the effort onto the analyst who needs to ask the right questions. What happens when Google, EMC, the open source community et al start integrating mathematically proven algorithmics, and adding simple access and query mechanisms to boot? “Here’s a bunch of data, find me all the anomalies and let me know why you think they might exist,” could be one such request. Or, for the more advanced class, “Where should we target our efforts?”
This is neither a criticism nor a request, simply an acknowledgement of what the current round of developments in the distributed data space will enable. It’s a no-brainer to suggest that some organisations (vendor and end-user) will be better positioned than others to take advantage of the inevitable. Right now however, the playing field is as level as it will ever be.