{"id":645631,"date":"2013-03-07T08:00:37","date_gmt":"2013-03-07T13:00:37","guid":{"rendered":"http:\/\/gigaom.com\/?p=616972"},"modified":"2013-03-07T08:00:37","modified_gmt":"2013-03-07T13:00:37","slug":"5-reasons-why-the-future-of-hadoop-is-real-time-relatively-speaking","status":"publish","type":"post","link":"https:\/\/mereja.media\/index\/645631","title":{"rendered":"5 reasons why the future of Hadoop is real-time (relatively speaking)"},"content":{"rendered":"

In some ways, Hadoop is a like a fine wine: It gets better with age as rough edges (or flavor profiles) are smoothed out, and those who wait to consume it will probably have a better experience. The only problem with this is that Hadoop exists in a world that\u2019s more about MD 20\/20<\/a> than it is about Relentless Napa Valley 2008<\/a>: Companies often want to drink their big data fast, get drunk on insights, and then have some more \u2014 maybe something even stronger. And with data \u2014 unlike technology and tannins \u2014 it turns out older isn\u2019t always better.<\/p>\n

That\u2019s a crude analogy, of course, but it gets at the essence of what\u2019s currently plaguing Hadoop adoption and what will propel it forward in the next couple years. The work being done by companies like Cloudera and Hortonworks at the distribution level is great and important, as is MapReduce as a processing framework for certain types of batch workloads. But not every company can afford to be concerned about managing Hadoop on a day-to-day basis. And not every analytic job pairs well with MapReduce<\/a>.<\/p>\n

In Part I of our four-part series on Hadoop, we looked at how the technology was born<\/a> and grew into the juggernaut it is today. In Part II, we laid out the map of the current products and projects<\/a> that comprise the Hadoop ecosystem. In this installment, we\u2019ll take a closer look at some of them and how they\u2019re positioning themselves to be important players down the road.<\/p>\n

If there\u2019s one big Hadoop theme at our Structure: Data conference<\/a> March 20-21 in New York, it\u2019s the new realization that people shouldn\u2019t be asking \u201cWhat\u2019s next after Hadoop?\u201d but rather \u201cWhat will Hadoop become next?\u201d. Based on what\u2019s transpiring today, the answer to that question is that Hadoop will become faster in all regards and more useful as a result.<\/p>\n

Interactivity, big-data-style<\/h2>\n
\"Source:<\/p>\n

Source: Shutterstock user hauhu.<\/p>\n<\/div>\n

As I explained with some detail a couple weeks ago, SQL is what\u2019s next for Hadoop<\/a>, and that\u2019s not because of familiarity alone or the types of queries permitted by SQL <\/del>on relational data<\/del>. It\u2019s also because the types of massively parallel processing engines developed to analyze relational data over the years are very fast. That means analysts can ask questions and get answers at speeds much closer to the speed of their intuitions than is possible when querying entire data sets using standard MapReduce.<\/p>\n

But just as SQL and its processing techniques bring something to Hadoop, Hadoop (the Hadoop Distributed File System, specifically) brings something to the table, too. Namely, it brings scale and flexibility that don\u2019t exist in the traditional data warehouse world, where new hardware and licenses can be expensive; so only the \u201cvaluable\u201d data makes its way inside and only after it has been fitted to a pre-defined structure. Hadoop, on the other hand, provides virtually unlimited scale and schema-free storage, so companies can store however much information they want in whatever format they want and worry later about what they\u2019ll actually use it for. (Actually, though, most Hadoop jobs do require some sort of structure in order to run, and Hadoop co-creator Mike Cafarella is working on a project called RecordBreaker<\/a> that aims to automate this process for certain data types.)<\/p>\n

How hot is SQL-on-Hadoop space? I profiled the companies and projects working on it on Feb. 21, and since then EMC Greenplum announced a completely rewritten Hadoop distribution<\/a> that fuses its analytic database to Hadoop, and an entirely new player called JethroData<\/a> emerged along with $4.5 million in funding. Even if there\u2019s a major shakeout, there will be a few lucky companies left standing to capitalize on a shift to Hadoop as\u00a0the<\/em> center of data gravity that EMC Greenplum\u2019s Scott Yara (albeit a biased source) thinks will be the data equivalent of the mainframe\u2019s demise.<\/p>\n

This is your database. This is your database on HDFS<\/h2>\n

The SQL versus NoSQL debate appears to be dying down as companies and developers begin to realize there\u2019s definitely a place for both in most environments, but a new debate \u2014 with Hadoop at the center \u2014 might be about to start up. At its core is the concept of data gravity<\/a> and the large, attractive (in a gravitational sense) entity that is HDFS. Here\u2019s the underlying question that might be posed: If I\u2019m already storing my unstructured data in HDFS and am expected to replace my data warehouse with it, too, why would I also run a handful of other databases that require a separate data store?<\/p>\n

This is in part why HBase<\/a> has attracted such a strong following despite its relative technical and commercial immaturity compared with comparable NoSQL database Cassandra<\/a>. For applications that would benefit from a relational database, startups such as Drawn to Scale<\/a> and Splice Machine<\/a> have turned HBase into a transactional SQL system. Wibidata, the new startup from Cloudera C0-founder Christophe Bisciglia and Aaron Kimball<\/a>, is pushing an open source framework called Kiji<\/a> to make it easier to develop applications that use HBase.<\/p>\n

\u201cIf you talk to anyone from Cloudera or any of the platform vendors, I think they will tell you that a large percentage of their customers use HBase,\u201d Bisciglia said. \u201cIt\u2019s something that I only expect to see increasing.\u201d<\/p>\n

MapR seems to think so, too: the Hadoop-distribution vendor is getting ahead of the game by selling an enterprise-grade version of HBase called M7<\/a>.\u00a0Should hot startups such as TempoDB<\/a> and Ayasdi<\/a> decide to take their HBase-reliant cloud services into the data center, they\u2019ll tap into Hadoop clusters, too.<\/p>\n

And the National Security Agency built Apache Accumulo<\/a>, a key-value database similar to HBase but designed for fine-grained security and massive scale. It\u2019s now being sold commercially by a startup called Sqrrl<\/a>. There\u2019s even a graph-processing project called Giraph<\/a> that relies on HBase or Accumulo as the database layer.<\/p>\n

Whatever \u201creal-time\u201d means to you<\/h2>\n

Real-time is one of those terms that means different things to different people and different applications. The interactivity that SQL-on-Hadoop technologies promise is one definition, as is the type of stream processing enabled by technologies like Storm<\/a>. When it comes to the latter, there\u2019s a lot of excitement around YARN as the innovation will make it happen.<\/p>\n

YARN, aka MapReduce 2.0<\/a>, is a resource scheduler and distributed application framework that allows Hadoop users to run processing paradigms other than MapReduce. This could mean things, from traditional parallel-processing methods such as MPI to graph processing to newly developed stream-processing engines such as Storm and S4<\/a>. Considering for how many years Hadoop <\/em>meant HDFS and MapReduce<\/em>, this type flexibility is certainly a big deal.<\/p>\n

\"figure1\"Stream processing, of course, is the antithesis of batch processing, for which Hadoop is known, and which is inherently too slow for workloads such as serving real-time ads or monitoring sensor data. And even if Storm and other stream-processing platforms somehow don\u2019t make their way onto Hadoop clusters, a startup called HStreaming has made it its mission<\/a> to deliver stream processing to Hadoop, and it\u2019s on other companies\u2019 radars, as well<\/a>.<\/p>\n

For what it\u2019s worth, though, VertiCloud<\/a> Founder and CEO and former Yahoo CTO Raymie Stata thinks we should do away with terms such as\u00a0<\/em>batch, real-time and interactive altogether. Instead, he prefers the terms synchronous and asynchronous to describe the human experience with the data rather than the speed of processing it. Synchronous computing happens at the speed of human activity, generally speaking, while asynchronous computing is largely decoupled from the idea of someone sitting in front of a computer screen awaiting a result.<\/p>\n

The change in terms is associated with a change in how you manage SLAs for applications. Uploading photos to Flickr: synchronous. Running a MapReduce job: most likely asynchronous. Ironically, according to Stata, stream processing data with Storm is often asynchronous, too. That\u2019s because there\u2019s probably not someone on the other end waiting for a page to update or a query to return. And unless you\u2019re doing something where guaranteed real-time latency is necessary<\/em>, the occasional difference between milliseconds and 1 second probably isn\u2019t critical.<\/p>\n