{"id":643375,"date":"2013-02-21T13:29:31","date_gmt":"2013-02-21T18:29:31","guid":{"rendered":"http:\/\/gigaom.com\/?p=612289"},"modified":"2013-02-21T13:29:31","modified_gmt":"2013-02-21T18:29:31","slug":"sql-is-whats-next-for-hadoop-heres-whos-doing-it","status":"publish","type":"post","link":"https:\/\/mereja.media\/index\/643375","title":{"rendered":"SQL is what\u2019s next for Hadoop: Here\u2019s who\u2019s doing it"},"content":{"rendered":"

When we first began putting together the schedule for Structure: Data<\/a> several months ago, we knew that running SQL queries on Hadoop would be a big deal \u2014 we just didn\u2019t know how big a deal it would actually become. Fast-forward to today, a mere month away from the event (March 20-21 in New York), and the writing on the wall is a lot clearer. SQL support isn\u2019t the end-game for Hadoop, but it\u2019s the feature that will help Hadoop find its way into more places in more companies that understand the importance of next-generation analytics but don\u2019t want to (or can\u2019t yet) re-invent the wheel by becoming MapReduce experts.<\/p>\n

In fact, there are now so many products and projects pushing SQL queries and interactive data analysis on Hadoop \u2014 including two more announced this week \u2014 that it\u2019s getting hard to keep track. But I\u2019ll do my best.<\/p>\n

Of course, Facebook began this whole movement to bring SQL database-like functionality to Hadoop when it created Hive in 2009. Hive, now an Apache project<\/a>, includes a data-management layer and SQL-like query language called HiveQL. It has proven rather useful and popular over the years, but Hive\u2019s reliance on MapReduce makes it somewhat slow by nature \u2014 MapReduce scans the entire data set and moves a lot of data over the network while processing a job \u2014 and there hasn\u2019t been much effort to package it in a manner that might attract mainstream users.<\/p>\n

And keep in mind that this next generation of SQL-on-Hadoop tools aren\u2019t just business intelligence or database products that can access data stored in Hadoop; EMC Greenplum, HP Vertica, IBM Netezza, ParAccel, Microsoft SQL Server and Teradata\/Aster Data (which this week released some cool new features<\/a> for just this purpose) all allow some sort of access to Hadoop data. Rather, these are applications, frameworks and engines that let users query Hadoop data from inside Hadoop, sometimes by re-architecting the underlying compute and data infrastructures. The beauty of this approach is that data is usable in its existing form and, in theory, doesn\u2019t require two separate data stores for analytic applications.<\/p>\n

Data warehouses and BI: The Structure: Data set<\/h2>\n

\"Structure:Data:<\/a>I\u2019m highlighting this group of companies first, not because I think they\u2019re the best (although that might well be), but because I\u2019m truly excited about the panel they\u2019ll be featured on at our conference next month. The panel is moderated by Facebook engineering manager Ravi Murthy\u2013 a guy who knows his way around a database \u2014 so they\u2019ll have to answer some tough questions from one of the most-advanced and most-aggressive Hadoop and analytics tools users out there:<\/p>\n

Apache Drill<\/a>: <\/strong>Drill is a MapR-led effort to create a Google Dremel-like (or BigQuery-like) interactive query engine on top of Hadoop. First announced in August<\/a>, the project is still under development and in the incubator program within Apache. According to its web site, \u201cOne explicitly stated design goal is that Drill is able to scale to 10,000 servers or more and to be able to process petabyes of data and trillions of records in seconds.\u201d<\/p>\n

Hadapt<\/a>:<\/strong> Hadapt, which actually launched at Structure: Data in 2011<\/a>, was the first of the SQL on Hadoop vendors and is somewhat unique in that it has a real product on the market and real users in production. Its unique architecture includes tools for advanced SQL functions and a split-execution engine for MapReduce and relational tasks, and both HDFS and relational storage. In October, the company announced a tight integration with Tableau Software<\/a> around advanced visual analytics. <\/strong><\/p>\n

\"HAD_Graphic2-scaled\"<\/a><\/p>\n

\"platforaarch\"<\/a>Platfora<\/a>: <\/strong>Technically not a SQL product, Platfora is red-hot right now and is trying to re-imagine the world of business intelligence for a big data world. Essentially an HTML5 canvas laid atop Hadoop and an in-memory, massively parallel processing engine, the company\u2019s software, which it unveiled in October<\/a>, is designed to make analyzing data stored in Hadoop a fast and visually intuitive process.<\/p>\n

Qubole<\/a>:<\/strong> Qubole is an interesting case in that it\u2019s essentially a cloud-based version of the popular Apache Hive<\/a> framework launched by the guys who created Hive while working at Facebook<\/a>. Qubole claims it auto-scaling abilities, optimized Hadoop code and columnar data cache make its service run much faster than Hive alone \u2014 and running on Amazon Web Services makes it easier than maintaining a physical cluster. <\/strong><\/p>\n

\"cache\"<\/a><\/p>\n

Data warehouses and BI: The rest<\/h2>\n

Citus Data<\/a>:<\/strong> Citus Data\u2019s CitusDB isn\u2019t just about Hadoop, but rather wants to bring the power of its distributed Postgres implementation to all types of data<\/a>. It relies on Postgres\u2019s foreign data wrappers feature to convert disparate data types into the database\u2019s native format, and then on its own distributed-processing technology to carry out queries in seconds or less. Because of its Postgres foundation, CitusDB can join data from different data sources and retains all the native features that come with that database. <\/strong><\/p>\n

\"citus_hadoop_architecture\"<\/a><\/p>\n

Cloudera Impala<\/a>:\u00a0 <\/strong>Cloudera\u2019s Impala might just be the most-important SQL-on-Hadoop effort<\/a> around because of Cloudera\u2019s expansive installation and partner footprints. It\u2019s a massively parallel processing engine that bypasses MapReduce to enable interactive queries on data stored in either HDFS or HBase, using the same variant of SQL that Hive uses. However, because Cloudera doesn\u2019t build applications, it\u2019s relying on higher-level BI and analytics partners to provide the user interface.<\/p>\n

\"impala\"<\/a><\/p>\n

Karmasphere<\/a>: <\/strong>Karmasphere is one of the first startups to build an analytic application atop Hadoop, and in its 2.0 release last year<\/a> the company added support for SQL queries of data in HDFS. Like Hive, Karmasphere still relies on MapReduce to process queries, which means it\u2019s inherently slower than newer approaches. However, unlike Hive, Karmasphere allows for parallel queries to run at the same time and includes a visual interface for writing queries and filtering results.<\/p>\n

\"multiple-large\"<\/a><\/p>\n

Lingual<\/a>:<\/strong> Lingual is a new open source project from Concurrent\u00a0(see disclosure)<\/em>, the parent company of the Cascading framework for Hadoop. Announced on Wednesday<\/a>, Lingual runs on Cascading and gives developers and analysts a true ANSI SQL interface from which to run analytics or build applications. Lingual is compatible with traditional BI tools, JDBC\u00a0 and the Cascading family of APIs. <\/strong><\/p>\n

Phoenix<\/a>: <\/strong>Phoenix is a new and relatively unknown open source project that comes out of Salesforce.com and aims to allow fast SQL queries of data stored in HBase, the NoSQL database built atop HDFS. Its stated mission: \u201cBecome the standard means of accessing HBase data through a well-defined, industry standard API.\u201d Users interact with it through JDBC interfaces, and its developers claim its sub-second response times for small queries and seconds-long response for querying tens of millions of rows.<\/p>\n

\"A<\/a><\/p>\n

A sample of Phoenix via the SQuirreL client<\/p>\n<\/div>\n

\"shark\"<\/a>Shark<\/a>:<\/strong> Shark isn\u2019t technically Hadoop, but it\u2019s cut from the same cloth. Shark<\/em>, in this case, stands for \u201cHive on Spark,\u201d with Hive meaning the same thing it does to Hadoop, but with Spark being an in-memory platform<\/a> designed to run parallel-processing jobs 100 times faster than MapReduce (a speed improve over traditional Hive that Shark also claims). Shark also includes APIs for turning query results into a type of data format amenable to machine learning algorithms. Both Shark and Spark are developed by the University of California, Berkeley\u2019s AMPLab<\/a>.
<\/strong><\/p>\n

\"Screen-Shot-2013-02-19-at-5.37.01-PM-300x235\"<\/a>Stinger Initiative<\/a>: <\/strong>Launched on Wednesday (along with a security gateway called Knox<\/a> and a faster, simpler processing framework called Tez<\/a>), the Stinger Initiative is a Hortonworks-led effort to make Hive faster \u2014 up too 100x \u2014 and more functional. Stinger adds more SQL analytics capabilities to Hive, but the most-important aspects are infrastructural: an optimized execution engine, a columnar file format and the ability to avoid MapReduce bottlenecks by running atop Tez.<\/p>\n

Operational SQL<\/h2>\n

Drawn to Scale<\/a>:<\/strong> Drawn to Scale is a startup that has built an operational SQL database on top of HBase<\/a>. The key word here is database, as its product, called Spire, is modeled after Google\u2019s F1 designed to power transactional applications as analytic ones. Spire has a fully distributed index and queries are sent only to the node with the relevant data, so reads and writes are fast and the system can handle lots of concurrent users without falling down.<\/p>\n

\"SpireArchitecture.015\"<\/a><\/p>\n

\"splice\"<\/a>Splice Machine<\/a>: <\/strong>Database startup\u00a0Splice Machine is also trying to get into the operational space by building its Splice SQL Engine atop the naturally distributed HBase database. Splice Machine focuses its message on transactional integrity, which is really where it separates itself from scalable NoSQL databases and analytics-focused SQL-on-Hadoop efforts. It relies on HBase\u2019s aut0-sharding feature in order to making scaling an easy process.<\/p>\n

\"Structure:Data:<\/a><\/p>\n

Feature image courtesy of Shutterstock user hauhu<\/a>.<\/em><\/p>\n

\"\"<\/p>\n

<\/a><\/p>\n

Related research and analysis from GigaOM Pro:<\/strong>
Subscriber content. Sign up for a free trial<\/a>.<\/p>\n