Building a data platform costs money, and lots of it<\/h2>\n
But before the details, a little history. Greenplum\u2019s flagship product is an analytic database powered by a massively parallel processing (MPP) and query engine. The company had raised nearly $100 million in venture capital around this technology since launching in 2003, but doing business in the enterprise software world is hard and expensive, and Greenplum needed more money.<\/p>\n
![\"Rob](\"http:\/\/gigaom2.files.wordpress.com\/2012\/03\/1z5o1154.jpg?w=300&h=200\")
<\/p>\nYara (left) with Pivotal Labs CEO Rob Me and Om Malik at Structure: Data 2012 (c) 2012 Pinar Ozger. pinar@pinarozger.com<\/p>\n<\/div>\n
\u201cI thought it was going to take another couple hundred million dollars in investment for us to complete the technical vision we had and go to market,\u201d Yara explained. But finding that kind of money wasn\u2019t so easy in an investment environment where everyone was gaga over social apps like Facebook and Zynga. When EMC approached with a deal like it gave VMware in 2003 \u2014 essentially near complete independence bolstered by a huge R&D and marketing budget \u2014 Greenplum couldn\u2019t refuse.<\/p>\n
Yara said Greenplum had known for a while that Hadoop was the key to any big data strategy going forward, but that it would take some time to build up its own technology. So, in 2011, it entered into a reseller agreement with Hadoop startup MapR<\/a> to offer a premium product to appease enterprise customers while Greenplum\u2019s engineers got to work on what would become Pivotal HD. That deal with MapR is still in place, but it\u2019s no longer the focal point of Greenplum\u2019s Hadoop strategy.<\/p>\n The technology inside Pivotal HD is what companies should come to expect from a Hadoop distribution, Yara explained. It\u2019s essentially the Greenplum Database with its POSIX file system ripped out and replaced by the Hadoop Distributed File System. Whatever users can do on Greenplum\u2019s flagship database, they can do on Pivotal HD, only they can run Hadoop MapReduce jobs and house an HBase database, too.<\/p>\n And when SQL-like features become an important part of Hadoop because it\u2019s so broadly installed that users are now seeking out broader utility, \u201cthat\u2019s when the bar gets raised in terms of the amount of capability that\u2019s required,\u201d Yara said. He said Pivotal HD includes years worth of investment in Hadoop cluster-management technology and professional support, too, and that they will cost half as much as what Cloudera and Hortonworks charge. It\u2019s designed to run smoothly wherever customers want it to \u2014 physical servers, virtual servers or even cloud servers.<\/p>\n Greenplum, by contrast, rebuilt its entire R&D team to focus on bringing 10 years of database technology to Hadoop. \u201cWe literally have over 300 engineers working on our Hadoop platform,\u201d Yara said. \u201c\u2026 We\u2019re bringing all the power of EMC and VMware behind it.\u201d<\/p>\n Looking past his competitive boasting, though, it\u2019s easy to see Yara\u2019s greater point when you ask him what all this Hadoop talks means for the data warehouse business on which Greenplum was built. He points to the mainframe business that fell from its high perch decades ago but still drives billions a year in revenue. A single MPP database system is still faster on certain workloads than SQL on Hadoop, but that gap will close over time and\u00a0 \u201cI do think the center of gravity will move toward HDFS,\u201d he said.<\/p>\n Josh Klahr, a Pivotal HD product manager, noted the importance of being able to process all of a company\u2019s data right in a single scalable data store rather than operating numerous systems. He pointed to one customer that\u2019s storing a petabyte of data in Greenplum Database but wants to grow its data volume to 20 petabyes over the next few years and needs something like Hadoop to do that both financially and technically. He said Netflix\u2019s decision to store all its data in Amazon S3<\/a> and bring analytic services to it is a good indicator of where the market is headed.<\/p>\n A few years ago, Yara acknowledged, embracing Hadoop as the future might have been a scary proposition. However, he said, \u201cNow, if you don\u2019t embrace Hadooop as the new database platform, if you\u2019re a database vendor, that\u2019s a grave mistake.\u201d<\/p>\n Related research and analysis from GigaOM Pro:<\/strong>Big investment, big aspirations<\/h2>\n
![\"hawq\"](\"http:\/\/gigaom2.files.wordpress.com\/2013\/02\/hawq.jpg?w=708&h=386\")
<\/p>\n![\"Structure:Data:](\"http:\/\/gigaom2.files.wordpress.com\/2013\/02\/structure-data_in-article-banners_300x2001.png?w=708\")
<\/a>Because they\u2019re so new, he said, competitive SQL-on-Hadoop offerings such as Cloudera\u2019s Impala<\/a> can only handle about 20 percent of real-world workloads. Looking back at the capital investment in analytics and big data technologies past, things like Netezza, Teradata and Aster Data, Yara proffered, \u201cI don\u2019t think you could build [a full SQL-on-Hadoop] system for less than $25 to $50 million over five years.\u201d (Some of those new technologies, by the way, will have a chance to state their cases during a Structure: Data<\/a> panel on March 21 that\u2019s all about Hadoop as the next-generation business intelligence platform.)<\/p>\nThe data warehouse is the new mainframe<\/h2>\n
![\"\"](\"http:\/\/stats.wordpress.com\/b.gif?host=gigaom.com&blog=14960843&%23038;post=613686&%23038;subd=gigaom2&%23038;ref=&%23038;feed=1\")
<\/p>\n<\/a><\/p>\n
Subscriber content. Sign up for a free trial<\/a>.<\/p>\n
<\/p>\n![](\"http:\/\/res3.feedsportal.com\/images\/emailthis2.gif\")
<\/a><\/td>\n![](\"http:\/\/res3.feedsportal.com\/images\/bookmark.gif\")
<\/a><\/td>\n<\/tr>\n<\/table>\n<\/div>\n