{"id":646877,"date":"2013-03-14T18:31:12","date_gmt":"2013-03-14T22:31:12","guid":{"rendered":"http:\/\/gigaom.com\/?p=620545"},"modified":"2013-03-14T18:31:12","modified_gmt":"2013-03-14T22:31:12","slug":"google-bigquery-is-now-even-bigger","status":"publish","type":"post","link":"https:\/\/mereja.media\/index\/646877","title":{"rendered":"Google BigQuery is now even bigger"},"content":{"rendered":"

Google might be upsetting a lot of people<\/a> with some of its recent \u201cspring cleaning,\u201d but its latest batch of updates to BigQuery<\/a> should make data analysts happy, at least.<\/p>\n

With the latest updates \u2014 announced in a blog post<\/a> by BigQuery Product Manager Ju-kay Kwek on Thursday \u2014 users can now join large tables, import and query timestamped data, and aggregate large collections of distinct values. It\u2019s hardly the equivalent of Google launching Compute Engine last summer, but as (arguably) the inspiration for the SQL-on-Hadoop trend that\u2019s sweeping the big data world<\/a> right now, every improvement to BigQuery is notable.<\/p>\n

BigQuery is a cloud service that lets users analyze terabyte-sized data sets using SQL-like queries. It\u2019s based on Google\u2019s Dremel querying system<\/a>, which can analyze data where it\u2019s located (i.e., in the Google File System or BigTable) and which Google uses internally to analyze a variety of different data sets. Google claims queries in BigQuery run at interactive speeds<\/a>, which is something that MapReduce \u2014 the previous-generation tool for dealing with such large data sets \u2014 simply couldn\u2019t handle within a reasonable time frame or level of complexity. Of course, if you want to schedule batch jobs, BigQuery lets you do that, too, for a lower price<\/a>.<\/p>\n

This constraint \u2014 and therefore the potential benefits of something like Dremel and its commercial incarnation, BigQuery<\/a> \u2014 wasn\u2019t lost on the Hadoop community, which itself had been largely reliant on MapReduce processing for years. In the past year, we\u2019ve seen numerous startups and large vendors<\/a> pushing their own Dremel-like (or MPP-like) technologies for data sitting in the Hadoop Distributed File System. If you happen to be in New York next week, you can hear some of the pioneers in this space talk about it at our Structure: Data conference<\/a>.<\/p>\n

Background aside, the ability to join large data sets in BigQuery is probably the most-important of the three new functions. Joins are an essential aspect of data analysis in most environments because pieces of data that are relevant to each other don\u2019t always reside within the same table or even within the same cluster. And joining tables of the size BigQuery is designed for can take a long time without the right query engine in place.<\/p>\n

\"How<\/a><\/p>\n

How to do a join in BigQuery<\/p>\n<\/div>\n

Kwek offers an anecdote from Google that shows why joins, and the new aggregation function, are important:<\/p>\n

\n

[W]hen our App Engine team needed to reconcile app billing and usage information, Big JOIN allowed the team to merge 2TB of usage data with 10GB of configuration data in 60 seconds. Big Group Aggregations enabled them to immediately segment those results by customer.\u00a0Using the integrated Tableau client the team was able to quickly visualize and detect some unexpected trends.<\/p>\n<\/blockquote>\n

\"\"<\/p>\n

<\/a><\/p>\n

Related research and analysis from GigaOM Pro:<\/strong>
Subscriber content. Sign up for a free trial<\/a>.<\/p>\n