With Hadoop and HDFS and related big data technologies, we\u2019ve pretty much licked the scale problem of handling petabyte upon petabyte of data. Next up: solving the speed problem.<\/p>\n
Right now running interactive queries across data sets spread among a thousand nodes is no mean feat. As a rule of thumb, you can run fast queries on old data (in a data warehouse) but running fast, interactive queries on massive distributed data sets is still the problem, according to speakers at a\u00a0Structure:Data 2013<\/a> session today that honed in on what problems real-time data analytics \u2014 when possible \u2014 could attack.<\/p>\n \u201cInteractive analytics is a complex problem. You have on one end a business users asking \u2018what if we did things a little different?\u2019\u201d said Silvius Rus, director of big data platforms for Quantcast. They may have 10 ideas on how to change something and 9 are bad, but one is good. They need to be to iterate queries and get the answer back in minutes not a day, he added.<\/p>\n On day one of the show, Paul Maritz,<\/a> the head of the new EMC-VMware Pivotal Initiative, talked about how companies need to have faster, more nimble feedback loops from their massive data stores. Telephone companies know they have dropped calls but they don\u2019t know who\u2019s call they dropped, and it can take days or even weeks to find out.<\/p>\n That\u2019s the sort of problem that this new world of fast, big data analytics can solve. In that world, the phone company \u201cat the very least, could text you an apology,\u201d he noted.<\/p>\n Panel moderator Michael Driscoll, CEO of Metamarkets, really wanted to hear about what this \u201cbig data utopia\u201d \u2014 where the system could ingest, transform and spit out answers to questions in real time \u2014 could mean. The\u00a0applications that could start coming out within months could be impressive, \u00a0according to Ashok Srivastava,\u00a0<\/a> chief data scientist for Verizon. There are the obvious things like real-time or near-real-time response to customer problems (see the dropped call issue above) and requests but he also foresees breakthroughs in cybersecurity. He cited earlier talks at Structure:Data about how systems can increasingly understand the motion and movement of people around the globe and the movements of concepts through society.<\/p>\n And he thinks real-time big data capabilities will play out in citizen science research. \u201cImagine taking your cell phone pictures and combining them with multiple millions of other cell phone pictures. That\u2019s something that can be used by scientists,\u201d he noted.<\/p>\n In health and medicine, the ability to query the most up-to-date personal health data along with historical data might enable real-time predictions about a person\u2019s health or the status of machine health, he added.<\/p>\n