{"id":648979,"date":"2013-03-26T22:48:04","date_gmt":"2013-03-27T02:48:04","guid":{"rendered":"http:\/\/gigaom.com\/?p=623537"},"modified":"2013-03-26T22:48:04","modified_gmt":"2013-03-27T02:48:04","slug":"top-techies-tout-their-top-tools-for-webscale-computing","status":"publish","type":"post","link":"https:\/\/mereja.media\/index\/648979","title":{"rendered":"Top techies tout their top tools for webscale computing"},"content":{"rendered":"

Developers are always on the lookout for new, better, faster, cooler tools, languages, compilers. And the popularity \u00a0of these toolsets ebbs and flows. One week Scala<\/a>\u00a0is at the top, the next it’s Go<\/a>\u00a0language.<\/p>\n

\"Ashok<\/a><\/p>\n

(L to R): Ashok Srivastava, Venture Advisor for Trident Capital and Chief Data Scientist, Verizon; Silvius Rus, Director, Big Data Platforms, Quantcast; Todd Papaioannou, Founder and CEO, Continuuity; Bhaskar Ghosh, Senior Director of Engineering, Data Infrastructure, LinkedIn; Michael Driscoll. CEO, Metamarkets Structure Data 2013 Albert Chau itsmebert.com<\/p>\n<\/div>\n

Last week it was Sawzall<\/a><\/strong>‘s\u00a0time to shine. The language, named after the popular saw <\/a>that cuts through anything (and I mean anything), comes\u00a0out of Google.<\/p>\n

Silvius Rus, director of big data platforms for Quantcast<\/a>, gave Sawzall a shout-out during a Structure Data Guru panel<\/a>\u00a0last week.\u00a0”It’s a lightweight language developed by Google that ridges procedural and interpretive languages,” Rus said.<\/p>\n

Michael Driscoll, CEO of Metamarkets<\/a> and moderator of the panel, later explained why that’s important. With a declarative language, the programmer tells the computer what to do in almost English-language-like sentences.\u00a0To tell the computer to draw a circle, a declarative or imperative programmer might say “draw.circle with a size attached,” Driscoll said.<\/p>\n

Procedural languages, on the other hand, are much more detailed step-by-step instructions — they sound more like math.\u00a0A procedural approach would “define the actual pointer and tell it to move one degree to the left and one degree up and the square root of 2 up to the diagonal and repeat X times,” Driscoll said.<\/p>\n

Sawzall is a nice blend between a declarative language that might be too high level to do all of what the programmer really wants and procedural, “which is way too in the weeds” to be fully productive, Driscoll said. More broadly, Sawzall is a powerful and compact language for log data aggregation and transformation. And, he added, it plays well with Hadoop MapReduce.<\/p>\n

New toolsets for webscale computing<\/h2>\n

Another tool ranking high on the hit list was YARN<\/strong> (or Yet Another Resource Manager)<\/a>\u00a0aka MapReduce 2.0, cited by Todd Papaioannou, founder and CEO of Continuuity<\/a> is a fan.<\/p>\n

Yarn was built to “just think about mass-produced jobs.” Continuity is building a real-time streaming engine called Big Flow and using Yarn for all the resource deployment and management.<\/p>\n

He also gave kudos to Weave<\/strong>, a higher-level framework. Weave “allows you to build a much wider class of applications on top of Yarn. So,t Yarn is \u00a0… something that we will be going forward with for at least the next half a decade [and] Weave allows you to actually build more wide scale applications on top of that.”<\/p>\n

Bhaskar Ghosh, \u00a0senior director of engineering at LinkedIn, touted Helix<\/a><\/strong>, a generic distribution cluster manager developed at LinkedIn and which is now an Apache incubator project. \u00a0Helix simplifies distributed system\u00a0development by separating cluster management from the primary component tasks of a distributed system, according to LinkedIn.<\/p>\n

Kafka, Storm slake the thirst for real-time frameworks<\/h2>\n

Driscoll also sees traction for Kafka<\/a><\/strong>, a real-time framework for ingesting and managing data streams and Storm<\/strong>,<\/a> out of Twitter<\/a>, for processing those streams. “Think of Kafka and Storm as the HDFS and MapReduce analogs but for real time — Kafka for storage and Storm for compute,” Driscoll said.<\/p>\n

On its blog,<\/a> LinkedIn describes Kafka as a distributed\u00a0publish-subscribe messaging system — also now an Apache project. Kafka is used by Twitter and Square for log aggregation, queeuing, and real-time monitoring and event processing.<\/p>\n

This list is by no means complete. When I spoke with Github co-founder Tom Preston-Werner<\/a>\u00a0a few weeks ago, he said\u00a0Clojure<\/a>, heretofore a rather obscure dynamic programming language, is gaining momentum. “It’s getting a lot of buzz round on the enterprise side,” Preston-Werner said.<\/p>\n

The continued popularity of the Java Virtual Machine has breathed new life into languages like Clojure and Scala, he added. Indeed, the JVM remains nearly ubiquitous and that is a huge advantage for languages that support it. If you’re a developer, you want the widest possible audience.<\/p>\n

“The JVM is still\u00a0the modern foundation that lets you\u00a0run everywhere and Clojure has benefited from that,” Driscoll agreed. “It’s certainly gained steam among an elite set of programmers in Silicon Valley.”<\/p>\n

Sawzall photo courtesy of<\/a>\u00a0Flickr user\u00a0Charles & Hudson<\/a><\/em>
<\/a><\/em><\/p>\n

\"\"<\/p>\n

<\/a><\/p>\n

Related research and analysis from GigaOM Pro:<\/strong>
Subscriber content. Sign up for a free trial<\/a>.<\/p>\n