{"id":647183,"date":"2013-03-17T17:16:11","date_gmt":"2013-03-17T21:16:11","guid":{"rendered":"http:\/\/gigaom.com\/?p=621428"},"modified":"2013-03-17T17:16:11","modified_gmt":"2013-03-17T21:16:11","slug":"can-we-please-stop-saying-unstructured-data","status":"publish","type":"post","link":"https:\/\/mereja.media\/index\/647183","title":{"rendered":"Can we please stop saying \u201cunstructured\u201d data?"},"content":{"rendered":"

Text. English. Chinese. Multi-structured. Language. Fuzzy. Logs. Hard-to-parse. Rich. Semi-structured. Whatever you want to call data that doesn\u2019t fit neatly into tidy little rows and columns these days, can we please stop calling it \u201cunstructured\u201d? I feel a bit like Don Quixote in even pursuing this topic, but after 15-plus years of (mostly) working on search and text processing (including writing a book on the subject<\/a>) I can\u2019t help but feel that it\u2019s time for the word \u201cunstructured\u201d to be retired and for us to find a better term to describe all of this stuff spewing from us and our computational creations.<\/p>\n

\"Structure:Data:<\/a>Why all the (somewhat tongue-in-cheek) vitriol towards such a simple word? When I\u2019m feeling cynical, I think that, in the early days of databases, someone coined \u201cunstructured\u201d as a derogatory term to mean \u201call the stuff a database isn\u2019t good at working on.\u201d\u00a0If \u201cstructured\u201d is good, then \u201cun\u201d-structured must be bad, right? The problem is that working with text is one of the defining computational challenges of our time. We need our best and brightest working on it; and not just so we can better target ads to consumers. It\u2019s too full of promise to describe with such a diminutive word as \u201cunstructured.\u201d Numerical data? Child\u2019s play! Text? Now there\u2019s a real challenge.<\/p>\n

Text is easily one of the most highly structured data types we face, filled with misspellings, misdirection, flowery language, ambiguity and implicit knowledge. Text is so often misunderstood that researchers in the field even have a metric (inter-annotator agreement<\/a>) that tracks how often two people examining the same piece of text agree on the answer to some question on the text. Authors like Faulkner and Joyce treat language as an art form, yet Joe Forum User can\u2019t, for whatever reason, write a complete sentence. I hate them and love them all, all at the same time. How good do you think a computer can be at parsing a single sentence that spans multiple pages, much less try to make sense of a sentence that doesn\u2019t even follow basic grammar rules?<\/p>\n

Sure, we\u2019ve made great progress, especially in recent years, in dealing with rich data like text, and big data and deep learning<\/a> techniques hold even more promise to unlocking some of the mystery. We can now detect the end of a sentence with a high degree of accuracy, find sentiment in tweets and locate the mentions of people on a page, just like your average fifth grader!<\/p>\n

Yet, despite all of these advances, we need at least an order of magnitude advance (if not two or more) in our ability to process rich data across a variety of domains for us to truly harness the opportunity this data presents in moving civilization forward. I hope you\u2019ll forgive me if the word \u201cunstructured\u201d leaves me feeling a bit empty inside when I think about that opportunity and the lack of inspiration it provides to potential contributors. As for me, I\u2019ll start by calling it \u201crich data\u201d from here on out, windmills be damned.<\/p>\n

Grant Ingersoll, who will be speaking at <\/i>Structure:Data on March 20-21<\/i><\/a>, is the CTO and cofounder of LucidWorks<\/i>. He also coauthored <\/i>Taming Text<\/a>, cofounded Apache Mahout and is a long-standing committer on the Apache Lucene and Solr open source projects.<\/i> He\u2019s engineered a variety of search, question-answering and natural-language processing applications. You can follow Grant on Twitter @gsingers<\/a>.<\/i><\/i><\/b><\/p>\n

\u00a0<\/p>\n

\"\"<\/p>\n

<\/a><\/p>\n

Related research and analysis from GigaOM Pro:<\/strong>
Subscriber content. Sign up for a free trial<\/a>.<\/p>\n