As organizations expand their digital infrastructures, the amount of data being collected is growing at an ever-increasing pace. This deluge presents a new set of challenges, and the most crucial of these — making sense of it all — depends on data visualization. Digital tools like Many Eyes and Tableau Software have empowered companies and the public to create visualizations using built-in templates, but they have also spurred a desire for more control over visual method, layout, style, and branding.
The Open Web, a platform for building web scale software using open technologies, is changing the landscape of data visualization. Open source libraries such as Raphael.js and d3.js democratize data visualization, making the process and code transparent. Additionally, the communities supporting these tools are vibrant.
With these new opportunities, however, comes a need to unravel the production pipeline from messy and complex data sources to the polished results we are all used to. It’s not as easy as it sounds. Deconstructing what actually has to take place in order for data visualization to happen is something we’ve been refining here at Bocoup.
Here’s our formula:
Planning for data visualization starts with a conversation. When our clients initially state their goals — “we want to create data visualization of X” — we tend to ask “why” repeatedly, in a user-centric way. By asking questions, we identify the audience and our goals as creators, which in turn informs the rest of the process. Answering these questions is not just important to moving forward — it often transforms the project entirely.
The next thing to remember is that it’s common to expend considerable effort transforming the data into machine-consumable forms, cleaning out any rogue data points and reformatting it into shapes (like tables or JSON formats) which lend themselves to computation. This is a process that anyone will tell you takes longer than expected, so don’t get too frustrated. It’s worth the effort.
After, data discovery focuses on understanding the parameters of the data, because the data alone can’t offer information on the underlying relationships. What outliers exist in the data? Are they meaningful or simply adding noise? For example, when exploring sub-prime lending, economists from Stanford uncovered what they thought was an error in their data: a large, unaccountable surge in loans in the early months of the year. Upon further investigation, they discovered the anomalous rise was due to the effect of the earned income tax credit, dramatically changing the direction of their research.
When the data’s forms, shapes, and curves become second nature, the really fun part begins: asking it more questions. It is tempting to start a data visualization project with an idea for a striking visual artifact illustrating a conclusion, and then work backwards. Much like our initial conversation to discuss goals, clients will often say, “we know that X is true and I want a data visualization showing that.” But is it really true? Even if it is, is that really the right fact to communicate about the data?
More often than not, challenging some of these assumptions can have a profound impact — not just from the perspective of the immediate project, but for an organization as a whole. The data, then, exists for two purposes: verifying what we already know, and exposing us to what we don’t. Having the time and space to explore both of these goals in a phase of analysis is crucial to preparing the data to be visualized. When we truly find the data to support what we’re trying to communicate, the visual forms will emerge naturally.
Now — yes now — you we start actually visualizing the data. It’s time to think about the audience. What are its characteristics? What decisions are readers or constituents expected to make based on the visualization? For highly-engaged audiences with their own questions, an exploratory tool is best, like this one from the University of Utah:
For a busy audience, a high-level, static graphic may be most appropriate. From The Economist:
If the data is complex, an interactive step-through interface focusing on a single narrative — like this feature from The Guardian — may best serve your readers (click on the previous link to view the graphic in full):
Many of these approaches can be combined in the right circumstances.
When the delivery format is understood, the constraints of the audience and their expectations clear, it’s time to play around with visual designs. Experimentation is crucial. Not all visual designs are created equal; depending on the relationships in the data, some may be more appropriate than others. Targeting a certain format and screen size introduces a different set of constraints and possibilities. The best solution is often the simplest one, and rapid iteration and prototyping is the only way to ensure we are representing the data and narrative accurately.
In practice, this pipeline feeds itself. As we understand our data, it brings up new and better questions for us to answer. Budgeting time for this exploration is necessary to creating a valuable resource that will help you communicate with your audience. The greatest mistake we see with data visualization is taking the easy way out — oversimplification of the data (such as averaging it out) can hide intricacies and important patterns, but often visualizing all the data, results in “hairballs” or distributions with no patterns.
In the end, the data visualization workflow requires you to be surprised. When working with data you can expect to be wrong, to fail fast and to fail often. More than any other engineering practice, data visualization requires an iterative approach to account for the changing nature of your findings as you work.
This approach is strongly supported by modern data visualization practices on the Open Web. The web offers us tools for rapid prototyping; instant support and feedback; development practices that can evolve fast and grow with the changing ecosystem, and powerful tools for interactive software development.
Working on the Open Web does not eliminate the complexity of the underlying problems we’re trying to solve, but it does offer users and new developers the ability to read, tinker with, and share code because of the collaborative nature of the ecosystem. You’re never alone. Regardless of whether you’re a startup or established organization, the Open Web can prove invaluable in making sense of your data and presenting it well.
Visualizing Data
An HBR Insight Center
Microsoft has released an app for Windows Phone called Authenticator, which is designed to generate security codes associated with two-factor authentication. Nothing special so far, other than Microsoft’s name being associated with the app. What is noteworthy is that, according to the release notes, you can use Authenticator “to help keep your Microsoft account secure”. Is Microsoft finally taking the user’s security seriously?