Author: Marshall Kirkpatrick

Geolocation is quickly emerging as a big new platform to build all kinds of cool services on top of. While there’s a whole lot of potential – there’s also growing concern about the privacy implications of this flush of data about where we are. Thus it’s timely that a committee of the US House of Representatives is holding a hearing next week to investigate the issues between commercial use of location data and consumer privacy.

Below you’ll find information about that hearing and five other top stories from the past 24 hours around the web on geolocation, selected with assistance from OneSpot.

Speaking of location, watch this space for forthcoming announcements about ReadWriteWeb research reports and events focused on location as a platform.

Sponsor

“It seems that over the years whenever LBS [location based system] technology makes advances into the consumer space the topic of privacy and security creeps up – and for good reason,” writes Glenn Letham of GISuser. Letham first spotted next week’s hearing titled Joint Hearing On “The Collection And Use Of Location Information For Commercial Purposes. (He describes his relationship with the hearing in comments below.) It is scheduled for next Wednesday at 10 AM EST.

We’ve requested the list of hearing witnesses from the committee and will update this post when we find out who will be speaking.

Best Practices Already Being Hashed Out

One likely suspect is Loopt, a very popular location based social network that transmits passive location data to a user’s chosen network of friends and allows them to push selected location updates out into public networks like Facebook and Twitter. Loopt CEO Sam Altman says he doesn’t know if anyone from his company is speaking at the hearing but that Loopt team members have testified before Congress about user privacy before and found it quite productive. Brian R. Knapp, Chief Privacy Officer and General Counsel at Loopt, says he’s been helping people from some other companies prepare for the hearing next week.

Altman says Loopt keeps a close eye out for abuse cases and has instituted warning systems like algorithmic monitoring of user behavior, SMS messages to make sure users know they are tracking their location and postal mail sent to the homes of children who sign in. Altman says Loopt participated extensively in the writing of the CTIA best practices document for user location data.

Does Altman think the location based economy needs some regulation? “I come up on the Libertarian side of government regulation,” he told us, “but it’s bad for everyone if someone is playing fast and loose with location data and something bad happens. Regulation may be too strong a word but we need some understanding industry-wide about how to respect privacy and keep people safe.”

While many location apps are based on explicit “check-ins” by users and others ask users to opt-in to allowing a service to know their location so that location-features may be leveraged, Altman believes that ongoing, passive location tracking will become more common in the future.

“When passive location becomes mainstream,” he told us, “and I think it will because there are so many upsides, over the next 6 months it’s going to become more important that everyone do it the right way.”

Location as a platform and the privacy challenges therein are going to be hot topics this year. Stick with ReadWriteWeb for ongoing coverage.

In other location news today…

MWC: Smaato Eyes Geo-Ad Markets Outside US
GPS BUSINESS NEWS

“Activity in the location-based advertising market is growing rapidly and not just in North America. That’s according to Ramy Yared, managing director of adsmobi, the newly-established media buying arm of mobile advertising firm Smaato…A recent report from JP Morgan’s analyst Imran Kahn forecast that mobile advertising is set to grow 45% to USD3.8 billion in 2010.”

Your phone can translate foreign language text just by looking at it through Google Goggles. A South Korean telecom has released a product similar to Google Goggles. The social content Augmented Reality mobile browser junaio will have a new version released at SXSW next month and there’s now an 8 minute video from TED available detailing Microsoft’s plans for Bing, including Augmented Reality.

Augmented Reality (AR), the practice of displaying data on top of our view of the world around us, is hot stuff. Below are the top stories on AR from around the web over the past 24 hours, selected with help from OneSpot. Watch this space: ReadWriteWeb will be releasing a research report on the use of AR for marketing very soon.

Sponsor

Google Demos New Translation Functionality For Google Goggles
Geek Sugar

“We’ve been able to translate languages with the help of Google Translate for a while now, but this new function within Google Goggles (which I’m already a big fan of) kind of blows my mind.”

metaio @ SXSW 2010
Augmented Reality Blog

Metaio will present a whole new version of its social content mobile AR browser junaio at SXSW this year. The company’s Unifeye Mobile SDK is also among the finalists of the “Accelerator” competition.

A New Augmented Reality (AR) Application in Korea : Ovjet
GIS + AR (Augmented Reality)

“Korean mobile network provider SK Telecom has revealed a new augmented reality application called Ovjet for Android-platform mobile phones. It seems like quite similar to the concept of Google Goggles. “

OllieBray.com: Microsoft Bing Maps augmented reality demo at the TED 2010

Short 8 minute TED Talk from Microsoft’s Blaise Aguera y Arcas on the company’s latest evolution of Bing Maps. Included an Augmented Reality demo. Here’s the video.

Discuss

Google Maps is a great program, but it’s always been disappointing to see where in the world it doesn’t offer much coverage. Today the Google Africa blog announced that local business data has been added for 30 countries in Africa.

“As well as searching online Maps for towns, highways, or roads,” Joe Mucheru and Jarda Bengl of Google wrote today, “Google Maps users can now find local businesses. This could be a burger place in Lagos, a garage in Kampala, a hairdresser in Accra or an airport shuttle in Dakar.”

Sponsor

Google added a large number of streets and roads throughout Africa to its maps in May of last year and has offered mobile location services in Africa for several years.

Today’s announcement included pointers to the Local Business Center and Maps API, so users in Africa can help fill out the maps all the more.

Discuss

Twenty-year PR veteran Sean Garrett just announced that he’s leaving the communications startup he helped found five years ago and taking a new position as vice president of communications at Twitter.

Garrett says he’s been consulting for Twitter for the past few months, via 463 Communications. Twitter has been hiring fast lately, often picking up smart people working at partner companies.

Sponsor

Garrett is a UCLA grad in Political Science and first worked in the office of California Governor Pete Wilson in 1990. He went on to do PR with many of the biggest tech companies in Silicon Valley.

Garrett created an account on Twitter on March 8th, 2007 (co-incidentally, one day after I joined, so our anniversaries are coming soon, Sean!) and pays attention on Twitter to people like Mike Masnick of the tech blog TechDirt and maverick PR guy Jeremy Pepper.

Twitter desperately needs communications help. The company has grown in importance faster than it’s been able to keep up with. Garrett’s phone will no doubt be ringing off the hook.

He’ll be working in the Communications Department with Jenna Sampson, who came from Yahoo!, former CNN correspondent Alexa Lee and Emily Pinkerton, among others. You can follow all four on Twitter via this list.

Discuss

Boulder, CO based Graphic.ly has put some of its venture capital to work and rolled up 10-year old comics community iFanboy. Graphic.ly is developing an as-yet unlaunched comic and graphic novel reading application for Windows, Adobe AIR and mobile phones. The company has raised just over $1 million in backing. According to Joseph Tartakoff at PaidContent, the company already has clients, including Disney’s Marvel.

The iFanboy site is a place for comics fans to find reviews, audio and video shows. It publishes under a Creative Commons license and sees just over forty thousand visitors per month, according to Quantcast. Some portion of those visitors are paid subscribers.

Sponsor

The iFanboy community appears to be embracing the merger enthusiastically in comments on the official announcement. That’s not something that can be taken for granted. The site’s journalistic objectivity comes under scrutiny now that one vendor owns iFanboy, something the site founders discuss explicitly in their announcement.

Graphic.ly CEO Micah Baldwin describes the deal in lots of detail on his personal blog and says the two companies were introduced by Joe Stump, co-founder of hot pre-launched geolocation API provider SimpleGeo.

Graphic.ly came from Boulder’s TechStars incubator and raised its money from a diverse group including but not limited to Houston’s DFJ Mercury, Starz Media, Northstar Equity Partners, Dave McClure, Chris Sacca and Paige Craig.

Comics and graphic novel fans eagerly await the launch of Graphic.ly. We eagerly await seeing this tech-savvy startup in an eye-catching niche publishing market make its next move.

Discuss

Lifehacker’s founding editor Gina Trapani has just announced that she’s joining Expert Labs, the government-oriented independent think tank led by Anil Dash and funded by the American Association for the Advancement of Science. Her first project: deploying her Twitter-based decision making software called ThinkTank for the White House.

The Obama administration announced the joint project with Expert Labs in a blog post earlier this month. As Trapani described it today: “The President has identified a series of scientific and technical challenges that are as important to the future as the moon landing was. And we want to help drive feedback on that list, and even suggest what other items should be on there that haven’t been included.”

Sponsor

Trapani explained today that the fundamental qualities of ThinkTank are that it leverages existing social connections (today on Twitter, soon on Facebook and later beyond), that it stores the data on your own server just like WordPress does and that it’s open source and community developed. The software sounds like “the WordPress of Twitter data harvesting.”

Trapani has been one of the foremost voices in tracking the ways the web can improve everyday life and work. She focuses on making new developments accessible and is also a respected software developer.

Discuss

Social network Facebook has passed Yahoo! and is now nearing Google in the #1 spot for most monthly unique visitors from the US, according to traffic analyst firm Compete. Compete just published its January numbers this morning and reported that nearly 134 million US web users visited Facebook last month. Google saw nearly 148 million.

Google’s US traffic grew by just over 1 million visitors between December and January. Facebook’s traffic grew by almost 1.5 million. Yahoo’s traffic fell by almost 1.5 million users. As Compete’s Aaron Prebluda writes, Google passed Yahoo 2 years ago this month “and never looked back.” This isn’t just jockeying between companies, though. We may be witnessing the eclipse of search by social networking.

Sponsor

Three weeks ago we wrote about other numbers that pointed to the rise of social networking vs search (Why is Google Afraid of Facebook? Because Social Networking Could Soon Pass Search) but those numbers were a little less clear.

As we wrote then:

What would it mean if social networking over-took search in terms of sheer visits online? It would mark a sea-change on the internet. No longer would our dominant use of the web be seeking out web-pages built by HTML web-masters! Now we would all be publishing tiny little updates that perhaps only our friends and family care about. We’d be subscribing, more than we ever did by RSS, to syndicated updates from organizations of interest, large and small. It would be (perhaps will be) a very different era and, to be frank, it’s going to be harder to monetize. There will be privacy battles. There will be new platforms for innovation.

It’s a pretty big deal. Things will really change if current trends continue and social networking rises to the top… And that’s a big reason why Google and Facebook are rivals.

Discuss

Geekdom may be a land of big personalities, but some peoples’ stories are better known than others. One story you might not know yet is the tale of Scott Kveton’s young, unusual, accelerated and admirable career. Not yet 40 years old, Scott Kveton built the organization that houses the Linux kernel and he saved the day when Firefox launched but all the download mirrors were overwhelmed. He was key to bringing the biggest web companies to the table to develop OpenID and other federated identity systems. Then he spent a year selling bacon on the internet, until flipping bacn.com to an acquirer and signing a deal to write a book about lightweight, agile startups.

Kveton is a dynamic, intelligent, skilled and flawed human being who is creating a very interesting life story for himself. Last night he announced his next big move. As you can expect, it could make a substantial impact on the future of the web.

Sponsor

Last night the innovative “mobile push as a service” company called Urban Airship that Kveton and a handful of savvy, unemployed friends launched just over six months ago announced it has raised $1.1 million in financing to grow into a bigger service provider. Push notifications and in-app purchases are a very big deal for the mobile developer ecosystem. Can Kveton build this behind-the-scenes group of engineers into a viable company? Early momentum makes odds look good and Kveton’s personal history says that one way or the other it’s going to be a wild ride.

Scott Kveton’s Heavy-Lifting Early Days

When Kveton was 31 years old he founded the Open Source Lab at Oregon State University in Corvallis. After working at various big tech companies for a few years, he had joined the University and cut its hardware budget by 75% the previous year – just by buying open source Linux servers. The school decided to put the budget surplus back into the paradigm that made the difference. Then Google, IBM and other big companies started giving the new Lab money to host open source projects they were working on. Soon Kveton had a staff of 25 students and contacts all over the Open Source world. That was 6 years ago and those contacts have been invaluable throughout the rest of his career.

Shortly after the Open Source Lab’s birth, the Linux kernel was in need of a new home. The parties responsible had built the server from the bare metal up, thus they were assured that no security compromise could be introduced from the outside at that core level and infect countless computers around the world. Arrangements were made to host the kernel at the Open Source Lab, the machine was flown from San Francisco in a Cessna and Kveton rolled out a red carpet. That essential piece of hardware remains at the Lab today.

Within that same first year of the Lab’s lifetime, Firefox was born and officially launched to the public. Kveton suspected that the public reception would be enthusiastic enough to melt the network of mirror download providers for the browser around the world. So he helped write a piece of software that would manage that network automatically when under a heavy load. Kveton says Mozilla didn’t want to use his program, but by 9:30 AM PST the mirror network was toast. With a workable network management system quickly put into place, Firefox went on to see 1 million downloads on its official launch date.

Going Startup

After continued success hosting other open source projects (like Drupal) at the Lab, Kveton decided he wanted to try something entrepreneurial.

He became the CEO of online identity provider JanRain and co-founded the OpenID Foundation. The Foundation brought together many of the biggest companies in the world and many innovative engineers, to talk about how companies would be better served by allowing users to travel freely from site to site with interoperable identities than by locking users in to a single, non-portable identity. Kveton was the Chair of the foundation for 2 years, then co-Chair for one year. He explained his interest and belief in the idea like this three years ago:

OpenID really is a grassroots, bottom-up approach. For something like this to be compelling there can be no hook back to the “mother ship”. It’s truly got to be open and decentralized and that’s one of the main reasons people are finding it compelling. Has federated identity failed? In the past, yes. I believe in 5 years, there will be a federated identity that people use all over the Internet; you’ll have one login and it won’t be controlled by anyone but you. OpenID is hopefully going to be the driver of that; the HTTP of identity. Nobody but you should own your identity.

Kveton gave OpenID all he had for several years. He says that in time his family suffered and that he needed to leave his post as CEO of JanRain. He says he’s still active with the company. He says this is an issue he struggles with: balancing work and family. “I’m a workaholic. That’s how it is, that’s one of my big priorities,” he told me with a little sadness. “It’s tough to balance out with family life and two kids. You do what you can and that’s the best you can.”

Kveton’s next stop was a brief one with a company called Strands. That company was aimed at providing recommendation services for music venues, banks and other organizations. Its top executives were from Spain, the company raised more than $50 million and one of its two main offices was inside an old mausoleum in Corvallis, Oregon. Here at ReadWriteWeb, we were very excited about Strands. Kveton only stayed at the company for 8 months.

His next stop was at identity provider Vidoop. That company imploded, telling some of its employees they wouldn’t be paid back wages but could take computers home from the office as compensation. According to OpenID community leader and now Google employee Chris Messina, though, the company fired Kveton months earlier, for going to South by Southwest on its dime but actually speaking at the event about…selling bacon on the internet.

That was Kveton’s next project, a website called Bacn. “I’d ran my own blog about bacon,” he told me,”and people would send me links on Twitter about bacon. I thought, ‘I wonder if there’s a business here.’ I talked to people who sell bacon on the internet and found out that they knew bacon but not the technology.”

Kveton talked to local web designer and information architect Jason Glaspey, just back from a months-long trip to Argentina, and to a young coding wizard named Michael Richardson, who was unemployed since Vidoop’s collapse. “The day after Christmas I said let’s launch a company at the Masterbacon conference on January 17th,” Kveton recalls. “No one said we couldn’t do it, so that’s how we did it. Loads of logistics, fulfillment, postage, etc. all got done in 21 days. We had to think fast. That was one of the funnest things I’ve ever done. We put a modest investment of $10k into it, bought the bacn domain, Jason’s wife did branding. We made shirts and paid some models with PBR and pizza. After it launched, the business grew fast. Once we did 680 pounds of bacon in one single day. It came in, we boxed it up and shipped it out in one day.”

At roughly the same time, Kveton took on a short-term role as the Interim President of the Software Association of Oregon. That was never intended to be a long-term gig and another opportunity quickly presented itself.

Urban Airship

Colleague Steven Osborn was consulting for Subattomic Studios, the gaming company behind Field Runners. That company wanted push notifications and in-app purchases to be added to its iPhone app and Osborn saw just how non-trivial it was to set up that infrastructure. He reached out to Kveton and Richardson about it. Kveton: “Steven said, ‘I don’t know if there’s a business here.’ I looked at it at said there’s a huge business! Michael asked if we could name it something with the word Airship in it. So we put it into the web 2.0 name generator and got Urban Airship!”

And that’s how Osborn, Richardson and Kveton started a company. That was in May. Come June, Kveton bought a car-load of Danishes and drove to the Apple World Wide Developer’s Conference where push notification was formally announced. He worked the line at the door, handing out pastries and telling developers about the company he was launching.

One person he talked to was his old friend from Mozilla, Bart Decrem. Decrem is a Belgian who left Mozilla to lead the social web browser company Flock, then left there and became CEO of Tapulous, the wildly successful maker of iPhone games like Tap Tap Revenge.

Decrem told Kveton that his company was big enough they preferred to build their own push notification infrastructure. Then Apple told Decrem that Tapulous had 8 days to implement push in order to launch as part of a big announcement. Decrem quickly got on the phone with Urban Airship. Kveton and team put the pedal to the metal and built push notifications for Tapulous ahead of a very tight deadline. (See our full coverage of that story: How Urban Airship Saved Tapulous’s Bacon on iPhone 3.0 Day)

From there the company grew fast. As it says on its website:

Within one month, Urban Airship launched, landed its first customer, and powered the first App Store app that used push notifications. In the subsequent –short — months, the team has signed more than 1,500 customers, sent more than 110 million messages and connected to more than 10 million mobile devices.

Why are push notifications and in-app purchases so important? As we’ve written before:

Push notifications, like when Tapulous now tells users who don’t have the app running that they’ve been challenged to a music-playing match by a friend, are something developers believe will increase ongoing engagement long after the initial download of an app. In-app sales will help monetize that engagement, something developers have found challenging after an initial flurry of sales, once they are lost in a sea of options in the app store and no longer making money sitting beside countless other apps on peoples’ phones.

Next Steps for the Airship

And so we catch up to today. Last night Urban Airship announced that it has transitioned from bootstrapped project to funded corporate venture. The round of funding was provided by San Francisco’s True Ventures and Seattle-based Founders Co-op.

What will the company do with the funding? So far its assets are a few laptops and some code. “More of the same on a larger scale,” Kveton says. “We’ll be adding engineering. We’re very excited about iPad, we’ll have full iPad support. All these other devices are going to enable push notification – the future of the mobile market makes the PC look tiny.”

As Urban Airship ramped up, Kveton and crew sold the Bacn to BaconFreak.com. It was proving a distraction and the team was happy to sell it just a year after it was founded. Another conference presentation about the rapid launch of Bacn.com lead to a book deal with the title “Makin’ Bacon: From Idea to Startup.” That’s due out in April.

Kveton has also helped launch a web technology incubator called P.I.E. with a select group of innovators and global advertising company Wieden+Kennedy. Kveton says WK likes to have Urban Airship in the building because the phone geeks help inform the advertisers about the cutting edge of what’s possible. WK and its clients in turn help inform Urban Airship’s perspective on a number of different markets relevant to the startup.

Even critics who call him things like a charismatic opportunist admit that Scott Kveton is a very likable guy – and he clearly knows how to leverage the relationships he’s built throughout his travels.

A book and a related incubator shouldn’t be too much for Kveton to balance with running a now-funded tech startup company. But all these circumstances beg the question: is Urban Airship where this high-energy innovator is going to land for a good long while?

“I am excited about taking a company through its life cycle,” Kveton told me last week. “This is one that I know I’m going to be able to drive to fruition, I intend to be here for the long haul.”

No matter how this latest venture goes, Scott Kveton has already changed the web and the world for the better, more than all but a handful of people online ever do.

Disclosure: The author was an early participant in the PIE incubator and has had a past consulting relationship with W+K.

Discuss

Location is going to be big, that seems to be the consensus among geeks, but just how big is it going to be? One metric to wrap your brain around came out over Twitter earlier today. According to SimpleGeo founder Joe Stump, the still-unlaunched but much anticipated service is now indexing more than 1 million location-based objects every hour.

That’s going to make for a very rich database that other services can tap into. SimpleGeo has taken $1.5m in angel funding from of Silicon Valley’s biggest-name investors to try and become the go-to geolocation database resource for the next generation of location-aware applications.

Sponsor

The company was founded by former Digg Chief Architect Joe Stump and the founder of AOL-acquired Social Thing Matt Galligan. Stump explained his company’s model to VentureBeat late last year:

“Location-based devices only provide a latitude and a longitude, sometimes an altitude,” he said. “What they don’t provide is a ZIP Code, city, state, county, weather data, messages and photos posted near the site. They don’t provide business listings, Wikipedia entries, census data (for demographics), articles written or posted near the location,” all of which SimpleGeo does. For example, a location-based game set in San Francisco could accurately display its players gleaming in the California sun, or obscured by Golden Gate fog, based on the real-time weather data from around town.

The company told Liz Gannes of Gigaom in November that it received 600 beta applications on its first day after announcing itself publicly. Gannes wrote at the time:

“We’re selling shovels at the beginning of a gold rush,” is how co-founder Matt Galligan put it on a call today. “You want to add location, just come to us — it’s done.” Though four-person SimpleGeo still measures its age in months, it already has a price sheet: free, $399/month for small businesses and $2,499/month for custom implementations.

This sort of business model for this particular market has been forecast for some time. 18 months ago analyst firm ABI Research, for example, made the following prediction:

“Location-based mobile social networking revenues will reach $3.3 billion by 2013, but successful business models may differ from what many observers expect,” says ABI Research principal analyst Dominique Bonte. “While location-based advertising integrated with sophisticated algorithms holds a lot of promise, the current reality rather points to licensing and revenue-sharing models as the way forward for social networking start-ups to grow their customer base and reach profitability…”

Twitter acquired oft-compared competitor GeoAPI late in 2009 but uptake of Twitter’s location data has been slow. Traditional vendors have also long offered geographic data. Those vendors may fall short of developer expectations if SimpleGeo can deliver things like an effective iPhone SDK, OAuth authentication and dynamic data from sources like Twitter and Flickr.

Can SimpleGeo jump to the head of the new location-based parade and capture what’s expected to be a huge market? Adding 1 million location-based objects every hour to its database sounds like a great place to come at that market from.

Discuss

Three weeks ago Facebook Platform Manager Dave Morin announced he was leaving the company to start a stealthy new startup with Napster founder Sean Fanning.

The new company is called Path and you can sign up to request access to the closed beta at Path.io. Unfortunately, someone forgot to tell Google that the new company was in hiding and so tonight we have a fair amount of information about what these internet veterans are up to, courtesy of Google’s cache of the otherwise locked-up site. The company describes itself as “a tool that facilitates the creation, sharing and correlation of lists.” List creation is nothing new, but that’s not a problem. When world-class social technologists tackle an age-old problem, the results can be quite interesting. Check out what this team has developed so far.

Sponsor

Morin and Fanning appear to be collaborating on the startup with Numair Faraz, an old friend of Napster, Plaxo and Facebook co-founder Sean Parker, and with some mysterious person named Fernando/L.C. from Portland, Oregon.

The site so far is a list of lists. You make a list, give it a name and add items to it. Then you can see who else has made a list with the same name, what’s on their list and what the most popular items are across all lists with the same name. Lists are things like “best coffee in San Francisco,” or “evil corporations.” Faraz started a list with that title, but Facebook is the only company on it so far.

Much like Delicious, there’s a nice clean URL structure like this: path.io/CharlesF/list/best_games or path.io/list/cute_videos or path.io/list/restaurants_in_sf/greens . The site appears aimed to have an API with advanced features like exposing geolocation, priorities and consensus.

It looks pretty basic so far, but there’s a long list of beta users who have created lists titled “Features that Path needs to add.” Some of those beta list builders are some very serious heavy-hitters, like Robert Kissinger (Director of Product Management & Interaction Designer at Electronic Arts), Joi Ito (CEO of Creative Commons) and Jordan Mendelson (Senior Product Manager, LinkedIn).

That Path team has put together a great list of beta testers, so when the company says it doesn’t know yet what exactly it’s going to launch, that has to be taken with a grain of salt.

Can a service for building lists really be that exciting? Many people asked the same question about a social Q&A site when another ex-Facebooker, Adam D’Angelo, recently launched Quora. (Which we also published the first screenshots of.) That site turned out to be amazing in terms of design and user experience.

We have high hopes for Path. It’s another example of a basic human experience turned exciting by the network effects, location awareness, APIs and other features of the new social web. Just like Twitter and Facebook asked you what you were doing and made that the foundation for a world-changing technology platform, now services like Foursquare and Gowalla are trying to do the same with where you are, Plancast with what you’re planning on doing in the future and, we suspect, Path with what you like to bundle together in a list. We look forward to seeing what this All-Star team puts together.

We don’t know yet when Path will launch. We’ll update this post with more information from the founders as we receive it and we’ll take a much closer look once we get access to the site itself.

Discuss

Facebook this weekend published a special Valentine’s day study about romantic relationships and happiness. The company’s Data Team sliced and diced the language used in millions of peoples’ status messages, then looked at how they varied depending on the relationship status the people listed themselves with.

Conclusions? Married people are the happiest, people in Open Relationships are the least happy. Men are less happy than women in an open relationship (believe it or not) and more happy in marriage. These findings are interesting, but what they really indicate is that there may be a modern-day Farmers’ Almanac for understanding our lives hidden behind the company’s doors. Facebook needs to set that data free or at least do more with it.

Sponsor

The Data Team’s conclusions were written up by intern Lisa Zhang, whose relationship status isn’t public on the site. Zhang explained the methodology like this:

We already have methodology for measuring the happiness of Facebook users: by considering how many positive words people use in their status updates (see the USA Gross National Happiness Index). This method allows us to see whether a person’s Facebook relationship status affects how positive and negative they are. We examined the use of positive and negative words in the status updates of all English speakers over the course of one week in January. To protect your privacy, no one at Facebook actually reads the status updates in the process of doing this research; instead, our computers do the word counting after all personally identifiable information has been removed.

One of the most significant findings reported is about people who listed themselves as in an Open Relationship, meaning typically that they are committed to one person but have sex with multiple people. Those people tend to be less positive than anyone else, less positive even than widowed people!

This is truly remarkable, if you think about it. For what tiny percentage of human history has it been a common practice to publicly declare your relationship Open in such a way? It’s a pretty new thing! How’s it working out for people? Apparently not so well!

That could be good information to know before making certain decisions (“if I knew then what I know now…”). Of course, how does having children affect happiness in married people? How happy do people in heterosexual vs same-sex Open Relationships tend to be? Do women who are engaged to men with feminist or liberal-sympathetic interests (Fan pages) express less negativity than other engaged women or is that just a facade that in reality means nothing?

There are lots of questions raised by this data, and it’s just one of an infinite combination of data points that Facebook could analyze.

It’s probable that some of this data includes important observations about the human condition. Information that could help people make better-informed decisions than ever before in human history; informed not only by our own experience, and the experiences of the people we observe in our immediate lives, but by the experiences of hundreds of millions of people around the world.

Other Examples of Important Advice

Remember the Farmers’ Almanac? It’s a book that’s been published annually since 1818, filled with advice about cooking, gardening, humor and weather predictions. The mysterious Farmers’ Almanac team (there’s only been 7 editors in the publication’s history) refuses to disclose its methods for weather prediction, but claims to have a 80 to 85% accuracy rate over its history.

Similarly, Facebook data might not be able to tell us with 100% accuracy whether people who move from Michigan to California, or who marry young, or who stop playing football and start playing basketball tend to be more happy or less happy than before – but that data could come a whole lot closer to telling us than anything we’ve had before has.

Of course we’re all special snowflakes, with infinite complications, and free will is important – but doesn’t it seem that there’s an incredible opportunity for world-wide self awareness hiding inside this social network where we’re typing our relationship status, our location and our interests into fields in a form?

Of course Facebook uses that data to target advertising. Why will the company tell an advertiser that I’m part of a group of college educated, white, married men, over 30 years old and living in the state of Oregon but it won’t tell me that among people in those circumstances I have a particularly geographically limited set of friends and should probably get out more if I want to really understand the world? Now that would be valuable information!

Hunch, the startup lead by Caterina Fake and Chris Dixon, makes that kind of data a big part of its social decision making service. The site provides a way for you to walk through various things you should consider in making decisions like what kind of car you should buy next or where you might like to go on vacation. It also asks you questions about yourself along the way.

Among 10k people Hunch asked “do you like Cilantro?” the ones that said they did were far more likely to prefer dark chocolate over milk chocolate. People who said they didn’t like cilantro are much more likely to prefer milk chocolate. (Probably because people who don’t like cilantro don’t know what’s good in life.) A Hunch study last week of people who own different breeds of dogs found that German Shepherd fans tend to rely more on intuition than common sense and Pug fans particularly enjoyed the movie The Shawshank Redemption.

How about Facebook coughs up the goods on the far greater supply of user data than any other site in the world has?

Unfortunately, Facebook is going in just the opposite direction. The company’s Data Team puts out some lightweight analysis like these Valentine’s Day conclusions about once a month. When software engineer Pete Warden tried last week to offer up user data he’d collected from 250 million Facebook users to the Academic community for study (see The Man Who Looked Into Facebook’s Soul) the company contacted him and told him to put a hold on the release while privacy concerns were evaluated. Last week the company took down Lexicon, its public tool for comparing how often different words were being used across the streams of Facebook users.

Come on, Facebook! Set the data free. It’s not about cilantro and chocolate, that’s just the fun stuff. There are important observations about humanity hiding in that data. Check out, for example, Hunch’s observations of hundreds of people who said they don’t believe Barack Obama was born in the US and what else they have in common.

If you don’t feel like you can set it free, then at least do something more serious with it. The Farmers’ Almanac is a mysterious organization, maybe some more mystery would be ok if it came along with a whole lot of Facebook data.

This is a historically unique opportunity and one that I hope Facebook will take ahold of soon. Think of all the heartbreak the company could help prevent if only people knew their odds of being happy in an Open Relationship, among the countless other decisions we make that would be well informed by analysis of aggregate user data.

Discuss

One day in December I was visiting family and picked up my niece Xander from daycare. We had to leave before her balloon got turned into an animal. It was disappointing. So when we got back to her house, I pulled out my iPhone and showed her an app called Aardvark. We asked Aardvark how we could turn one long balloon into a balloon animal. My niece jumped up and down with excitement every time we got an iPhone push notification that someone out on the internet had an answer to offer. The first 3 people said “draw eyes on it and call it a snake.” That was funny the first time. Then, someone came through with a great link to good instructions for making a balloon animal. We made one, we were happy and proud, and we’d become the kind of people who knew how to make balloon animals.

A month later, Xander was visiting my house and we gave her a package of balloons. She whipped up a giraffe, a horse and a princess crown in minutes. Her mom asked her how she did it and you know what she said? “The Aardvark taught me how to do it!” Google announced today that it bought the company that made that iPhone app. It feels like some closure on my past year of hunting the story of the Aardvark, both personally and professionally.

Sponsor

I’ve asked for and received from Aardvark advice on cooking, home repair, what color shirt to wear on TV, whether I can easily catch a cab at a particular BART stop and how to make balloon animals.

Today Google officially announced the acquisition of Aardvark and its availability in Google Labs. I thought I’d take this opportunity to share a few stories about my year following this company and using its service. There’s no knowing how much attention the project will get inside Google, so this may be a case of “it was fun while it lasted.” But it sure was fun. And perhaps this acquisition won’t be the last we hear of Aardvark, after all.

Founded by ex-Google employees, here’s how Aardvark describes its team that built the system: “Over 2009, the company built an amazing technical team of over twenty people, including engineers from each of Silicon Valley’s major technology companies, four AI Ph.D.s, and founders from a dozen different successful startups.” Those people are all Google employees now and have a tidy pile of money.

Aardvark’s investors got a little bit of money out of the deal, but seeing one of the leading examples of what some people believe is the future of search (social search) sell for a mere $50 million does raise questions. With a total of $1.3 billion invested in various companies, lead backer August Capital is probably disappointed at this small exit, even if it is nearly 10X the $6 million that Aardvark had raised.

The price may well be based on the company’s failure to find a substantial number of users. Aardvark said earlier this month that it had fewer than 100,000 registered users. So be it. The founders will now return to Google, their former employer, with a powerful proof of concept, an eye for the huge Google user base and several million dollars in each of their pockets. Maybe they’ll continue to work on Aardvark itself and maybe they won’t. Only time will tell.

In the mean time, I’ve had a great time using Aardvark and have even put it to work for me professionally.

Last October I was walking down the street in San Francisco after lunch, headed back to the Moscone Center to see what was rumored to be a big announcement at the Web 2.0 Summit. Microsoft was going to announce that it cut a deal with Twitter to include Tweets in Bing search results. The much more connected Kara Swisher broke the news and I was trying to think of how to add value to the conversation with my coverage. So I put out a tweet: “Are there any User Experience experts at Web 2.0 who can talk to me about Bing/Twitter integration?”

By the time I sat down on the floor of the crowded convention hall before the announcement, I hadn’t gotten a single reply on Twitter. So I decided to fire up Aardvark. I asked it by IM, “are there any UX pros available right now to give me a live reaction to some news about to break?”

I was quickly delivered 3 suitable User Experience design professionals from around the country, asking me how they could help, through the Aardvark IM interface. I typed, copied and pasted as fast as I could – sending them the link to bing.com/twitter as soon as it was available, getting their thoughts, asking follow-up questions, separating three live interviews in one chat stream (chatting with Aardvark) into three separate interview chunks of text. It was crazy! I typed and thought and parsed as fast as I possibly could and then boom – within minutes of the announcement being over, I had a blog post up. Three User Experience professionals react to the way that the first major search engine to do so integrated the Twitter stream. It was quite a rush and something I couldn’t have done in any other way, without Aardvark.

We all knew that Aardvark was born and raised to be sold, probably to Google. When Michael Arrington broke the news 2 months ago that Aardvark was in talks with Google, it wasn’t a surprise. Nor was it a surprise that TechCrunch broke the story, Arrington has held an annual summer event at the offices of August Capital, Aardvark’s lead investor, for years. He’s very connected to the circle of people around Aardvark, as he is with hot Silicon Valley startups quite often.

A few days after that news came out, Aardvark CEO Max Ventilla was a guest on Leo Laporte’s show This Week in Tech. I ended up butting in far too much, explaining Vark and telling stories in the TWiT chat room that Leo asked Max about live on the show. I was a little worried that Max was going to get tired of me. I’d been trading emails with him, cursing him for giving exclusives to other media outlets, interviewing him at length for our research report on the future of the Real-Time Web and just generally being a harassing fan and overeager news blogger.

After the show, I shot him an email anyway. I told him that I’m not connected enough to break the news that Aardvark is about to sell, but I’d like to try to out-write my competitors. Just like the New York Times writes obituaries for famous people before they die, I’d like to spend some time with him so I can write the story of Aardvark ahead of time, before it gets acquired.

He told me there was no rush, that nothing was really happening, but agreed to schedule a call. We scheduled some time, but that morning a pipe exploded in the basement of my house. I emailed him and said I’d reschedule sometime soon.

That was two weeks ago. I never got to dive deep into the story of Aardvark, before it got acquired, and now there’s no telling what the future will bring for the company. But I did have a great time chasing Aardvark around in my personal and professional life over the last year. I know how to make a mean sweet potato and butternut squash soup thanks to Aardvark, and I’m not afraid to put certain Arm and Hammer products on my carpet to vacuum up, even if they aren’t labeled for it explicitly. Thanks, Aardvark community.

These days I haven’t been responding to my Aardvark IMs as much as I used to. I used to answer lots of questions, so now I get a lot of questions. Most of them are on topics I have no interest in. I spent the end of last year chasing down the next social search company that caught my fancy, the then-unlaunched Quora, built by some of the very first people to join and leave Facebook in the early days. I posted the first screenshots of Quora and use it regularly still, but as a web technology writer it’s my job to be looking for the next new thing.

I still enjoy Aardvark and I love the ideas behind it. We’ll see what happens to it at Google, but if absolutely nothing else: my niece and I now know how to make balloon animals. I think that’s very cool.

Congrats on your sale, team Vark, and good luck changing the world of search at Google.

Discuss

For all our supposed internet leadership, we North Americans are more likely to use Internet Explorer than people in any other continent in the world. Respected website traffic analysts Quantcast see a whole lot of people flying around the web every day and this morning the company published some browser numbers broken down by continent.

What part of the world has the highest percentage of people who use the best browser available, Google’s Chrome? Good job, South America, you’re number one. North America isn’t just #1 in I.E. use (not that there’s anything wrong with that), we’re also in last place for Chrome and Firefox. Check out the chart below.

Sponsor

What’s the takeaway here? At the very least it means that we North Americans should remember where we stand in terms of online sophistication by our mainstream population, relative to some other places in the world.

George Stephanopoulos. Wolf Blitzer. Ana Marie Cox. Three powerful people that you might want to get in touch with, especially if you’re in D.C. One man who has the ear of all three of those powerful people is Tony Fratto, known as @TonyFratto on Twitter. Managing Director at Hamilton Place Strategies, the founder of RooseveltRoom.net, a CNBC Contributor and a former US Treasury & White House official, Mr. Fratto is the least well-known (in Twitter terms) of the mere 41 people that all three of our sought-after stars have opted-in to following on Twitter. He could be the diamond in the rough, the guy you want to know. He’s probably got some important things to say, too.

These days, it’s all about who you don’t know. That’s the theory behind a group of very interesting software projects being built on top of the giant graph of friend/follower connection data that Twitter exposes about its users. These tools unearth potential connected influencers like Tony Fratto.

Sponsor

Using People to Find People

Name 3 people whom you admire, despise, work with or otherwise pay attention to and tools like HiveMind, Follower Wonk and Twiangulate will quickly calculate who all those people are following in common on Twitter. Tell those services who you are and they’ll expose the people from that list that you aren’t yet following yourself.

Services like this stand in an interesting place online: they aren’t too hard to build and they delivery huge value to their users, but so far they have had a hard time getting people to try them out.

Using social network data is easiest with Twitter today, because of its more open nature, but this kind of work is being done on top of Facebook as well. (See this week’s article The Man Who Looked Into Facebook’s Soul for example.) When it comes to using people to find people, it’s almost literally a 2.0 version of what some of the biggest companies in Web 1.0 did.

Adam Lindemman, CEO of a synaptic web company called Imindi that burned up trying to enter the earth’s atmosphere last year and has since been mothballed, put it very well:

“This is essentially collaborative filtering used in recommendation, for people as the object rather than Books, CDs or Movies (Amazon) or search results (Page Rank). In other words we are seeing the emergence of something like the kinds of tools that helped us navigate the information and e-commerce glut in Web 1.0 and using them to help us manage the main glut of Web 2.0 – which is the abundance of people and the need to literally mine that noise for signal.”

We spoke with the creators of those three services in particular (Twiangulate, HiveMind and Follower Wonk) to find out what they’re thinking as they build and promote social graph analysis tools on top of Twitter.

The Fabulous Uses of Friend Network Analysis

Henry Copeland, the creator of 8-year old blog ad network BlogAds, is building Twiangulate on the side, late at night.

Copeland is the one who told us about Tony Fratto, based on the Twiangulate map of common connections among those three media stars. He also points to a map his service created of the most inter-connected, yet otherwise obscure, friends of actress Demi Moore. That’s kind of creepy.

A more benevolent use case? Say you’re a reporter covering New York city schools. Who might be a good source to contact? Here’s what Copeland recommends: Look at Mayor Michael Bloomberg’s list of followers on Twitter. See which ones look like education-related organizations. Use Twiangulate to see who both that organization and the Mayor are following. Look for an education-related organization on that list and add it to your comparison. Repeat, until you’ve replaced the Mayor and are just looking at a list of who is being followed by 3 of the most-connected education organizations in the city. There’s a good list of people to be in contact with. That’s pretty hot stuff.

Freaky Geeky Stuff

Why are services like this not taking off like wildfire yet? “Right now we’re each building a fascinating tool, but we haven’t quite shaped the handle right so it fits in the user’s hand,” Copeland says of the three companies we looked at doing similar kinds of work.

“Then, what exactly does the customer look like? In many cases the right user doesn’t yet exist. There is not yet 10k people designated as blahblahblow specialists at companies across America. There are geeks and aficionados and crazy people with visions of something that might be palm trees on the horizon… or might be a mirage.

“People need to develop habits of thinking about this stuff. They need someone to explain it to them. They need to develop an easy-to-use vocabulary set for understanding it and communicating about it. Right now ‘social network analysis’ just doesn’t trip off the average person’s tongue.”

It may not roll of the average person’s tongue, but there’s clearly some very serious interest in it. Kevin Marshall is the creator of a bundle of social graph analysis tools called Wow.ly, including one much like Twiangulate, called HiveMind or Grou.pe. HiveMind will show you who you aren’t following that a group of selected Twitter users are – and it will email you when that list changes.

“I have a couple of friends that are in the VC business around New York city,” Marshall told us.

“They wanted make sure that they knew about any new deals and companies at least as fast as their competitors. So this is all just one tool to help with that sort of thing…without having to constantly check who their competitors are following manually.

“It actually feels a little like an invasion of privacy when you start getting the alerts though, because you get to see when people you are interested in start or stop following other people. But from a ‘business’ point of view, I think it’s really powerful.”

The Low Hanging Fruit

No potential privacy violation would be complete without politics and advertising. Sure enough, our third miner of the new social graph focuses on those two parts of the world.

Peter Bray lives in Portland, Oregon and built FollowerWonk.com. Follower Wonk displays common followers, as well as people that users are following mutually. It also offers some visualizations and the ability to search inside users’ Twitter bio fields. Bray thinks this technology has real potential to change the world, in the short term even.

“As Follower Wonk matures, we imagine an application where a political campaign or a brand can slice & dice followers into highly leveragable segments. For example, a Democratic candidate might be able to find cross-over votes by targeting Twitter users who follow the competing GOP candidate, but also, say @sierra_club or @barackobama. Campaigns, or brands, can find and utilize these natural ‘tension points’, not only in terms of actual individual users, but also what interests (environment?) or messages might be potent. As social media matures, these sorts of strategies, already so common in the world of list management, are naturally going to play a significant role.”

We asked Mr. Bray, if these tools were so darned potent, why aren’t more people excited about them yet?

“In the universe of Google users, the percentage using, say, Google Analytics or Website Optimizer is teensy,” he replied.

“Similarly, the total number of Twitter users that are going to find value in slicing & dicing their followers is small. But I think that the benefits of such user profiling and tracking are potent, and increasingly evident among brands and campaigns.

“We’ve seen this in one of our other applications, Versionista.com, where the McCain site used our Web page tracking and versioning tool to highlight changes to Obama’s policy pages. Who would have thought that tracking changes to a campaign Web site would reveal important talking points? And similarly, you’re right that few people right now realize the power of Twitter, and social media generally, in terms of the ability to find and engage a receptive audience through advanced profiling and analytics. Or the ability to deduce new messages and arguments.”

Can One of These Visions Come to Fruition?

These are three men have built different tools that do roughly the same things but in different ways, and with different go-to-market visions. There are others, too, like Tickery and Klout. Will any of them be able to popularize something where there’s a level of strategic thought required in order to capture substantial value from a social networking world all-too-often dismissed as trivial?

It’s a complicated but thought provoking question. One that will be discussed on Twitter no doubt. Even there, Henry Copeland, Kevin Marshall and Peter Bray will take different approaches, though. There is not a single person that all three of them are following! There also wasn’t anyone else that was following all three of these trailblazers, either. Until I just did.

Is this kind of analysis a privacy violation? How can privacy on sites like Twitter, Facebook and whatever comes next be balanced with interests in innovation and profit? These are the kinds of questions that the web will need to ask if services like these three become popular. They offer so much value, it seems inevitable that they will.

You can find and follow the ReadWriteWeb team on Twitter here and on Facebook here.

Disclosure: Color commentary above was provided by CEO of Imindi, a company the author has had a consulting relationship with in the past. It was also quite articulate.

Discuss

Facebook announced this morning that its wildly popular Instant Messaging service now supports the open IM standard XMPP/Jabber. That means that 3rd party developers will be able to build support for Facebook Chat into their websites and chat applications with ease.

Standards are great like this for making development simpler but the other promise of technical standards so far remains unrealized. Interoperability is the big promise of open standards in general and XMPP chat specifically, but at launch Facebook Chat by XMPP does not federate with other XMPP servers. So this isn’t about interoperability, it’s about further extending Facebook around the web.

Sponsor

Does Facebook have plans for the future to federate with, say Google Talk, the other leading XMPP chat service online? Or with people using Jabber directly? We haven’t gotten a response from the company yet, but we really doubt it. Facebook could change the game in a big way around IM interoperability if it did so. Unfortunately, this is much more likely to be a case of an open technical standard being used to extend the dominance of a closed market leader.

Update: Facebook’s Malorie Lucich responded and told us that interoperability “isn’t something we’re announcing today, but we are looking into it.” Lucich is the same Facebook team member who advised users last month about how to use Facebook to subscribe to syndicated news sources, so she’s cool. Looking into it? We sure hope so.

Facebook is supporting the open standard, though! And thus it will be much easier for outside developers to build on top of it. That’s great. The open standard of XMPP now has all the more support behind it, all the more reason for developers to implement on it – now it offers access to 400 million Facebook users. That’s nothing to underestimate the importance of.

We sure would love to see someone step up and use open standards to support interoperability between people on different IM platforms, though. It would be great if it was Facebook that did it.

Discuss

Google rolled out a social stream service today called Buzz. It looks on the surface like Facebook, FriendFeed and other stream reading and writing services. It will compete with Facebook and Twitter. Under the covers, though, this major product was built by a team of people taking a radical new approach to online publishing: Buzz is all about open, standardized user data.

Google Buzz data can be syndicated out to other services using the standard data formats called Atom, Activity Streams, MediaRSS and PubSubHubbub. That couldn’t be more different from Facebook. Google has taken open data standards to battle against a marketplace of competitors that are closed and proprietary to varying degrees. This is a very big deal.

Sponsor

Google Buzz was presented as a destination site, but a look at its APIs and developer roadmap indicate that it may actually intend to be a platform – the central hub for a world of distributed social networking. “This is a downpayment on where we’re going with the open, social web,” Google Open Web Advocate Chris Messina told us.

It’s tempting to recoil at the thought of Google powering one more part of our lives online, and our friends’ activity streams are a very important part of the online experience now. But if the growing number of data portability and open web advocates the company has hired can do their jobs well – then Google Buzz could be a big force for good.

People will build services on top of analyzing your public Buzz activity. They will build new applications for publishing to Buzz, just like the Twitter ecosystem has today. Planned support for things like the Salmon commenting standard mean that comments left on Buzz could appear out on blog posts around the web, and comments on blog posts could be viewed inside of Buzz when the post links are shared.

The use of full email addresses in @ public replies demonstrated today seems to indicate that it will be a cross-platform messaging service. Facebook users can only message other Facebook users but Buzz users may be able to reply to people using email IDs from other networks. That’s hot stuff.

Once Activity Streams consumption, @ messages that look like Webfinger profiles to me and Salmon are in place then Buzz users should be able to read, comment on and message to conversations with people who have never seen Buzz in their lives, simply by subscribing to their feeds. There’s huge potential for interoperability here.

Facebook and Twitter will face renewed pressure to publish and consume standardized data feeds as well now. If Buzz is big enough, it could break the dam holding back a flood of standardized data. Where there is standized data, there is scalable network effects, consumer choice, competition and thus innovation.

Buzz’s embrace of the open web could make it a very important player in the development of the future.

Update: One critique to take into consideration is this. Google has scooped up a substantial number of formerly independent open web advocates – most recently Chris Messina, who was the leading spokesperson for the Activity Streams standard. See How Chris Messina Got a Job at Google. In that article we included the following argument from Yahoo’s Eran Hammer-Lahav, the best-known technologist working to develop and support open login standard OAuth. This perspective is important to consider in thinking about the Buzz announcement and standards.

“This is clearly a big win for Google,” Hammer-Lahav told us.

“Messina and Smarr are huge assets in the social web space. My concern is specific to Google. With Messina, Smarr, [inventor of OpenID and more Brad] Fitzpatrick and others all working for Google, focusing on the Social Web, there is less and less incentive for Google to reach out. Google has a strong coding culture which puts running code ahead of consensus and collaboration. Now with so many bright minds in house, they are even less likely to reach out. A week ago, you would have to get at least Google, Plaxo, and Messina (representing the independent voice) to collaborate. This week it’s just Google.

“While I am certain that Messina and Smarr will keep their independent voices, and am not suggesting they will ‘sell out’ or alter their principles, they no longer need to surface many of their ideas out to the community. They can just have an quick internal meeting and ship products.”

Is Google centralizing too much of the decision making about the future of an ostensibly decentralized web? Time will tell, but this may be the heart of the battle for the future of the social web.

Discuss

Youth social networking researcher danah boyd has observed that many people presume the way they use social networks is the way everyone uses them. “I interviewed gay men who thought Friendster was a gay dating site because all they saw were other gay men,” she says. “I interviewed teens who believed that everyone on MySpace was Christian because all of the profiles they saw contained biblical quotes. We all live in our own worlds with people who share our values and, with networked media, it’s often hard to see beyond that.”

Now picture our perspective leaving our own experiences, zooming out and up until we can see how all the different groups are interacting on a worldwide social network. That bird’s-eye view could be both beautiful and horrible if the resolution was clear enough. That’s what a Ramen-eating, ex-Apple engineer named Pete Warden is about to release to the public this week.

Sponsor

This Wednesday, Warden will make Friend, Fan page and name data from hundreds of millions of Facebook users available to the academic research community. It’s a move that Facebook has to have seen coming, a move that many in the data-centric community have been calling on the company itself to do for years, and an event that’s been complicated by Facebook’s recent privacy policy changes, which have muddied the waters of right and wrong but rendered even more data available for outside analysis.

If what people call Web 2.0 was all about creating new technologies that made it easy for everyday people to publish their thoughts, social connections and activities, then the next stage of innovation online may be services like recommendations, self and group awareness, and other features made possible by software developers building on top of the huge mass of data that Web 2.0 made public. It’s a very exciting future, and Warden is about to fire one of the earliest big shots in that direction.

Nerds in Space: Social Graph Analysis For Solving Large-Group Problems

Warden studied Computer Vision in college in the U.K., then got into game development. After moving to L.A., he spent six years building graphics drivers for the original Playstation and the XBox. Then he started his own independent business, where, thankfully, he open-sourced much of his work (something he’s still doing today).

When he found out that starting his own business wasn’t going to work with his immigration status, he was very fortunate to have also caught Apple’s eye with the software he had been releasing to the public. Apple bought his company in order to bring him on board. The proceeds of that small sale are now sustaining his next project after going independent again.

After spending five years at Apple struggling to navigate the maze of people and connections and types of expertise in order to get the information he needed, Warden decided to go independent and build a company that solved exactly that kind of problem. “I can’t think of a better big company to work for, but it was still a big company,” he says. “It was hard to find the right people to talk to, whether for particular expertise or for contacts at external companies.” And so Warden left Apple to build a company that would use social graph analysis to solve problems like that. He called the company Mailana.

We’ve written here a number of times about Mailana’s tool that analyzes the social graph of any Twitter user. Enter the username of someone on Twitter and Mailana will show you which 20 other people the user has exchanged the largest number of reciprocal public @ replies with. Find someone interesting or important? Mailana’s Twitter analyzer will tell you who they most regularly interact with. See, for example, The Inner Circles of 10 Geek Rockstars on Twitter.

Pulling Down the Facebook Social Graph

Now Warden is about to unveil a much larger project along the same vein. For the past six months he’s been crawling public profile pages on Facebook. He now has more than 215 million of them indexed and updated about once a month. When he began he was using the Web crawling service 80legs, but over time he had to build his own crawling infrastructure.

When I talked to him this afternoon, he had already begun uploading 100 GB of user data onto his server to make it available for academic research starting on Wednesday. Warden says he’s removed identifying profile URLs but kept names, locations, Fan page lists and partial Friends lists. All those fields of data are just waiting to be analyzed and cross referenced. That’s one very rich resource.

Yesterday Warden posted some of his own initial observations from the data on his personal blog. Those included:

• In almost every state in the Southern U.S., God is number one most popular Fan page among Facebook users. Among people in the L.A., San Francisco and Nevada regions? “God hardly makes an appearance on the fan pages, but sports aren’t that popular either,” Warden writes. “Michael Jackson is a particular favorite, and San Francisco puts Barack Obama in the top spot.” In the Oregon and Idaho region? Starbucks is number one.
• In the Mormon-influenced areas of Utah and Eastern Idaho, the most popular Fan pages are The Book of Mormon, Glen Beck and the vampire book Twilight, which was authored by a Mormon.
• The bulk of Warden’s posted analysis yesterday was about location networks. People in the western U.S. tend to have Facebook friends all over the country; people in the southern U.S. tend to mostly be friends with people who have remained in the same area.

Taking a Deeper Look

These observations are interesting, but they are only the beginning of what’s possible. Name, location, friends and interests are great data points to analyze. Warden has written a program that will estimate gender as well, based on names. All these data points can be cross-referenced with outside data, too. Members of Facebook’s own staff did this kind of analysis when they compared user last names to U.S. Census data, which allowed them to estimate changes in Facebook’s racial composition over time based on the likelihood of people with particular last names to report a particular racial backgrounds.

“I’m mostly thinking ‘What do I try first?’,” Warden says. “There’s so many interesting ways to slice the data – especially as I’m starting to get changes over time. I’m also trying to map out political networks in aggregate; how polarized the fans of particular politicians are – so how likely a Sarah Palin fan is to have any friends who are fans of Obama, and how that varies with location too. One of my favorite results is that Texans are more likely to be fans of the Dallas Cowboys than God.”

Warden says he hasn’t talked to anyone from Facebook since he started crawling the site, but he did get an email from someone on the security team asking him to take down instructions he’d posted that exposed a security hole that made harvesting peoples’ email addresses easy. So the company is paying attention. “I’d love to see them put me out of business by putting decent data out there,” Warden says. He says his Amazon Web Services bill was over $5,000 last month.

Why is he indexing all this content and why is he going to hand it over to the academic world later this week? “I am fascinated by how we can build tools to understand our world and connect people based on all the data we’re just littering the Internet with,” Warden says.

“Nobody thinks about how much valuable information they’re generating just by friending people and fanning pages. It’s like we’re constantly voting in a hundred different ways every day. And I’m a starry-eyed believer that we’ll be able to change the world for the better using that neglected information. It’s like an x-ray for the whole country – we can see all sorts of hidden details of who we’re friends with, where we live, what we like.”

For a great example of the kind of social impact that data analysis can make, Warden points to some of the fascinating ways that GIS data is illuminating the intersection of race and public services. Data has shed light on social injustices for decades, and measurable information about the interactions of hundreds of millions of people every day on Facebook offers opportunities to discover both good and bad news about the contemporary human condition.

Warden says he’s not yet been able to interest any investors in his ideas for businesses based on this data, so his girlfriend Liz Baumann, a former insurance actuary, stepped in to help and is now running much of the crawling. He says he’s now focused on “working on ways of presenting all this information in a form that answers questions for people willing to pay.” His first experiment along those lines is the very interesting FanPageAnalytics.com.

What does Pete Warden hope for from this week’s public release of all this Facebook data? “Hopefully I’ll get to see a bunch of interesting [academic research] papers come out of it, worst case. And I’d like to be the guy people turn to when they need stuff like this.”

Already well-respected among a fringe group of bleeding-edge geeks, we hope that Warden’s work on social graph analysis will end up impacting a far larger number of people than may ever know his name.

Discuss

As more and more of our friends and favorite organizations start publishing updates online, being able to organize them well is becoming even more important. Niche-popular desktop social media stream-reader Tweetdeck issued a software update this morning and the most striking change is in its handling of user groups. It’s beautiful. The new Tweetdeck is faster, more flexible and easier to navigate.

Groups, we have argued, are the secret weapon of the social Web. Here are five ways that the new Tweetdeck gets groups right, and that Facebook, the world’s dominant social-media-stream reader by a long-shot, could learn from what Tweetdeck is doing. That would drastically improve Facebook’s own user experience.

Sponsor

Internet startup investor John Borthwick of Betaworks has told us that he invested in Tweetdeck specifically because its column metaphor represented a drastic break from the page-based metaphor of the rest of the Web and the Instant Messaging metaphor of most other Twitter clients. That’s how Tweetdeck works: It lets you put your friends and contacts on Twitter, Facebook, MySpace and LinkedIn into grouped columns across your screen. It’s a powerful system and the clear leader in the ecosystem of interfaces built around Twitter. Competitor Seesmic has a similar offering and is also based on columns for groups.

These applications may be more overwhelming than many mainstream users are looking for, but the principles could be adapted to Facebook’s own interface in some very interesting ways.

The Problem With Facebook’s Group Support

Groups of users, which Facebook calls Lists, are extremely helpful in prioritizing messages by their source. They enable users to subscribe to more sources of information in total without fear they will miss high-priority content. Groups help contextualize messages in a stream, and with good search support they can help you target queries and unearth the information you’re looking for within a limited space of trusted, topical sources of information. Last month, Facebook suggested its users subscribe to news organizations on the social network and put those updates in a special list called News, for example.

Unfortunately, Facebook has never treated Groups with the respect that they deserve. The newest redesign pushes friends Lists a click removed from the front page of the site, even. (It took me three clicks from the home page to see the view pictured on the right, for example.) The company is instead focused on serving up content from favored sources using the new News Feed (vs. Live Feed) algorithm. This algorithm says that the more you’ve interacted with a source of information in the past, the more likely you are to want to read that person’s updates in the future. News Feed is a self-reenforcing paradigm that simplifies and narrows a user’s universe by taking editorial control out of their hands and putting it in the hands of a black-box formula.

How could Facebook better handle groups? Let’s take a look at how Tweetdeck does it.

Tweetdeck’s Superior Handling of Groups

• Buttons to navigate directly

  The new Tweetdeck update today added a series of buttons at the bottom of the application for your group columns. Hover over them and you’ll see the column name. Click on that button and you’ll be brought immediately to that column, a much nicer experience than the awkward old scrolling. The hover display does show some information that’s probably not useful to anyone (API calls remaining) but putting rapid group navigation in a good place in the app makes a big difference. It would also be nice if those buttons were configurable in some way – like the red one or #3 is for my @ replies column. But Tweetdeck now makes it much, much easier to navigate between groups than Facebook does. Facebook almost seems to be discouraging use of groups by burying them several clicks below top level options.


• Different Notification Priorities

  Tweetdeck makes it easy to set up different levels of notification for different groups. For example, my “high-priority” Twitter group sends full messages as a pop-up in the corner of my screen when it updates, but my column for people I actually know in real life on Facebook (like my family members) throws a pop-up and an audio notification in Tweetdeck when there’s something new. That’s awesome.

  Facebook is all about the notifications – why not let me get a special notification when someone in a particular group has posted an update?


• Shareable Groups, Suggestions etc.

  For years Twitter has been fundamentally public and Facebook private, but for better or for worse that’s changing on Facebook. Isn’t it time for Facebook to offer bundles of pages or people that you can follow all at once? Twitter and Tweetdeck make this easy with lists, sharable lists and the Tweetdeck directory of recommended lists to follow. Facebook users would love something like that! My college newspaper staff as one big list of Facebook users to follow? My football team? Suzy Bright’s curated list of top sex writers to follow on Facebook? People would eat that kind of stuff up.

• Keyword Filters

  Last week I was working when the season premier of Lost came on TV. I’m likely to watch it later on DVD. Tweetdeck let me add a filter to all of my groups to hide any posts that included the word Lost! Sick of hearing about the iPad? No problem! Tweetdeck does a great job of building value on top of these groups of contacts: filter for, filter out keywords, analyze a group for its most-used words. There are lots of possibilities. Facebook users would probably like these same options.

• A Group for New Friends

  Tweetdeck has a great feature that lets you create a column to display the bios of people who have just started following you. From there you can click once to follow them back. Facebook could easily do something like that, bring new friends and/or recommended friends up higher into a very visible place in the interface. Facebook wants users to go beyond connecting with people they already know in real life (sorry, users, that’s what Facebook has decided) so why not create a group that’s made up of people who are fans of the same pages you are or are otherwise recommended to connect with?

There are all kinds of ways that Facebook could offer meaningful support for user groups and turn the News Feed into a more powerful tool, with more control for users and more value in the long run. Tweetdeck is doing a pretty darned good job of exactly that.

Discuss