How internet companies profit from data on the web
It is not alone. Across the internet economy, companies are compiling masses of data on people, their activities, their likes and dislikes, their relationships with others and even where they are at any particular moment—and keeping mum. For example, Facebook, a social-networking site, tracks the activities of its 400m users, half of whom spend an average of almost an hour on the site every day, but does not talk about what it finds. Google reveals a little but holds back a lot. Even eBay, the online auctioneer, keeps quiet.
“They are uncomfortable bringing so much attention to this because it is at the heart of their competitive advantage,” says Tim O’Reilly, a technology insider and publisher. “Data are the coin of the realm. They have a big lead over other companies that do not ‘get’ this.” As the communications director of one of the web’s biggest sites admits, “we’re not in a position to have an in-depth conversation. It has less to do with sensitive considerations like privacy. Instead, we’re just not ready to tip our hand.” In other words, the firm does not want to reveal valuable trade secrets.
The reticence partly reflects fears about consumer unease and unwelcome attention from regulators. But this is short-sighted, for two reasons. First, politicians and the public are already anxious. The chairman of America’s Federal Trade Commission, Jon Leibowitz, has publicly grumbled that the industry has not been sufficiently forthcoming. Second, if users knew how the data were used, they would probably be more impressed than alarmed.
Where traditional businesses generally collect information about customers from their purchases or from surveys, internet companies have the luxury of being able to gather data from everything that happens on their sites. The biggest websites have long recognised that information itself is their biggest treasure. And it can immediately be put to use in a way that traditional firms cannot match.
Some of the techniques have become widespread. Before deploying a new feature, big sites run controlled experiments to see what works best. Amazon and Netflix, a site that offers films for hire, use a statistical technique called collaborative filtering to make recommendations to users based on what other users like.
The technique they came up with has produced millions of dollars of additional sales. Nearly two-thirds of the film selections by Netflix’s customer come from the referrals made by computer.
EBay, which at first sight looks like nothing more than a neutral platform for commercial exchanges, makes myriad adjustments based on information culled from listing activity, bidding behaviour, pricing trends, search terms and the length of time users look at a page. Every product category is treated as a micro-economy that is actively managed. Lots of searches but few sales for an expensive item may signal unmet demand, so eBay will find a partner to offer sellers insurance to increase listings.
The company that gets the most out of its data is Google. Creating new economic value from unthinkably large amounts of information is its lifeblood. That helps explain why, on inspection, the market capitalisation of the 11-year-old firm, of around $170 billion, is not so outlandish. Google exploits information that is a by-product of user interactions, or data exhaust, which is automatically recycled to improve the service or create an entirely new product.
Vote with your mouse
But although Google’s system was an improvement, it too was open to abuse from “link spam”, created only to dupe the system. The firm’s engineers realised that the solution was staring them in the face: the search results on which users actually clicked and stayed. A Google search might yield 2m pages of results in a quarter of a second, but users often want just one page, and by choosing it they “tell” Google what they are looking for. So the algorithm was rejigged to feed that information back into the service automatically.
From then on Google realised it was in the data-mining business. To put the model in simple economic terms, its search results give away, say, $1 in value, and in return (thanks to the user’s clicks) it gets 1 cent back. When the next user visits, he gets $1.01 of value, and so on. As one employee puts it: “We like learning from large, ‘noisy’ data sets.”
Making improvements on the back of a big data set is not a Google monopoly, nor is the technique new. One of the most striking examples dates from the mid-1800s, when Matthew Fontaine Maury of the American navy had the idea of aggregating nautical logs from ships crossing the Pacific to find the routes that offered the best winds and currents. He created an early variant of a “viral” social network, rewarding captains who submitted their logbooks with a copy of his maps. But the process was slow and laborious.
Wizard spelling
Google applies this principle of recursively learning from the data to many of its services, including the humble spell-check, for which it used a pioneering method that produced perhaps the world’s best spell-checker in almost every language. Microsoft says it spent several million dollars over 20 years to develop a robust spell-checker for its word-processing program. But Google got its raw material free: its program is based on all the misspellings that users type into a search window and then “correct” by clicking on the right result. With almost 3 billion queries a day, those results soon mount up. Other search engines in the 1990s had the chance to do the same, but did not pursue it. Around 2000 Yahoo! saw the potential, but nothing came of the idea. It was Google that recognised the gold dust in the detritus of its interactions with its users and took the trouble to collect it up.
Two newer Google services take the same approach: translation and voice recognition. Both have been big stumbling blocks for computer scientists working on artificial intelligence. For over four decades the boffins tried to program computers to “understand” the structure and phonetics of language. This meant defining rules such as where nouns and verbs go in a sentence, which are the correct tenses and so on. All the exceptions to the rules needed to be programmed in too. Google, by contrast, saw it as a big maths problem that could be solved with a lot of data and processing power—and came up with something very useful.
For translation, the company was able to draw on its other services.
Its search system had copies of European Commission documents, which are
translated into around 20 languages. Its book-scanning project has
thousands of titles that have been translated into many languages. All
these translations are very good, done by experts to exacting standards.
So instead of trying to teach its computers the rules of a language,
Google turned them loose on the texts to make statistical inferences.
Google Translate now covers more than 50 languages, according to Franz
Och, one of the company’s engineers. The system identifies which word or
phrase in one language is the most likely equivalent in a second
language. If direct translations are not available (say, Hindi to
Catalan), then English is used as a bridge.
Google was not the first to try this method. In the early 1990s IBM
tried to build a French-English program using translations from Canada’s
Parliament. But the system did not work well and the project was
abandoned. IBM had only a few million documents at its disposal, says Mr
Och dismissively. Google has billions. The system was first developed
by processing almost 2 trillion words. But although it learns from a big
body of data, it lacks the recursive qualities of spell-check and
search.
The design of the feedback loop is critical. Google asks users for
their opinions, but not much else. A translation start-up in Germany
called Linguee is trying something different: it presents users with
snippets of possible translations and asks them to click on the best.
That provides feedback on which version is the most accurate. Voice recognition highlights the importance of making use of data exhaust. To use Google’s telephone directory or audio car navigation service, customers dial the relevant number and say what they are looking for. The system repeats the information; when the customer confirms it, or repeats the query, the system develops a record of the different ways the target word can be spoken. It does not learn to understand voice; it computes probabilities.
To launch the service Google needed an existing voice-recognition system, so it licensed software from Nuance, a leader in the field. But Google itself keeps the data from voice queries, and its voice-recognition system may end up performing better than Nuance’s—which is now trying to get access to lots more data by partnering with everyone in sight.
Re-using data represents a new model for how computing is done, says Edward Felten of Princeton University. “Looking at large data sets and making inferences about what goes together is advancing more rapidly than expected. ‘Understanding’ turns out to be overrated, and statistical analysis goes a lot of the way.” Many internet companies now see things the same way. Facebook regularly examines its huge databases to boost usage. It found that the best single predictor of whether members would contribute to the site was seeing that their friends had been active on it, so it took to sending members information about what their friends had been up to online. Zynga, an online games company, tracks its 100m unique players each month to improve its games.
“If there are user-generated data to be had, then we can build much better systems than just trying to improve the algorithms,” says Andreas Weigend, a former chief scientist at Amazon who is now at Stanford University. Marc Andreessen, a venture capitalist who sits on numerous boards and was one of the founders of Netscape, the web’s first commercial browser, thinks that “these new companies have built a culture, and the processes and the technology to deal with large amounts of data, that traditional companies simply don’t have.”
Recycling data exhaust is a common theme in the myriad projects going on in Google’s empire and helps explain why almost all of them are labelled as a “beta” or early test version: they truly are in continuous development. A service that lets Google users store medical records might also allow the company to spot valuable patterns about diseases and treatments. A service where users can monitor their use of electricity, device by device, provides rich information on energy consumption. It could become the world’s best database of household appliances and consumer electronics—and even foresee breakdowns. The aggregated search queries, which the company makes available free, are used as remarkably accurate predictors for everything from retail sales to flu outbreaks.
Together, all this is in line with the company’s audacious mission to “organise the world’s information”. Yet the words are carefully chosen: Google does not need to own the data. Usually all it wants is to have access to them (and see that its rivals do not). In an initiative called “Data Liberation Front” that quietly began last September, Google is planning to rejig all its services so that users can discontinue them very easily and take their data with them. In an industry built on locking in the customer, the company says it wants to reduce the “barriers to exit”. That should help save its engineers from complacency, the curse of many a tech champion. The project might stall if it started to hurt the business. But perhaps Google reckons that users will be more inclined to share their information with it if they know that they can easily take it back.
