Excerpted with permission fromSearch: How the Data Explosion Makes Us Smarter, by Stefan Weitz, a senior director of search at Microsoft. Available from Bibliomotion, Inc. Copyright © 2014.
Search, in its enlightened form, has the potential to be the hinge that finally connects humanity with machines in a way that lets us transcend our biological limitations. We are at a point in technical history where we have the pieces necessary to augment human abilities in ways that will give superpowers to those of us with access to these systems. The capacity to know anything that can be known and do anything anywhere in the world, no matter where your physical body lies, is as close to a reality as it has ever been.
Machine Learning and Intelligence
Machine learning—an area of AI focusing on systems that “learn” from data in order to navigate future similar scenarios—is one of the ways we’ve managed to give systems like search the ability to make sense of our analog world. Many of the seemingly magical experiences we have with technology, such as the incredible advances in speech recognition and the more personal interactions we have with retail sites like Amazon, come from this field of study. Machine learning, with its focus on computers’ ability to learn without explicit programming, is integral to bridging the gap between humans and search systems.
Think of machine learning in relation to Pandora. When Tim Westergren started the personalized Internet radio service, he employed hundreds of people to listen to music and identify its “features.” According to Westergren, the Music Genome Project was an effort to understand music at its most fundamental level by using several hundred attributes that could describe songs.
The group decided to classify music into a number of “genomes”: Pop/Rock, Hip-Hop/Electronica, Jazz, World Music, and Classical. Each piece analyzed by one of the companies’ genomists would be classified according to nearly four hundred characteristics. They called these characteristics (what we call “features” in search) “genes.” In the field of genetics, genes are the sequences of nucleotides that determine whether our eyes are blue, our hair is brown, and whether we tan; they express our inherited makeup. The Music Genome Project focused its energy on classifying what “made up” each song—they were looking for the determining characteristics: Was there a syncopated rhythm? Were there harmonies in the chorus? Did the piece have a string flourish? Were there strong female vocals?
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
Each song was analyzed by a human and catalogued according to these sorts of traits, with the result that each of the more than three hundred thousand songs in the index had hundreds of descriptions given in granular detail. Moreover, songs weren’t catalogued by just one person—the project aimed for constant quality control through analysis by multiple listeners, and the vocabulary and training each genomist received ensured consistency in the musical analysis.
In the case of the Musical Genome Project, humans applied digital attributes to the analog world. They took an analog source—music tracks—and analyzed them to derive “features” of the music. This is what we call “training data,” and it is at the heart of a type of machine learning. To understand machine learning, imagine a black box with two sets of data on either side of it. On the left side we have a set of seemingly random data, and on the right side we have what the “sorted” data would look like. So, for example, on the left we might have weather statistics for every day of the past year. On the right, we could have the batting average of every hitter in baseball, by day, over the same year. In supervised machine learning, the system figures out the patterns in a set of data—say, discovering that certain hitters perform better on sunny days (even if they’re indoors!). The machine finds a pattern that joins the inputs from the left side of the box and outputs on the right side of the box. It then uses this pattern to make future predictions for similar scenarios.
The process gets even more interesting as you add more features and begin to add weights to the different variables. Imagine that you have information like “average temperatures for June in Seattle” and a fact like “In June, robberies in Seattle are 1.2 times the yearly average.” The system is then left to analyze these facts and predict an outcome. In this case, the system might determine that when temperature rises above 68 degrees in Seattle, robberies are likely to increase. As you add more variables (zip code, time of day), you could begin to build a probabilistic model for when a robbery would occur at your house.
In the case of Pandora, machine learning is used to fine-tune the stations that listeners select. If a person likes a particular song, and Pandora knows all four hundred characteristics of that song, it can predict with some accuracy that the listener will like songs that share those characteristics. When users interact with the system—giving a thumbs-up or thumbs-down to particular songs—the system can further refine its predictions. Maybe a song that Pandora thought I would like but didn’t had a higher weighting of the brass instruments gene—knowing that I didn’t like that song allows Pandora to rebuild its model for me, and lets that station make predictions for songs that don’t have as much brass.
Other examples abound. IBM’s Watson, which famously trounced human opponents in the Jeopardy game show, has had its capabilities extended to recipe analysis. It turns out that just 10 ingredients could yield a trillion or quadrillion possible permutations, especially when you take into account amounts of particular ingredients, like a quarter versus a half teaspoon of red pepper flakes. Watson has now been trained to predict whether a recipe will be pleasant or terrible, surprising or familiar, and it even understands good pairings for particular dishes. This isn’t about having a machine with 10 thousand recipes in an index, rather it’s about Watson having enough training data to begin to understand, much like Pandora does, what will generate a result that meets certain criteria. The complex interplay of factors in a successful recipe is stunning when one stops to think about it. Cooking time, temperature, the interactions of salts and acids: all these elements must be measured and combined in a way that produces a delicious dish. Some have even claimed this new Watson capacity as the first true creativity shown by a machine.
What about People as a Search Engine?
Beyond more advanced technology, there is an interesting interim step in applying human characteristics to intelligent systems. So far, we have talked about how search can augment our existence by giving us information and helping us with tasks in our lives. But what if it was the other way around? What if people augmented search? Adding humans to the mix can help systems deal with unfamiliar situations and resolve the impedance mismatch.
What if we could fuse human and search capacities to deliver a better overall experience? Edith Law’s work on training systems to better handle complex queries is seminal in this space. She looked at what happens when a person queries a search engine about a high-level task such as, “How do I help a family member with ALS?” Today, the engine would return many pages with potentially helpful articles from a variety of sources, but that likely isn’t what you, a novice searcher with respect to ALS, would find most helpful. You would want to understand what steps to take to help your family member and in what order, who you might talk to, and how you might best handle the personal affairs of the loved one.
What if the system could divide that initial, ambiguous question into pieces and figure out ways to answer each piece? The search system could handle the rote gathering of definitions, names of medical specialists, and lists of drugs, while humans could be asked the softer questions such as, “When can I expect my father to stop being able to eat on his own?” or “How can I help my mother deal with the impact of the disease on my dad?” Then the search engine would recompose the steps or assemble plans to form a coherent response.
Indeed, leveraging individuals’ abilities to both solve problems and route information is a key component of the strategy of the team that placed first in the DARPA Network Challenge. The challenge was to find ten moored weather balloons scattered across the United States. You can imagine the near impossibility of 10 individuals finding these balloons on their own—indeed, the contest was designed to see how human networks would form to address the challenge. How would one person who saw a red balloon in her backyard alert another person who was looking for it? How would all the reports be reconciled, and how would they be validated? The winning team introduced an incentive mechanism that encouraged individuals to look for balloons and to let their friends know about the task. Rather than rewarding only the person who found the balloon, they rewarded those who invited the person who found the balloon, too. In other words, their system rewarded not just the task completer but the interim people who made it possible.