The Story of Search: How Google beat Overture and Yahoo by backing the long tail
Gary Flake, Microsoft technical fellow and director of Microsoft Live Labs, first became renowned for failure.
In 2003, he joined Overture as chief science officer. At the time, as he reminded an audience at the 2009 Advertising Research Foundation's 2009 Annual "Re:think" conference, the search-engine business largely was a duopoly. In fact, a year later, Overture had a 55-percent market share, Google 35 percent, and a variety of other providers shared the remaining 10-percent.
Five years on, Flake said, "the pie had grown by a factor of four. And it had changed from a duopoly to a monopoly." In 2008, Google's market share was 80 percent; Yahoo, which had acquired Overture in 2003, had 15 percent and Microsoft rounded out the selection with five percent.
"Google's dominance almost didn't happen," Flake told the ARF audience. And, the drivers were as much Overture's failure to understand the market dynamics as they were Google's successful understanding of the search value proposition. From a personal perspective, he added, such a momentous change in just a half-decade led to two questions: " 'WTF?' or 'How did I lose so badly - with greatness in my grasp - and snatch defeat from the jaws of victory?'"
"Google's dominance almost didn't happen," Flake told the ARF audience. And, the drivers were as much Overture's failure to understand the market dynamics as they were Google's successful understanding of the search value proposition. From a personal perspective, he added, such a momentous change in just a half-decade led to two questions: " 'WTF?' or 'How did I lose so badly - with greatness in my grasp - and snatch defeat from the jaws of victory?'"
Although acknowledging he is a technologist and only a "tourist" in the marketing world, Flake began answering both queries by acknowledging the need to address three different constituencies:
*Customers: "They only want what they want."
*Customers: "They only want what they want."
*Advertisers: "They want low cost and low risk."
*Media/Publishers: "They need to engage customers and they want to do so at a low cost and with low risk."
"For each to get what it wants, someone has to sacrifice. If a publisher wants to make more money, an advertiser has to pay more. If an advertiser wants lower risk and still get out in front of customers, the customers may not get what they want."
In the case of paid search, a customer types in a query; advertisers, in advance, bid on a click because they presume a click translates to interest; and, with each click, publishers presumably make money. "If the interests of all three partiers are aligned, new value is created to all parties. It's something all three want: Something is exchanged at a pricing that's market-determined."
When GoTo.com - the original name of Overture - was founded in 1988, "There were valid questions about the model", Flake said. Would users actually be willing to pay up on a sponsored search result? Up to that point, search had been almost entirely non-commercial. Would destinations show these server-sponsored ads, when so much emphasis had been placed on preserving an editorial voice on search-engine pages? Would advertisers be willing to take the risk of a new medium that was completely new with no demonstrated return on investment?
The hesitation all added up to "a serious cold-start problem." The solution, Flake offered, was to make paid search completely transparent.
Overture tried to engender confidence with three strategic platforms: Exact search meant that users would get exactly what they wanted. Type in "Flowers" and you'd get flowers. Type in "Flowers" and "Mother's Day" and you'd get a list of sites that specifically matched those criteria. "But if you typed in, 'Where can I buy flowers for my beautiful mother in San Jose,' - and, I kid you not, we received long verbose proposals like that - you'd likely not get anything."
The reasoning, Flake explained, was that "we felt that the thoughtful advertiser wanted to know precisely what they were getting." Three other Overture features that reinforced the concept of transparency: A "human" editorial filter that reviewed every ad, "high-touch relationships with advertisers through all parts of the workflow," and partnerships with such premium destinations as MSN, AOL, Yahoo, and Microsoft.
In the case of paid search, a customer types in a query; advertisers, in advance, bid on a click because they presume a click translates to interest; and, with each click, publishers presumably make money. "If the interests of all three partiers are aligned, new value is created to all parties. It's something all three want: Something is exchanged at a pricing that's market-determined."
When GoTo.com - the original name of Overture - was founded in 1988, "There were valid questions about the model", Flake said. Would users actually be willing to pay up on a sponsored search result? Up to that point, search had been almost entirely non-commercial. Would destinations show these server-sponsored ads, when so much emphasis had been placed on preserving an editorial voice on search-engine pages? Would advertisers be willing to take the risk of a new medium that was completely new with no demonstrated return on investment?
The hesitation all added up to "a serious cold-start problem." The solution, Flake offered, was to make paid search completely transparent.
Overture tried to engender confidence with three strategic platforms: Exact search meant that users would get exactly what they wanted. Type in "Flowers" and you'd get flowers. Type in "Flowers" and "Mother's Day" and you'd get a list of sites that specifically matched those criteria. "But if you typed in, 'Where can I buy flowers for my beautiful mother in San Jose,' - and, I kid you not, we received long verbose proposals like that - you'd likely not get anything."
The reasoning, Flake explained, was that "we felt that the thoughtful advertiser wanted to know precisely what they were getting." Three other Overture features that reinforced the concept of transparency: A "human" editorial filter that reviewed every ad, "high-touch relationships with advertisers through all parts of the workflow," and partnerships with such premium destinations as MSN, AOL, Yahoo, and Microsoft.
To explain how Overture went wrong, Flake used three models of the new digital word that "that might explain the past and look toward the future":
- The Long Tail
- The Innovator's Dilemma
- Network Effects
The Long Tail
Before either organization fully realized the model or its implications, Overture focused on the head of the long-tail model and Google concentrated on the tail. And, as Flake described, those orientations would be the paid-search market-mover.
From day 1, Google defaulted to the approximate match. "'Where can I buy flowers for my beautiful mother in San Jose" generated a bunch of responses - florists in the specific market, 1-800 order-by-phone services, even grocery stores that offered plants as a sideline ordering. There were no specific matches (the head), but scores of approximate matches (the tail) that better served the needs of the consumer.
Similarly, Google used automated click-through rates (CTR) instead of staff people to determine whether a search was relevant to a query. If a search seemed to work, it was kept. If not, it was rejected from the system.
A CTR filter also served as a proxy for the relevancy that the destination partners had provided for Overture.
The pattern was not an isolated one. Flake pointed to other instances of head/tail distinctions that have become more common as the model has become better understood: mainstream media (head) and news aggregators/citizen journalism (tail); network TV (head) and "stupid YouTube videos (tail); radio (head) and podcasts (tail); RIAA (head) and unsigned artists (tail); shrink-wrap software (head) and software mashups (tail).
The Innovator's Dilemma
"The first companies in an industry (the innovators) must be willing to eventually destroy their own business to create something new," Flake told the ARF assembly. "They must destroy their business before someone else does."
Flake's career began in the hardware industry, "where the epitome was to program on a supercomputer." Although "it took decades to unfold," priorities for hardware moved from supercomputers to main frames to scientific workstations to personal computers to laptops to handhelds and to cell phones…. Comsumerization of that market actually drove innovation and drove the bigger things out of business."
"Innovators start off by doing something very natural," Flake said. "They focus on a small number of large, high-margin customers. They want to make money, they want to prove the model works as soon as possible. And they want to maximize their own ROI."
Late arrivals, by contrast, are left to focus on lower-margin customers - again, a common-sense strategy: Why go head-to-head with the market leader when there's a whole pool of customers for whom they don't have to compete?
"Meanwhile, through competition, margins begin to shrink," Flake continued. "Both the original innovator and the newcomer invade each other's space, looking for more business."
The difference in their histories, however, begins to reveal different strengths: The older companies - the original innovators - have not had to learn to grow up looking for more efficient ways to do business. The younger companies have a heritage of going head-to-head with competitors, of scaling up, on learning new ways to operate more efficiently. "And the late arrivals win because they can take the lessons of optimization" they've learned working on the tail and apply them to the head, Flake said.
In the paid-search business, the cycle of evolution took just 18 months to unwind. And, the change happened as - naturally enough - the principal players tried to move from their position of strength to the areas where they still could grow. As paid search matured, he added, the industry survivors naturally try to capture a fuller market share by moving to the opposite end of the long tail. Yahoo, which had purchased Overture in 2002, tried to expand its market from the head to the tail; Google attempted to move from the tail to the head.
But, as Yahoo discovered, it was much easier to move from the tail to the head than from the head to the tail.
Network Effects
"If you're the only person in the world with a telephone, it doesn't have much use to you," Flake told the ARF audience. "If everyone else in the world has a telephone, it has great use to you because potentially you can call anyone."
Any kind of network that has more participants simply provides both greater individual value and greater aggregate value, Flake continued. And, as networks grow, "virtual cycles emerge."
In an eBay network, he explained, the more buyers there are, the more opportunities there are to sell. And the more sellers who participate, the more opportunities there are to buy. It's a model that's replicated in operating systems (developers and users), file formats (writers and readers) and search engines (authors and searchers), marketing (advertisers and consumers), and payments (payers and payees).
"In the virtuous cycle of paid search," the director of Microsoft's Live Labs added, "You need advertisers. The more advertisers you have, the more bids you have. The more bids you have, the more traffic you have. The more traffic you have, the more money you get per search. And, with the more money you get, the more syndication you get. And, as you get more syndication, you get more traffic. And it's traffic inventory that pulls in the advertisers and the process begins to snowball."
Overture - and, in time, Yahoo - operated independently and allowing the cycle to develop "organically," said Flake. Google, by contrast, "primed the pump with a destination site that could effectively make them as powerful as any affiliate on the network. And, in doing so, they were able to bootstrap their own network in a way that was quite stunning."
Overture, he explained, "did not understand that one network could prime another…. We were constrained by our own idea, by our focus on the head [of the long tail]. We didn't understand how it all could play out so rapidly."
The future plays out with "an even longer tail" and as "tools become simpler, more powerful and more prevalent, the pools of creators will increase dramatically." Everyday examples include desktop publishing, digital photography, garage bands, Songsmith, podcasting, and blogging. "What does it take to make an online business?" Flake asked. "Ten years ago, you needed a substantial amount of money. Today, it's $5 or for free…. The barriers to entry are dropping to zero."
And, as the opportunities proliferate, so will the occasions grow that enable additional long-tail partnerships - pools of intelligence that can overlap with (and reinforce) one another.
From day 1, Google defaulted to the approximate match. "'Where can I buy flowers for my beautiful mother in San Jose" generated a bunch of responses - florists in the specific market, 1-800 order-by-phone services, even grocery stores that offered plants as a sideline ordering. There were no specific matches (the head), but scores of approximate matches (the tail) that better served the needs of the consumer.
Similarly, Google used automated click-through rates (CTR) instead of staff people to determine whether a search was relevant to a query. If a search seemed to work, it was kept. If not, it was rejected from the system.
A CTR filter also served as a proxy for the relevancy that the destination partners had provided for Overture.
The pattern was not an isolated one. Flake pointed to other instances of head/tail distinctions that have become more common as the model has become better understood: mainstream media (head) and news aggregators/citizen journalism (tail); network TV (head) and "stupid YouTube videos (tail); radio (head) and podcasts (tail); RIAA (head) and unsigned artists (tail); shrink-wrap software (head) and software mashups (tail).
The Innovator's Dilemma
"The first companies in an industry (the innovators) must be willing to eventually destroy their own business to create something new," Flake told the ARF assembly. "They must destroy their business before someone else does."
Flake's career began in the hardware industry, "where the epitome was to program on a supercomputer." Although "it took decades to unfold," priorities for hardware moved from supercomputers to main frames to scientific workstations to personal computers to laptops to handhelds and to cell phones…. Comsumerization of that market actually drove innovation and drove the bigger things out of business."
"Innovators start off by doing something very natural," Flake said. "They focus on a small number of large, high-margin customers. They want to make money, they want to prove the model works as soon as possible. And they want to maximize their own ROI."
Late arrivals, by contrast, are left to focus on lower-margin customers - again, a common-sense strategy: Why go head-to-head with the market leader when there's a whole pool of customers for whom they don't have to compete?
"Meanwhile, through competition, margins begin to shrink," Flake continued. "Both the original innovator and the newcomer invade each other's space, looking for more business."
The difference in their histories, however, begins to reveal different strengths: The older companies - the original innovators - have not had to learn to grow up looking for more efficient ways to do business. The younger companies have a heritage of going head-to-head with competitors, of scaling up, on learning new ways to operate more efficiently. "And the late arrivals win because they can take the lessons of optimization" they've learned working on the tail and apply them to the head, Flake said.
In the paid-search business, the cycle of evolution took just 18 months to unwind. And, the change happened as - naturally enough - the principal players tried to move from their position of strength to the areas where they still could grow. As paid search matured, he added, the industry survivors naturally try to capture a fuller market share by moving to the opposite end of the long tail. Yahoo, which had purchased Overture in 2002, tried to expand its market from the head to the tail; Google attempted to move from the tail to the head.
But, as Yahoo discovered, it was much easier to move from the tail to the head than from the head to the tail.
Network Effects
"If you're the only person in the world with a telephone, it doesn't have much use to you," Flake told the ARF audience. "If everyone else in the world has a telephone, it has great use to you because potentially you can call anyone."
Any kind of network that has more participants simply provides both greater individual value and greater aggregate value, Flake continued. And, as networks grow, "virtual cycles emerge."
In an eBay network, he explained, the more buyers there are, the more opportunities there are to sell. And the more sellers who participate, the more opportunities there are to buy. It's a model that's replicated in operating systems (developers and users), file formats (writers and readers) and search engines (authors and searchers), marketing (advertisers and consumers), and payments (payers and payees).
"In the virtuous cycle of paid search," the director of Microsoft's Live Labs added, "You need advertisers. The more advertisers you have, the more bids you have. The more bids you have, the more traffic you have. The more traffic you have, the more money you get per search. And, with the more money you get, the more syndication you get. And, as you get more syndication, you get more traffic. And it's traffic inventory that pulls in the advertisers and the process begins to snowball."
Overture - and, in time, Yahoo - operated independently and allowing the cycle to develop "organically," said Flake. Google, by contrast, "primed the pump with a destination site that could effectively make them as powerful as any affiliate on the network. And, in doing so, they were able to bootstrap their own network in a way that was quite stunning."
Overture, he explained, "did not understand that one network could prime another…. We were constrained by our own idea, by our focus on the head [of the long tail]. We didn't understand how it all could play out so rapidly."
The future plays out with "an even longer tail" and as "tools become simpler, more powerful and more prevalent, the pools of creators will increase dramatically." Everyday examples include desktop publishing, digital photography, garage bands, Songsmith, podcasting, and blogging. "What does it take to make an online business?" Flake asked. "Ten years ago, you needed a substantial amount of money. Today, it's $5 or for free…. The barriers to entry are dropping to zero."
And, as the opportunities proliferate, so will the occasions grow that enable additional long-tail partnerships - pools of intelligence that can overlap with (and reinforce) one another.
