Google Faces an Innovator's Dilemma with ChatGPT and Conversational Search
Google execs discuss their response to ChatGPT and new search competitors
Among all of the stories around ChatGPT, two questions frequently popped up:
Could ChatGPT be the model for a better search engine?
Is Google falling behind on the next phase of search technology?
A little-covered story from CNBC in December hinted at Google’s rationale for its response or non-response to the ChatGPT phenomenon. The summary bullet points from the article were:
Google employees asked executives at an all-hands meeting whether the AI chatbot that’s going viral represents a “missed opportunity” for the company.
Google’s Jeff Dean said the company has much more “reputational risk” in providing wrong information and thus is moving “more conservatively than a small startup.”
CEO Sundar Pichai suggested that the company has chat products underway for 2023.
There has been a lot of discussion about whether Google should be concerned about the rise of ChatGPT, You.com, and other conversational search products. On one side, you have a group that rightly claims ChatGPT and similar offerings are so frequently wrong you cannot rely on them without verification. They also make a conceptual argument with far weaker evidence that Google has nothing to fear.
On the other side, you have people saying this is the end of Google without any substantial evidence other than their observation from testing ChatGPT. Then there are the people that say they are not giving up Google today, but ChatGPT just showed us what the future of search looks like.
The intensity of this conversation is likely to heat up in 2023. Microsoft is expected to integrate ChatGPT technology into its Bing search engine. Startups are securing funding to target this opportunity. And Google says it will have a new product in this category later this year.
Since I have answered these questions and responded to comments so many times on LinkedIn, I thought it would make sense to provide a more in-depth assessment all in one place. This also provides an opportunity to evaluate Google’s likely response and the factors driving its decisions.
Conversational Search
ChatGPT seemed to be so good at providing answers that many people immediately saw it as superior to their Google search experience. There are eight key differences between the dominant web search experience today and what ChatGPT provides. The first four are negatives, and four are positive.
Information is not current
Responses are sometimes inaccurate or wholly fabricated
Responses are all presented as fact without modifiers
There are no sources provided to check on the veracity of the statements
The responses include more detail and better answers when they are correct
The responses can answer questions that have not been directly addressed in that form previously on the web
The responses can include answers to abstract questions
The conversational interface enables you to refine your search to gather more information or redirect the question if the initial intent understanding is incorrect
The Bad
For example, on the first point, ChatGPT does not have current access to the internet, so all of the information is about 16 months out-of-date. If you are asking questions about something that is from before that time, you may be okay. Otherwise, you are out of luck. You.com, a semantic search engine, does have current access to the internet to address that gap and has conversational search.
However, there is a second issue that these Google alternatives readily admit. ChatGPT and other conversational search solutions we have tested often provide inaccurate and even wholly fabricated results. Sometimes part of the answer is true, and some details are wrong. Google built its business on better, more accurate results than competitors. The new competitors are not close to matching Google on this parameter.
The third issue is how the responses are presented. Whether the details are true or false, they are presented as confidently and eloquently as a university professor. There is concern that this will lead to people relying on incorrect information.
Compounding the truthfulness issue, no sources are ever provided, which makes it harder to verify the statements. YouChat from You.com does provide links sometimes to sources, but a significant amount of the time, they are dead links. At least with Google search’s ten blue links, the user can see what sources are making a claim and assess their reliability. Even with Google answer boxes and knowledge panels, you typically have a link to see where some or all of the data comes from.
The Good
Beyond the negatives, there are several positives to ChatGPT and other conversational search providers. The answers are sometimes extremely good. They provide more detail than typical Google search results and deliver it in a way that is easy to understand. It is like getting an answer from a domain expert. You also don’t need to click a link and search for the answer on the page that Google suggests, so that saves time.
ChatGPT, like InstructGPT before it, can also answer questions that are not directly addressed in existing internet content. For example, what if you are asking a question that requires you to consider information from multiple sources to answer it completely? ChatGPT can synthesize the information and produce a novel answer that was not previously published on a webpage.
The ability to answer abstract questions is closely related to this. Answering abstract questions often requires the synthesis of knowledge from multiple sources. Google can provide you a link to what it deems as a reliable source that has addressed the question but cannot answer it or offer some sort of novel response based on a corpus of expert commentary. ChatGPT can.
Finally, ChatGPT enables true conversational search. That means you can ask a question and then refine your questions about it in subsequent questions, and ChatGPT will remember what was asked and answered already in the conversation and build upon it. This is particularly helpful when you are looking for additional detail beyond the initial response or if the natural language understanding (NLU) misunderstood your intent.
What is Conversational Search Again?
Google characterized conversational search in a November 2020 blog post, saying:
When you’re having a conversation with someone, you might ask multiple questions in a row on the same topic. Wouldn’t it be weird if, in between every question, the person forgot what you were talking about? Well until recently, that’s kind of what would happen on Search.
This blog post introduced Google’s conversational search product. However, its definition of conversational search is not a chat conversation between a human and an AI answer engine. Google defines conversational search as maintaining context from a search session and providing additional information or prioritizing results based on a specific domain it has identified.
This isn’t really conversational. The user is not having a conversation as we would normally define it. There is no back-and-forth exchange. Instead, Google is doing a lot of contextual search refinement at the session and user levels. This is valuable and does often provide a better search experience. But, it is also misleading to call it conversational.
Now that a lot of people have experienced a true conversational search experience and like it, what is Google going to do?
The Innovator’s Dilemma
The Innovator’s Dilemma is a 1997 book by Clayton Christensen that describes why established market leaders often struggle to adjust when new technologies or behaviors begin to take hold in their industries. Despite having more knowledge and resources, they often cede market share to smaller competitors and sometimes lose their entire market advantage over time.
I’ve had the opportunity to teach this concept to business leaders over the past 20 years and had the chance to speak with Christensen shortly after his book was published. His central thesis is that market leaders are typically focused on maximizing profits from their current product portfolio and are unwilling to risk cannibalizing their existing products or alienating customers with a new or cheaper alternative. This means that new competitors, often startups, have the opportunity to leverage disruptive innovations to establish a beachhead in the market and eventually surpass the incumbent leaders.
Incumbent market leaders have assets. They loathe undermining the value of those assets. Some of those assets are financial, others are intellectual property, others are customer and business relationships, and some are reputational. This means that incumbents believe they have less latitude when responding to market changes because they must balance the new opportunity without impairing the value they have already created.
Google’s Innovator’s Dilemma
Consider the response to ChatGPT’s rise to an employee forum by Google’s AI chief Jeff Dean. CNBC reports, “Dean told employees, emphasizing that the company has much more ‘reputational risk’ and is moving ‘more conservatively than a small startup.’” Dean also said, “This really strikes a need that people seem to have, but it’s also important to realize these models have certain types of issues.” He’s referring to the first four points on the list above and maybe a few more. Google surely know a lot about this subject matter.
“We are absolutely looking to get these things out into real products and into things that are more prominently featuring the language model rather than under the covers, which is where we’ve been using them to date. But, it’s super important we get this right,” said Dean.
The “reputational risk” for Google is true. It has a reputation for being the search engine with the highest quality results. If a Google conversational search chat starts spitting out inaccuracies, consumers may start to question whether the core Google search product also has issues.
On top of this, Google has created a very profitable ad business on the back of its current search model. It is not clear if its current ad model would carry over directly to a conversational search model, but it likely would need to change, and maybe significantly. Financial implications invariably receive a lot of scrutiny and can further delay the introduction of new products that threaten the cash cow.
Google does have its own large language models, such as LaMDA and PaLM. LaMDA is available in a closed beta release. In the ChatGPT analysis in December in Synthedia, I compared it to LaMDA. There are some significant differences, most notably that there are only three narrow use cases you can employ LaMDA with today. That means a true comparison is hard to construct. In addition, it could be PaLM that ultimately powers Google’s answer to ChatGPT, and it is not available to the public. The key point is that Google is not starting from scratch.
Some market leaders identified as facing the innovator’s dilemma also had access to the new disruptive technology that ultimately undermined their market dominance. However, they either delayed or refused to introduce it to the market in an effort to protect their existing business or assets. This suggests Google is not facing an unusual situation overall. It is just new for its search business.
Google CEO Sundar Pichai suggested that the company will have a chat-based search product in 2023, according to CNBC. The question is what it will include and how it stacks up to a new expectations benchmark set by ChatGPT.
But Wait, There are All of These Problems
Let’s face it. There are a lot of so-called “disruptive technologies.” Many spark interest but never go anywhere because they face technical or market adoption problems. So, it’s not always a bad strategy for the incumbent leaders to take a wait-and-see approach and move quickly when they see the market changing.
The real question is whether conversational search is likely to become a disruptive innovation that reshuffles consumer behaviors and preferences. I think it will.
The question is the timing. I suspect Google has time, as there are no competitors that are wrapping up actual market share at the moment. The key risk for Google is that OpenAI has established strong mindshare around what conversational search should be, and that could become a longer-term threat to Google’s market dominance.
In the meantime, there are all of these problems, as enumerated above. Don’t those create a risk that this technology will never go anywhere beyond the imaginations of a few techno-optimists? Well, there are always problems.
Consider the case of mobile phones. I was doing work for AT&T when the mobile, pre-smartphone revolution was in full force. AT&T was not without competition in landline and long-distance telephony, but it certainly was a giant in terms of market share and scope of services.
Company executives thought that mobile phones would be niche products because they had a much lower quality of service. Calls were dropped frequently, and the quality of the connections was not on par with landline phones. In addition, they assumed that offering an unlimited long-distance phone plan at a flat rate was a recipe for financial ruin for the start-ups in the space.
They were colossally wrong on both where the market would wind up and when the changes would take hold. In many ways, mobile telephony was inferior based on the criteria of the landline model and consumer expectations that built up over the years. It did offer, however, a new feature: the ability to make or receive a call from anywhere. That novel concept had a lot of utility and led to a wholesale change in behaviors, expectations, and priorities.
The problems we see around conversational search as it is manifested today are technical and discrete in scope. These seem like solvable engineering challenges if there is market demand for the product, which seems likely.
To carry the mobility analogy further, we might look back in a decade and say, “mobility and search were both gigantic changes that were further amplified by the new applications enabled by the evolutions to smartphones and conversational search, respectively.”
New Technology Breeds New Ideas
Conversational search today is not anywhere near the significance of mobility or smartphones and may never reach that level of impact. However, for the first time in many years, there is now an alternative vision to Google’s model of search, and many of us believe it could be very useful. I am not expecting the Google search model to disappear entirely. It will persist and remain dominant for some time.
At the same time, it seems inevitable that true conversational search will be a complementary feature to the 10 blue links model, and this may occur even before the technical deficiencies are completely resolved. Google may also come to lead this new search market segment and retain its unrivaled search dominance. Technology disruptions create opportunities for new entrants, but they don’t predict who will win. The question today is how well Google will navigate its innovator’s dilemma.
Another downside to consider: ChatGPT is computationally expensive. OpenAI has struggled to keep it online as its popularity has grown, and there have been reports that the cost of doing so is significant. Google certainly has more resources to deliver true conversational search, but at what cost? The economics of powering Google-scale search with a large language model are likely quite different from the current search infrastructure. What is the cost of delivering conversational search at 100,000 results per second?
I expect Google to add the Google Assistant to their search bar at some point, but not before they've worked out how they can keep the ads flowing :)