What is OpenAI's Q*? How Aligned Incentives are Fueling a Questionable Narrative.
Here is what the public does and doesn't know and the implications
News broke late Wednesday about an internal project at OpenAI called Q* (pronounced Q-Star) that is generating a lot of discussion online and across media outlets. All things OpenAI are top of mind following its soap opera-style drama over the past week. In addition, this story includes a favorite element of both the “AI doomers” (i.e., safetyists worried about cataclysmic AI risk) and “AI boomers” (i.e., accelerationists promoting unchecked investment).
In some people's opinion, Q* is said to move the world closer to the realization of artificial general intelligence (AGI), which is frequently referred to now as artificial super intelligence (ASI) or just super intelligence. However, almost no one has seen anything about this. The catalyst for all of the online chatter is reporting on the existence of a letter that is not available for open examination.
What is Q*?
The news of Q* immediately gained currency with the doomers who offered it as evidence that OpenAI’s board may have been justified in its concerns over Altman’s ambivalence about the potential safety issues of a recent technology breakthrough. Reuters broke the story Wednesday afternoon around 3:00 pm PST, followed quickly by CNBC. According to Reuters (emphasis Synthedia):
Ahead of OpenAI CEO Sam Altman’s four days in exile, several staff researchers wrote a letter to the board of directors warning of a powerful artificial intelligence discovery that they said could threaten humanity, two people familiar with the matter told Reuters.
The previously unreported letter and AI algorithm were key developments before the board's ouster of Altman…The sources cited the letter as one factor among a longer list of grievances by the board leading to Altman's firing, among which were concerns over commercializing advances before understanding the consequences. Reuters was unable to review a copy of the letter. The staff who wrote the letter did not respond to requests for comment.
After being contacted by Reuters, OpenAI, which declined to comment, acknowledged in an internal message to staffers a project called Q* and a letter to the board before the weekend's events, one of the people said. An OpenAI spokesperson said that the message, sent by long-time executive Mira Murati, alerted staff to certain media stories without commenting on their accuracy.
Some at OpenAI believe Q* (pronounced Q-Star) could be a breakthrough in the startup's search for what's known as artificial general intelligence (AGI), one of the people told Reuters. OpenAI defines AGI as autonomous systems that surpass humans in most economically valuable tasks.
Given vast computing resources, the new model was able to solve certain mathematical problems…though only performing math on the level of grade-school students.
When Incentives Align
A key reason why this story is gaining so much currency is its existence aligns with the incentives of so many constituencies. It is not because we had a slow news week around the U.S. Thanksgiving holiday. There was a landslide of high-profile technology news this past week as many companies attempted to take advantage of intense interest around OpenAI’s management shake-up saga.
In response to a LinkedIn post about this development, I noted that it is important to recognize that the existence of a hypothesis or a letter does not confirm its veracity. Interestingly, the leak of the paper’s existence benefits multiple parties.
Former OpenAI board members can say their actions were justified. While Reuters notes that the case against Altman was not based on a single ingredient but rather a cocktail of charges, the board members that threw the company and industry into turmoil can lean on their position as an attempt to protect the world from imminent harm. The New York Times reported that OpenAI board member Helen Toner told company management, “The board’s mission was to ensure that the company creates artificial intelligence that ‘benefits all of humanity,’ and if the company was destroyed, she said, that could be consistent with its mission.”
AI doomers can say, “We must pause AI Research.” There is a vocal constituency of AI researchers, business leaders, policymakers, regulators, politicians, and commentators who would like to see the advance of AI research slow down. All of them cite the potential for catastrophic harm to humanity, while some harbor conflicts of interest as they hope to expand their influence or close the gap to competitors with a technology lead.
AI boomers can say, “Think of the potential benefits to humanity.” The uber techno-optimists, many of whom invest in or work in developing AI technology, want to stoke enthusiasm that progress toward AGI or ASI is bringing the world closer to realizing the goal. This helps justify their investment thesis or life’s work. It also offers a platform to discuss how generative AI will benefit the world beyond helping users write pithier emails or code software faster. A knock-on effect is that it drives more interest in products in this category and can increase private company valuations.
Media news sites generate a lot of clicks. While many media organizations may have planned for a slow news week, OpenAI’s tumult was a bonanza of clicks. That intense interest was about to wane as OpenAI resolved its leadership crisis, and people turned their attention to holiday activities or simply getting back to work on everyday business issues. Throwing an AGI/ASI story on the fire helped stoke continued interest and extended the original OpenAI story.
OpenAI gets more positive publicity about its technological leadership. If OpenAI is positioned as potentially having a technical breakthrough, it further differentiates the company from competitors. It also shifts the focus away from the self-imposed reputational black eye at a time when the company needs to re-establish trust with partners and customers. Note this also arrives at a time when OpenAI is seeking a higher valuation from investors in advance of a new funding round. The most recent investment round was at risk due to the management fiasco but could proceed with renewed vigor given ambiguous claims about new technology advances.
Granted, the presence of aligned incentives around hyperbolic claims doesn’t mean they are untrue. It is possible that Q* does point to a significant breakthrough. It is also possible that the significant breakthrough represents more evidence that the current path of development is leading toward some people’s goal of AGI/ASI.
However, given the track record of the unseen claims that crop up in the AI market, I’d suggest you retain some skepticism. The biggest technical and product breakthroughs around generative AI have not been secrets. Consider that legitimately important advances, such as the attention mechanism, generative adversarial networks (GANs), diffusion models, reinforcement learning, BERT, GPT-3, Stable Diffusion, ChatGPT, and Claude 2, were made available for inspection. Q* may be tremendous, or it may turn out to be oversold like GPT-3 was in mathematical abilities or the tenuous claims earlier this year around emergent abilities in LLMs.
Skepticism about the Path to AGI
There is an interesting phenomenon at play. Extreme views gain the most traction in popular discourse. Breathless headlines about robot overlords and AGI’s risk to humanity drive a lot of news article clicks and, therefore, the favor of news editors. YouTube's and X’s recommendation algorithms ceaselessly promote videos and posts that claim AGI’s arrival is imminent, exciting, and a positive development. Those algorithms are optimized for generating user engagement. These stories meet business objectives.
However, not everyone is convinced either of these extreme positions merits the currency media and social networks afford them. Synthedia reported last week that Sam Altman expressed skepticism that the current path of large language model (LLM) development would lead to AGI during an appearance before the Cambridge Union two weeks ago.
I think we need another breakthrough. I think we still we can push on large language models quite a lot and we should and we will do that. You know, we can take our current the hill that we're on and keep climbing it and like the peak of that is still pretty far away but within reason. I mean, you know, if you push that super, super far maybe all this other stuff emerged. But within reason, I don't think that will do something that I view as critical to an AGI.
To stick with that example from earlier in the evening in physics, let's use the word super intelligence now; if super intelligence can't discover novel physics, I don't think it's a super intelligence. And training on the data of what you know, teaching [it] to like clone the behavior of humans and human text, I don't think that's going to get there. And so there's this question which has been debated in the field for a long time of what do we have to do in addition to a language model to make a system that can go discover new physics and that'll be our next quest.
If you prefer the viewpoint of an AI scientist, look no further than Yann LeCun, Meta’s chief AI scientist and long-time computer science professor at NYU. He is also one of three computer scientists to be jointly awarded the Turing Award for the most significant contributions to AI research. LeCun posted details behind his skepticism about LLMs evolving into AGI yesterday on LinkedIn.
There are well-credentialed professionals on both sides of this debate, and a lot more that fall somewhere in between. You will not see this in most Q* news coverage or YouTube videos by self-appointed AI influencers. You are also unlikely to see these more nuanced positions from avowed AI boomers and doomers. Agendas matter.
The Real Q* Story
Ilya Sutskever is one of the world’s most prominent AI researchers and has been instrumental in developing generative AI as we know it today. He also has a tendency to speak openly about catastrophic AI risk even while noting its great potential for good. In a recent TED Talk that was published just three days ago and has already racked up over 500,000 views on the TED website and another 300,000 on YouTube, Sutskever talks about both risks and benefits posed by generative AI.
The official OpenAI statement about Sam Altman’s firing focused on his not being “consistently candid” with the board. That narrative was soon replaced in media accounts with Sutskever’s concern about AI safety in relation to some recent advances in OpenAI technology and Altman’s seeming disregard for its implications.
If part of your mission is to ensure AI doesn’t destroy humanity, you will fire a friend and co-founder for the greater good. Then Sutskever reversed course and pledged support for Altman’s return. How should you interpret that sequence of positions by someone who is committed to AI safety and has intimate knowledge surrounding the supposed technology breakthrough?
At the same time, former AI board member Helen Toner and her colleagues have come under fire for their mismanagement of the debacle. No one wants to appear incompetent. And someone wrote the paper expressing concern. They don’t want their concerns swept under the rug by the returning management team.
Leaks of the letter's existence help elevate both constituencies as noble in their aims. It also creates a potential scenario where OpenAI management must justify that the new technology is not harmful to humanity or a significant step toward harm. It’s always fun to be forced into proving a negative. The fact that the letter was characterized by a source but withheld from review by others makes it a one-sided argument without the benefit of scrutiny.
Q* may be amazing or dangerous. The agendas and incentive alignment warrant skepticism regarding all claims until there is an opportunity to review the evidence. I suspect history will find it is less momentous than Q*’s current portrayal, but that may not convince the extreme doomer and boomer crowds who will continue to be driven by verificationist tendencies.
Let me know what you think in the comments. Is Q* a flashpoint, an opportunistic vehicle for promoting specific agendas, or something else?
Synthedia is a community-supported publication. Please check out our first sponsor, Dabble Lab, an independent research and software development agency that is entirely automated with a mix of in-house AI tools and GitHub Copilot. The founder, Steve Tingiris, is also the author of Exploring GPT-3, the first developer training manual for building applications on GPT-3.
