Will OpenAI's G3PO Open-Source LLM Become Meta Llama's Biggest Challenge

Two tech giants facing a showdown in the open-source LLM market.

Meta’s release of Llama 2 continues to shake up the large language model (LLM) generative AI segment. Since its launch two weeks ago, several tuned variants have knocked it off the top of Hugging Face’s Open LLM Leaderboard.

Stable Beluga, Stability AI’s fine-tuned version of Llama 2 70B, made a brief appearance in the leaderboard’s top spot. That was recently displaced by Upstage’s Llama 20 70b Instruct model, another fine-tuned version of Meta’s flagship open-source model. However, both of these variants offer non-commercial licenses only. Meta’s Llama 2 foundation model offers a permissive commercial license with some caveats.

Upstage claims that its new model now outperforms GPT-3.5. I believe that may be true of the TruthfulQA test, which is part of the evaluation, but it still trails OpenAI’s proprietary model in the other three core tests. The key point is that Llama 2 has led to a rapid rise in the out-of-the-box performance of open-source models.

This is not even the first time this has happened. After the release of Llama 1, several fine-tuned variants showed up and knocked it down the leaderboard. The proprietary to open-source LLM lead is closing and likely is already at parity for some use cases. As the LLM leader, OpenAI has the most to lose if open-source starts to become a preferred option.

OpenAI’s Response - GP3PO

OpenAI’s proprietary models still hold a performance edge, but that is not expected to last. In the internally circulated Google Has No [AI] Moat memo, which was eventually confirmed as authentic, the author starts by saying, “We have no moat and neither does OpenAI.” The Google researcher continued (emphasis the author’s):

We’ve done a lot of looking over our shoulders at OpenAI. Who will cross the next milestone? What will the next move be?

But the uncomfortable truth is, we aren’t positioned to win this arms race and neither is OpenAI. While we’ve been squabbling, a third faction has been quietly eating our lunch.

I’m talking, of course, about open source. Plainly put, they are lapping us. Things we consider “major open problems” are solved and in people’s hands today.

…

While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months.

OpenAI’s response may be to take on the open-source models directly. The Information reported in May that OpenAI was working on an open-source LLM. Last week, The Information reported that the LLM’s name is G3PO but could not discover when the model will be released. Granted, a named model sounds more promising than rumors about an unnamed model.

This may be a critical period for OpenAI. The obvious parallel is its experience with the once industry-leading DALL-E text-to-image generator. Stability AI introduced the open-source Stable Diffusion and quickly gathered up market share and mind share among developers and end-users. The risk is that Llama could fill the same spoiler role in the LLM segment.

With that said, GPT-3 and GPT-4 seem to have a short-term technical moat and a longer-term moat around awareness and trust. DALL-E never had that level of trust and awareness. Don’t underestimate how much further ahead in the market OpenAI is in LLMs than it was when Stable Diffusion emerged.