Meta is all-in on AI. There are two key drivers behind this strategy.

1. Meta’s core business revolves around gaining and maintaining user attention. That attention is then monetized through ads or commerce. If AI assistants begin to consume more of a user’s attention, Meta wants to maintain the dominance it accumulated in that metric during the recent mobile era.

2. Meta’s tech giant rivals are all investing heavily in AI. The company believes it is critical to make it hard for rivals to gain a foothold and expand into the consumer “attention economy” by using generative AI as a wedge. Thus, it offers its AI assistant features to consumers free of charge, and it also offers frontier-quality foundation models to developers at no cost. Both moves make it harder for rivals to generate revenue they can invest back into AI innovation.

AI Assistants for Consumers

Meta AI is the company’s answer to ChatGPT. When it arrived in September 2023, it was driven by the Llama 2 70B model and generally trailed OpenAI’s ChatGPT and Microsoft’s consumer-oriented Copilot in performance. The Gemini assistants from Google that arrived in early 2024 also had some performance advantages. None of this was a surprise. Meta’s 70B parameter large language model (LLM) was competing against foundation models with far higher parameter counts and training data.

With the introduction of Meta’s Llama 3 405B parameter LLM earlier this month, Meta AI is now facing more of a level playing field. The default solution remains the Llama 3 70B model, and it appears to be sufficient for many users around the world. However, there is also a new limited-use option that employs the 405B model. This is a significant step in bringing Meta AI closer to parity with rivals.

Most-Used AI in 2024?

Mark Zuckerberg commented on the progress and direction for Meta AI during the company’s quarterly earnings call this past week. Unsurprisingly, his aims are ambitious. According to Zuckerberg:

Last quarter we started broadly rolling out our assistant Meta AI and it is on track to achieve our goal of becoming the most used AI assistant by the end of the year. We have an exciting road map ahead of things that we want to add, but the bottom line here is that Meta AI feels like it is on track to be an important service and it's improving quickly both in intelligence and features.

Some of the use cases are utilitarian like searching for information or role playing difficult conversations before you have them with another person and other uses are more creative like the new imagine yourself feature that lets you create images of yourself doing whatever you want in whatever style you want. And part of part of the beauty of AI is that it's general so we're still uncovering the wide range of use cases that it's valuable for.

Note the goal to become the most-used AI assistant by year-end. That will not be easy with ChatGPT’s lead in the segment, indicators that suggest Google is likely to become more aggressive in its growth efforts in consumer segments, and the introduction of Apple Intelligence later this year. There are also rumblings of an upgraded Alexa.

I am skeptical that Meta can reach its goal by year-end. However, Meta has the most formidable application-based distribution in the market through Facebook, Instagram, and WhatsApp. In addition, it depends on what metrics you want to apply to “most used.” Will that be the most users, most usage time, highest average session length, or some metrics concatenation? A metric that includes total users or sessions would be the most likely to achieve the goal.

Meta CFO Susan Li also added some additional details regarding Meta AI adoption during the recent earnings call. WhatsApp is leading the adoption path for the company. However, it appears Zuckerberg is betting that Instagram will drive the next surge in growth.

People have used Meta AI for billions of queries since we first introduced it. We're seeing particularly promising signs on WhatsApp in terms of retention and engagement which has coincided with India becoming our largest market for Meta AI usage. You can now use Meta AI in over 20 countries and eight languages. And in the US we're rolling out new features like Imagine edit which allows people to edit images they generate with Meta AI.

Attention!

Facebook and Instagram became the front door to digital experiences for many consumers. They are often the first apps opened on a consumer’s mobile device and often lead to other digital activities ranging from browsing and shopping to messaging. If a user needs to execute a task that requires an AI assistant, the risk for Meta is that its users may start their digital journeys somewhere else or end an app session early to access generative AI features elsewhere. The result would likely be a declining average session time and maybe fewer total sessions.

Meta AI provides an opportunity to mitigate the risk of a more favored AI assistant stealing user attention. If the company executes well, Meta AI could also lead to more and longer sessions. Raymond James, managing director of equity research, Josh Beck, commented after the earnings call that Meta AI may already be leading users to spend more time in Meta apps.

Meta AI…is a chatbot. They're using Instagram and Facebook to distribute it. …There's starting to be kind of a new conversation on Meta that you would have never really considered before. So, “Hey Meta, can you help me plan this trip with my friends?” [is] not something you really would have ever considered on your time on Instagram or Facebook but it is completely net new. It's driving more time. It's helping their users kind of find a new use case.

Zuckerberg commented along these same lines during the earnings call during the analyst questions period. However, he cautioned that monetization through increased attention may not arrive soon.

For things like Meta AI or AI Studio, I mean these are things that I think will increase engagement in our products and have other benefits that will improve the business and engagement in the near term. But before we're really talking about monetization of any of those things by themselves, I mean I don't think that anyone should be surprised that I would expect that that will be years…

I mean we obviously want to kind of grow that and grow the engagement on that to be a lot deeper, and then we'll focus on monetizing it over time.

Zuckerberg commented to analysts in October 2023:

Whenever there's more engagement in the apps, that creates the opportunity for more monetization.

Meta’s economic model is based on user attention. Just about every strategic and app feature decision the company has made over the last decade has focused on driving more and longer sessions. One area where the company has struggled to drive longer session times and advertising revenue is WhatsApp. Meta AI could provide an ideal tool for transforming that app experience into a more easily monetizable vehicle for the company and extend beyond simple communications. With that said, it may be some time before investors see a direct impact on revenue.

Customizing Consumer AI Experiences

Zuckerberg also highlighted the company’s most recent feature launch, Meta AI Studio. It appears that he expects creators on Instagram to be the first power users of the solution by enabling them to craft a customized character to interact with their followers. However, it is also a new feature that everyday users may adopt to enhance their experience on the company’s social networks. Zuckerberg commented:

An important part of our vision is that we're not just creating a single AI but enabling lots of people to create their own AIs. This week we launched AI Studio which lets anyone create AIs to interact with across our apps.

I think that creators are especially going to find this quite valuable. There are millions of creators across our apps and these are people who want to engage more with their communities and their communities want to engage more with them, but there are only so many hours in the day. So, now they're going to be able to use AI Studio to create AI agents that can channel them to chat with their community, answer people's questions, create content, and more. I'm quite excited about this.

But this goes beyond creators to anyone is going to be able to build their own AIs based on their interests or different topics that they are going to be able to engage with or share with their friends.

In some ways, this seems similar to GPT's functionality in ChatGPT and characters from Character AI. However, Zuckerberg seems to be thinking bigger. He mentioned business messaging as a key feature. That suggests creators and small businesses might begin to leverage the capability to answer fan and customer queries, which would be a meaningful change in feature value.

AI Celebrities Personas Fail

This all sounds good, but Meta’s ability to read the market has already faced headwinds. During the announcement of Meta AI last fall, Zuckerberg spent just 21% of the nine-plus minutes talking about that product and 79% promoting AI characters that mimicked celebrities such as Kendall Jenner, Snoop Dogg, and Tom Brady. Users did not share that enthusiasm. The Information reported in July:

Meta Platforms has scrapped its first high-profile foray into consumer artificial intelligence—chatbots played by celebrities and social media influencers such as Charli D’Amelio, MrBeast and Paris Hilton…

None of the AI chatbots amassed particularly big followings, especially compared with the personal accounts of the celebrities. In some cases Meta paid millions of dollars to license these celebrities’ likenesses.

…

But the celebrity chatbots didn’t gain much of a fanbase—and some users found them weird.

Snoop Dogg’s AI character had only about 15,000 followers on Instagram, compared with 87.5 million on the rapper’s personal account, as of late June.

Jenner’s Billie was the most successful of the bunch based on followers, amassing about 179,000. But that still paled in comparison to the real-life Jenner’s Instagram account, which currently has 294 million followers.

The Consumer AI Assistant Showdown

Thus far, the rise of Generative AI-enabled assistants has primarily been a competition among new and old tech giants, such as OpenAI, Microsoft, and Google. In addition, the proving ground has revolved around production and productivity—creating text and images or making it more efficient to work with text. Meta is taking a different approach. It is focused on consumption and connectivity.

It seems unlikely that Meta AI will become the productivity-focused AI assistant of choice. However, it could become a leading leisure and entertainment-focused AI assistant. The big sign that Meta is succeeding will be if it starts to become the reason users open its apps as opposed to accessing them occasionally when already in the apps. I suspect the latter scenario is more likely.

I’d also be surprised if characters take off on Instagram. There may be a flurry of activity over the next 4-6 weeks. However, the key metric to watch is user retention. Most of these types of features, such as Meta’s AI characters and OpenAI’s GPTs, showed strong initial demand that soon tapered off. Consumers seem to want digital assistants for productivity tasks. It is unproven whether they will adopt them for entertainment use cases. The sustainability of character customization has only shown staying power for digital boyfriends and girlfriends on Character AI. So, much of Meta AI’s strategy is headed into unknown territory.

We’ve seen previous clashes around AI assistants — Siri and Google Now, Alexa and Google Assistant. Generative AI represents a continuation of that trend. However, now, the technology is much more useful, the investment if larger, there are more competitors, and there is a growing use case segmentation. Meta AI is taking the consumer route. OpenAI, Microsoft, and Anthropic are largely taking the business and personal productivity route. Google is likely to do both. Let the games begin (again)!

Share

Subscribe now

Many thanks to our title sponsor.

• Dabble Lab: AI-powered bespoke software development services

ChatGPT was an extraordinary tool from day one. It could eruditely summarize information, transform it into another style, facilitate brainstorming, and generate content. It could even answer questions to a degree. However, the answers were sourced from the foundational model’s training data, which was not up-to-date and didn’t provide an easy way to verify the response’s veracity. That was somewhat rectified with the Bing and Browsing plugins and then somewhat improved after the native Bing integration in 2023. Still, it has always been somewhat dissatisfying.

That dissatisfaction was cemented when compared with Perplexity’s generative AI-enabled search that typically leveraged OpenAI’s large language models (LLM). Google’s Bard was also better than ChatGPT search, though it was too often wrong. Gemini has significantly improved that experience and its accuracy. Even Google’s generative AI-enabled add-on to traditional search was superior.

This is critically important because search has always been one of the most obvious and beneficial use cases. Regardless of how far ahead OpenAI may have been with LLM performance, ChatGPT was inferior at search. OpenAI has set out to rectify that deficiency with SearchGPT.

SearchGPT Prototype

OpenAI announced a waitlist for the SearchGPT prototype this week and offered a few sneak peeks at what the new search experience may look like.

We’re testing SearchGPT, a prototype of new search features designed to combine the strength of our AI models with information from the web to give you fast and timely answers with clear and relevant sources. We’re launching to a small group of users and publishers to get feedback. While this prototype is temporary, we plan to integrate the best of these features directly into ChatGPT in the future. 

Getting answers on the web can take a lot of effort, often requiring multiple attempts to get relevant results. We believe that by enhancing the conversational capabilities of our models with real-time information from the web, finding what you’re looking for can be faster and easier.

This sentiment is exactly right. Traditional search is significantly inferior to generative AI-enabled search. The ability to have a conversation with your search results is a significant benefit in extending your research or clarifying your results. It eliminates the need to start over when traditional search just isn’t sending you the right backlinks.

In addition, traditional search assumes that users will become their own information archeologists. Google or Bing may point you to a potential source of an answer to your query, but not the answer. It also may ignore some important nuances that could significantly improve the results. Generative AI-enabled search gives you an answer. No more information archeology is required. If you want to consult the sources, they are provided. That is useful for verifying the results. However, as these searches have improved, errors have become less and less frequent You get an answer instead of a link to find an answer on your own.

These results save users time and often provide additional context that they might have missed. When you summarize the best information from three to ten search results, you typically receive a more well-rounded result. To do that on your own is very time-consuming as you could easily be required to read 1,000 to 10,000 words before figuring out doing your own summary.

New Search User Experience

Anyone who has used Perplexity or Google’s Search AI Overviews will recognize some familiar output formats. However, there appear to be some new user experience layouts that may offer slight improvements to existing styles.

One new layout promises to offer more data on the screen. Notably, there are not ads anywhere. However, that doesn’t mean ads won’t show up at some point. Perplexity has indicated that it will consider ads, while Microsoft and Google have already experimented with ad placement for some searches.

Another change for OpenAI is that new search capabilities will be far more visually oriented than we have seen from ChatGPT to date. Images and videos will be some search results formats, and it looks like the search will likely include carousel options.

Granted, we don’t know what the final user experience will be like at this point. However, it appears that OpenAI is going for a mix of the styling offered today by Perplexity and Google.

The New Search Wars are Upon Us

SearchGPT is just a prototype feature for now, but it is an indication that OpenAI is now taking search seriously. That means the search wars are about to heat up. Google still has more assets to leverage in these battles. However, OpenAI, Perplexity, and others indicate the start of the long journey toward eroding Google’s once unassailable dominance.

Share

Subscribe now

Many thanks to our title sponsor.

• Dabble Lab: AI-powered bespoke software development services

The large language model (LLM) market has experienced several noted upheavals in 2024. New models have emerged to challenge OpenAI’s unmatched quality supremacy of 2023 to the point where there are several leading foundation models that can legitimately claim to be “market-leading.”

• Google Gemini 1.5 introduced an industry-leading 1 million, then a 2 million token context window along with public LLM benchmark scores nearing OpenAI’s leading models.

• Anthropic’s Claude 3.5 Sonnet showed public benchmark results that beat those of OpenAI’s GPT-4.

• OpenAI introduced GPT-4o, the latest multi-modal AI foundation model demonstrating industry-leading performance across numerous benchmarks.

The result is greater user choice of AI frontier models. Frontier models are the LLMs and multimodal AI models that include the most impressive features, quality, and benchmark performance results. To date, this category has been dominated by proprietary models. Meta’s Llama 3.1 405B deserves recognition as the first open model to join the elite frontier model category. The implications are significant.

From Niche Leader to Overall Competitor

Meta made headlines in 2023 when it introduced the first Llama 2 70B model. It did not deliver the same performance as GPT-4, the leading model at the time. However, it was the most impressive LLM in the open model category, which meant it was free for nearly anyone to use—provided the developer could set up the servers, manage the installation, and pay some fairly hefty cloud computing bills.

With the July 2023 release, Meta carved out a leadership spot in the open LLM category. Offerings from X.ai, Databricks, and TII sought to challenge Meta’s leadership throughout late 2023 and into 2024. Meta reasserted its clear open model leadership credentials with Llama 3 early this year.

Llama 3.1 405B moves Meta up a notch. It retains and extends Meta’s lead in the open model category. The new release also places Meta among the leaders in frontier LLMs alongside OpenAI, Anthropic, and Google. This is particularly noteworthy when considering findings from a16z research earlier in 2024.

Forty-one percent of enterprises that have already deployed generative AI expect to increase the adoption of “open-source” models when they match performance with the leading proprietary models. Llama 3.1 405B will put that sentiment to the test. Comparable performance has arrived.

Leveled Up Performance

The evidence of Llama 3.1 405B’s performance rise can be seen in its improved scores on popular public LLM benchmarks. Its MMLU scores surpass GPT-4 and nearly match GPT-4o and Claude 3.5 Sonnet. Llama 3.1 also showed standout performance in reasoning, math, and long-context benchmarks.

Comparing LLMs based on public benchmarks can be useful but doesn’t necessarily tell users the entire story of model performance. Model developers selectively publish benchmark results, and strong results for a handful of tests don’t necessarily translate into an optimal selection for a specific use case. Still, the benchmarks represent objective analyses of quality that users can consult before implementing or testing a model.

A key reason the 405B model performs so well is the data volume and quality used in training. It was trained on 15 trillion tokens. The new model also provides a 128K context window, which is sufficient for most use cases today. A research paper outlining the new model capabilities offered additional details on the impact of data.

Compared to prior versions of Llama, we improved both the quantity and quality of the data we use for pre-training and post-training. These improvements include the development of more careful pre-processing and curation pipelines for pre-training data and the development of more rigorous quality assurance and filtering approaches for post-training data. We pre-train Llama 3 on a corpus of about 15T multilingual tokens, compared to 1.8T tokens for Llama 2.

Excelling in Small Packages

For developers considering smaller models, the Llama 3.1 series also includes the 70B and 8B (B = billion parameters) model sizes. The 70B model handily outpaces GPT-3.5 Turbo, and 8B shows a similar lead over Mistral 7B Instruct.

Broad Availability

A key drawback of many open models is that they are not widely supported as end-points. That means users need to deploy the models and manage the infrastructure in order to run inference jobs. However, that is slowly changing for open-source overall and has rapidly changed for Llama. Meta announced ten partners hosting Llama 3.1 405B models today. All of them offer real-time inference, over half provide batch inference, and all but one offer knowledge bases and guardrails.

High performance coupled with ease of access is critical for market penetration. Llama has generated a lot of enterprise interest, but there have been barriers to trial. The latest release shows that Llama 3.1 smartly adds more options for users to access the models, including those from each of the three leading cloud hyperscalers.

Open Leadership?

Meta CEO Mark Zuckerberg highlighted the market shift that Llama 3.1 405B represents in a blog post announcing the new models.

Today we’re taking the next steps towards open source AI becoming the industry standard. We’re releasing Llama 3.1 405B, the first frontier-level open source AI model, as well as new and improved Llama 3.1 70B and 8B models. In addition to having significantly better cost/performance relative to closed models, the fact that the 405B model is open will make it the best choice for fine-tuning and distilling smaller models.

Will this lead to a shift away from proprietary AI models? I don’t think so in the near term. However, there is strong interest among enterprise buyers for models that are frontier-grade and also open. Unsurprisingly, enterprises like the idea of asserting further control over their AI solutions along with enhanced privacy. Meta’s Llama 3.1 “Herd of Models” will likely be the catalyst that drives more enterprise trials of open models.

Share

Subscribe now

Many thanks to our title sponsor.

• Dabble Lab: AI-powered bespoke software development services

OpenAI’s generative AI market leadership has converted to solid revenue and customer growth. Recent reporting by The Information revealed that CEO Sam Altman told company employees “OpenAI has more than doubled its annualized revenue to $3.4 billion in the past six months or so.”

With continued growth at a comparable rate, those figures suggest OpenAI will finish 2024 with over $4 billion in annual revenue. The biggest revenue driver is ChatGPT. According to the article’s sources, it represents about $3.2 billion in annualized revenue, 94% of the total. If this were the consumer edition alone, it would suggest over 13 million consumers are paying the $20 monthly ChatGPT fee. More likely, 8-10 million consumers are subscribers, and the balance is coming from businesses. According to The Information:

That revenue rate puts the ChatGPT-maker far ahead of its rivals. Last fall, for instance, rival Anthropic told investors it was generating revenue at a $100 million annualized rate, with plans to reach more than $850 million in annualized revenue by the end of 2024. Cohere, a Canadian OpenAI rival, was generating just $22 million in annualized revenue in April.

Ahead of Azure

The remaining $200 million in annualized revenue is supposedly coming from payments for access to OpenAI’s foundation models. It is unclear whether this includes revenue sharing from Microsoft’s Azure, but it is very likely it does.

Another article from The Information reports that Altman told staffers that providing model access is a $1 billion annualized revenue business. That would suggest it reflects about 29% of total revenue, while ChatGPT for consumers and enterprises is closer to 70% of total revenue. There is an inherent risk in putting too much stock in rumors reported by Silicon Valley tech publishers. However, we are seeing some consistency around the fact that OpenAI is now generating billions of dollars in annual revenue, most of which is through ChatGPT.

The Information also suggests that OpenAI is not generating more revenue from selling access to its AI foundation models than Microsoft Azure because it hit the billion-dollar annualized run rate months ahead of the cloud giant.

The growth of OpenAI’s API business has put pressure on Microsoft’s rival business of reselling OpenAI’s models. Managers at Microsoft recently changed the way they charge customers to use Azure OpenAI Service to take some of the steam out of OpenAI’s customer wins, according to someone involved in the decision.

For instance, Microsoft cut the minimum amount customers must spend to reserve servers on the Azure OpenAI Service to guarantee access during periods of peak demand. Microsoft dropped the price 25% to roughly $185,000 per month for OpenAI’s GPT-4 model, according to someone with knowledge of the move.

Azure originally lured enterprise users to its OpenAI Service by convincing them it would provide a more reliable and secure service. For much of 2023, the reliability and stability claims around the API were likely true. However, OpenAI is winning over more enterprises with more consistency and earlier access to its newest model releases.

OpenAI provides Microsoft access to its new models at the same time it announces them. However, Azure typically took about four weeks to test and provision those models in 2023 and early 2024. More recently, that lag time has decreased, specifically for the GPT-4o model.

Revenue Builds Buyer Confidence

The most important element of this news is that it will likely help OpenAI extend its lead. Technology buyers take many factors into account regarding vendor selection. Product quality and effectiveness are important, as is business viability. Many enterprise buyers are reluctant to bet on start-ups that are pre-revenue as it creates more uncertainty about their market staying power.

Granted, revenue does not mean profitability. OpenAI still has high costs and faces financial risk. It also is being compared with its AI model rivals. That includes well-financed tech giants such as Google and Meta. It also includes startups like Anthropic, which have far less revenue and proven business viability. Anthropic’s latest AI foundation model may be better than GPT-4o based on some benchmarks. However, the company is almost certainly viewed as a bigger risk for any enterprise considering a long-term bet on an AI partner, given its smaller customer and revenue base.

Higher revenue means that OpenAI can more easily cover expenses and invest more in product development, marketing, sales, and acquisitions than rivals without having to raise as much outside capital. The Information also reported that OpenAI has a 200-person sales team today, up from 10 in 2023.

Share

Subscribe now

Many thanks to our title sponsor.

• Dabble Lab: AI-powered bespoke software development services

Stable Diffusion was once and likely still is the most widely used text-to-image AI foundation model. It rose to prominence in 2022 as an open-source alternative to OpenAI’s DALL-E, Midjourney, and other proprietary offerings. The control and cost benefits of Stable Diffusion’s open-source model made it attractive for many software makers who wanted to add AI-enabled image generation to their products.

The core technology behind Stable Diffusion is Latent Diffusion, which was developed at Ludwig Maximilian University in collaboration with Runway. However, Stability AI became the driving force behind model advances, new features, and adoption. The company delivered updates to the open-source model and also provided a hosted service for users who wanted easy access to the software.

Unfortunately, Stability AI was plagued by poor economic performance, rapid staff turnover, and controversies that undermined investor confidence in the company and its founder, Emad Mostaque. Synthedia wrote in November 2023:

If Stability AI didn’t have the funding, personnel, and product doubts hanging over it, the company might have already hit the aspirational $4 billion valuation. The AI image generation segment is not as large as LLMs, but it is important and large enough to matter. Canva and Picsart users have created millions of images courtesy of Stable Diffusion-powered features. They are not alone.

The more salient question is how many companies are or will engage Stability AI for its value-added services. Today, it offers convenience on top of a free product that is not terribly hard to operate…Stable Diffusion has a large market share and developer ecosystem but there is no guarantee Stability AI will benefit from that earlier work.

….

OpenAI’s [November 2023] debacle looked like a comedy. Stability AI’s turmoil may well become a tragedy without swift changes or the arrival of a white knight.

New CEO + New Funding

This past week, a white knight arrived with a cash infusion, a new CEO, and a famous Facebook and Napster alumnus. According to The Information:

Prem Akkaraju, former CEO of visual effects company Weta Digital, will step into the role, according to a person involved in the decision.

The 51-year-old Akkaraju is part of a group of investors including former Facebook President Sean Parker that has stepped in to save Stability with a cash infusion that could result in a lower valuation for the firm, which develops products that use Stable Diffusion, an open-source AI image generator.

Mostaque stepped down from his board seat and CEO position in March. Many viewed this as necessary before new investors would come in to rescue the company. Shan Shan Wong, former Stability AI COO, and Christian Laforte, former CTO, took over as co-CEOs after Mostaque’s departure. Their LinkedIn profiles still list each of them as Interim co-CEO.

The new CEO does not have a notable background in AI, but Weta Digital (aka Weta FX) is a special effects agency that has worked on films ranging from The Hobbit and Avatar to the Avengers and Guardians of the Galaxy. That suggests Akkaraju fits Stability AI’s target customer profile. Stable Diffusion, Stable Video Diffusion, and Stable Audio Diffusion are all tools that visual effects creatives and agencies could employ in their daily work.

Stability AI has many technology assets and many users of the core text-to-image model. It also has a large staff of AI researchers, who may present significant value on their own. Maybe a film industry veteran and Sean Parker can help the one-time unicorn get back to a valuation of $1 billion.

Share

Subscribe now

Many thanks to our title sponsor.

• Dabble Lab: AI-powered bespoke software development services

OpenAI announced Friday it was acquiring Rockset, “a leading real-time analytics database that provides world-class data indexing and querying capabilities.” Rockset was founded in 2016 by two former software engineers at Facebook. The company’s focus was “cloud-native” databases, and it moved into serverless analytics and microservices three years later. In April 2023, the company introduced serverless vector search with a demonstration that leveraged an OpenAI embeddings model.

Rockset has raised $117 million in venture funding to date, according to Crunchbase, with Sequoia and Greylock investing in multiple rounds. The company raised $40 million in 2020, and $51 million in Q3 2023.

The acquisition announcement was accompanied by FAQs that indicated Rockset will be shutting down access to its software and working to transition customers to alternative providers. Month-to-month contract holders have until the end of September to transition, while free trial users will be unable to upgrade or use the service going forward. The entire Rockset team will be integrated into OpenAI, which will have exclusive access to the company’s technology.

Serverless RAG

You are probably familiar with retrieval augmented generation (RAG) if you are following the evolution of the enterprise generative AI market. It has quickly become the most popular technique for grounding AI model responses to corporate data. The impact is a lower “hallucination” rate, more precise answers, and explainability via citations. Serverless RAG emerged over the past year to address scaling challenges associated with vector databases.

Pinecone introduced a serverless RAG product in January 2024, along with claims of 50x cost savings. Existing vector store services typically charge users for peak compute capabilities whether or not they are using it. Serverless RAG only charges users for computing resources when queries are active and uses less memory for indexes. Turbopuffer is another software package that offers serverless RAG indexing, though there appear to be some differences between the three models.

Phases of Data in Generative AI’s Evolution

Over the past year, the application of data to foundation model performance improvement has experienced several phases. We first saw the scaling phase, where more data tokens used in training was the dominant strategy. That was followed by the data curation and data enhancement phases that focused on data quality. The Rockset acquisition is related to a new phase related to data access and data ops.

This phase is less about making the models better and more about improving the user output when AI foundation models are connected to enterprise data. Rockset is a step in that direction and also represents the next wave of RAG innovation and adoption.

Cohere was once positioned as an OpenAI alternative and general-purpose AI foundation model developer. It shifted its focus to data retrieval in mid-2023 and now promotes the tight integration of its large language models (LLM) with RAG. This shift reflects the first wave of enterprise adoption. Everyone want to use LLMs with their own data. RAG is the simplest and most reliable way to meet this objective today. While Cohere is not using serverless RAG today, its Command R+ models highlight the enterprise knowledge retrieval focus.

OpenAI has focused so far on delivering strong AI foundation models and has left the RAG work up to developers that integrated knowledge retrieval solutions. With that said, OpenAI already has some RAG/vector features in its GPTs and through the Assistant API. Rockset will provide additional in-house expertise and tooling to enable more capabilities directly and at a lower operating cost.

Share

Subscribe now

Many thanks to our title sponsor.

• Dabble Lab: AI-powered bespoke software development services

Anthropic released a new AI foundation model today. Claude 3.5 Sonnet is the latest iteration of the large language model (LLM) that has morphed into a multimodal AI model that includes capabilities around language and images.

The new offering arrived less than four months after the introduction of the Claude 3 model family, which was among the first to claim superiority over OpenAI’s GPT-4. Last month, OpenAI introduced GPT-4o, which set a new standard for quality and capability. While Claude 3 Opus is still widely regarded as a leading frontier AI model, the introduction of Claude 3.5 Sonnet is an important step, given Anthropic’s objective to surpass the performance of the OpenAI GPT-4o offering.

Claude 3.5 Sonnet raises the industry bar for intelligence, outperforming competitor models and Claude 3 Opus on a wide range of evaluations, with the speed and cost of our mid-tier model, Claude 3 Sonnet.

Claude 3.5 Sonnet is now available for free on Claude.ai and the Claude iOS app, while Claude Pro and Team plan subscribers can access it with significantly higher rate limits. It is also available via the Anthropic API, Amazon Bedrock, and Google Cloud’s Vertex AI.

…

Our aim is to substantially improve the tradeoff curve between intelligence, speed, and cost every few months. To complete the Claude 3.5 model family, we’ll be releasing Claude 3.5 Haiku and Claude 3.5 Opus later this year.

Benchmark Standard

Claude 3 Opus is widely regarded as one of the top three generative AI foundation models. Many see it as superior to GPT-4 Turbo and competitive with GPT-4o. However, Claude 3.5 Sonnet has set a new standard among Anthropic’s AI models. This is significant, in part, because it is a smaller model.

The diagram at the top of the page emphasizes the implications of delivering higher quality with a smaller model. Claude 3.5's Sonnet is “more intelligent” than Claude 3's Opus, and it delivers this quality at a far lower price point.

It is worth noting that the Claude 3 announcement included results from ten language and reasoning AI model benchmarks and the Claude Sonnet 3.5 only presents eight. The missing two benchmarks are ARC-Challenge and Hellaswag. You can draw your own conclusions about why results for those benchmarks didn’t make it into the current announcement.

However, it is to Anthropic’s credit that the current release added data for 0-shot MMLU, which is one of two reported results where GPT-4o outperformance Claude 3.5 Sonnet. Granted, the results are so close that you can easily conclude the models are comparable for the undergraduate level knowledge domain.

The other benchmark where Claude 3.5 Sonnet is surpassed by GPT-4o is MATH. GPT-4o achieved a score of 76.6% for 0-shot chain of thought, whereas 3.5 Sonnet was only 71.1%.

The new model excels across the other benchmarks. Some show comparable performance to GPT-4o, while others demonstrate a significant lead. The HumanEval benchmark for coding was seven percentage points higher, and GPQA Diamond for graduate-level reasoning was six percentage points. Anthropic also reported better scores in four out of five vision benchmarks compared to GPT-4o.

Addressing the Google Challenge

GPT-4o is clearly a target competitor for the Claude 3.5 model family, but so are Google Gemini 1.5 and Gemini 1.5 Flash. First, the Gemini 1.5 models dethroned Anthropic as the leader in LLM context window size. While Anthropic was promoting an industry-leading 200,000 token context window, Google casually introduced a model that could do 1,000,000. Less than three months later, it said a 2,000,000 token context window would be available this year.

Google also started establishing impressive benchmark scores, and then, with Gemini Pro 1.5 Flash, it added high quality combined with low latency and cost. Anthropic had held the position as the key OpenAI foundation model alternative for a year. Suddenly, Google was making a credible claim to that position. The higher performance and lower latency of Claude 3.5 Sonnet directly address advances made in the Gemini 1.5 Pro and the Flash models.

What’s Next?

Anthropic has raised a lot of funding and secured a few customers along the way. Its $18 billion valuation during the most recent funding round will be hard to justify. However, Anthropic had already established itself as OpenAI’s key LLM substitute. That position is under threat, but Claude 3.5 is likely to keep Anthropic in the first tier of LLMs.

Beyond performance, Anthropic will add new experiences and capabilities that may provide enhanced value for users.

In addition to working on our next-generation model family, we are developing new modalities and features to support more use cases for businesses, including integrations with enterprise applications. Our team is also exploring features like Memory, which will enable Claude to remember a user’s preferences and interaction history as specified, making their experience even more personalized and efficient.

OpenAI has not wrapped up the market. The company is clearly ahead of its competitors in several areas and has a far larger market share for model use and digital assistant adoption. Anthropic represents the key alternative to OpenAI. Claude 3.5 Sonnet and its sister models are likely to keep the company at or near the top of the foundation model hierarchy.

Share

Subscribe now

Runway announced its new Gen-3 Alpha model that features more expressive faces, sharper video, and more control over style and motion than the company’s earlier generation models. However, Runway is not promising minute-long generative AI videos like OpenAI.

The upgrade by Runway arrives at an important time for the startup. OpenAI shocked many industry insiders with the quality, versatility, and duration of videos produced by its Sora text-to-video model. That immediately led to comparisons with Runway and competitor Pika Labs, which had been seen as leaders in the segment before Sora’s debut.

Runway’s CTO, Anastasis Germanidis, told VentureBeat, the Gen-3 Alpha model will be available “in the coming days,” first to paid users and then for free users. This means it will likely be in the hands of creators before Sora, which OpenAI has made available to only a few artists. According to Runway’s announcement:

Trained jointly on videos and images, Gen-3 Alpha will power Runway's Text to Video, Image to Video and Text to Image tools, existing control modes such as Motion Brush, Advanced Camera Controls, Director Mode as well as upcoming tools for more fine-grained control over structure, style, and motion.

Control Matters

VentureBeat’s interview with Germanidis also revealed some of Runway’s learnings since launching the Gen-1 and Gen-2 models. According to Germanidis:

When we first released Gen-2, you could only prompt it in a very simple manner with text, and we quickly added a lot of controls around camera motion, around object motion, and that proved to be really important for how people are using Gen-2 today.

So in Gen-3 Alpha, we basically invested a lot more time and resources in that, and spent a lot of time on the captioning of data that gets put in while doing training.

Now you can prompt it with really complex interactions, and you can prompt it with how the camera moves, in different styles, in the interaction of characters. So that’s a big focus, number one.

Video simply has more variables than images. That means incoherence is a bigger risk and users want more control over the output. That led to a number of new features to manipulate the video outputs and ultimately influenced the training for Gen-3.

Scale Matters

Germanidis also discussed the importance of scale in training data and the training process. Greater scale led to fewer errors and reduced the incidence of incoherence.

It’s been proven on the language domain how much scaling compute can lead to a much greater range of capabilities for models. And we’re seeing the same with video.

With increased compute we saw that the model was able to learn things like geometric consistency of objects and characters — not morphing as the video progresses over time, which has been an issue with previous video generation models. We learned the prompt adherence improves quite a bit as you scale.

Customization Matters

Another element of Runway’s announcement was easy to miss, but it may turn out to be very significant. The company will now customize its models for select customers.

As part of the family of Gen-3 models, we have been collaborating and partnering with leading entertainment and media organizations to create custom versions of Gen-3.

Customization of Gen-3 models allows for more stylistically controlled and consistent characters, and targets specific artistic and narrative requirements, among other features.

Companies can request fine-tuning or a custom-trained model. This has the potential to make Runway the most customer-centric of the text-to-video model developers.

OpenAI is unlikely to offer these features, and it may not even enable customers to do it on their own. Pika seems unlikely to offer these features as it is more focused on building out editing tools. The more likely rivals in this segment will be service providers that employ Stable Video Diffusion as their foundation model. However, they will not have the intimate knowledge of the text-to-video model that you find in research teams that created the foundation models.

It is unclear how big the market is for custom text-to-video AI models. If it turns out to be a significant priority to high-volume users, Runway may be in a position to realize the value that early investors were confident of.

Competing Against Giants

Runway’s biggest challenge is competition from OpenAI, which seemingly has unlimited data, unlimited computing capacity, and a large cadre of skilled AI research scientists. It is unlikely to win on technical merit alone. That makes its move to build more creator-friendly models, and tooling a smart move. As the video imagery improves, the features beyond the prompt may well determine the user’s choice of text-to-video engines.

This will be important as it continues to grow into its valuation from late 2022.

Share

Subscribe now

Many thanks to our title sponsor.

• Dabble Lab: AI-powered bespoke software development services

The South China Morning Post reported on a new study by researchers at the National Defense University in China that created an AI-powered “virtual commander” to oversee war game exercises. While most generative AI solutions are focused on enabling productivity, the research highlights how the technology's unique capabilities can also be valuable in simulations.

This “virtual commander”, strictly confined to a laboratory at the Joint Operations College of the National Defence University in Shijiazhuang, Hebei province, mirrors the human commander in all ways, from experience to thought patterns to personality – and even their flaws.

In large-scale computer war games involving all branches of the People’s Liberation Army (PLA), the AI commander has been granted unprecedented supreme command authority, learning and growing fast in the endlessly evolving virtual wars.

…

Campaign-level military simulations often require the participation of human commanders to make on-the-spot decisions in response to unexpected events. But the number of senior PLA commanders and their availability is very limited, making it impossible for them to participate in a large number of war simulations…

The AI commander can stand in for human commanders when they cannot participate in a large-scale virtual battle or exercise command authority. Within the boundary of the laboratory, it can freely exercise this power without any interference from humans.

The study also considered how the “AI commander” could exhibit distinct personality traits or attempt to replicate the strategic and tactical styles of famous military leaders from history. You can imagine how this can also be applied to contemporary military leaders to enable troops to understand the expected style of their commanders better so they are not learning that for the first time in combat. Alternatively, it could be used to reflect the style of military leaders among adversaries for similar reasons.

Filling a Void

A 2021 analysis by the Institute for the Study of War emphasized the important role of wargaming in the Chinese military. It also pointed out that wargaming was particularly important to the Chinese because it lacked “contemporary operational experience.” In other words, China has not been in a recent “hot war” where its forces were exposed to combat situations.

The Chinese People’s Liberation Army (PLA) is faced with the challenge of preparing for future warfare during peacetime as a force that lacks contemporary operational experience. Among the methods through which the PLA seeks to enhance its combat readiness are sophisticated wargaming and realistic, force-on-force exercises. Chinese military leaders regard wargaming (bingqi tuiyan, 兵棋推演) as an important technique by which to “learn warfare from the laboratory” for training purposes and to promote insights on the dynamics of future combat.

Insight for Businesses

Given the gaps in senior leadership availability and the perceived importance of wargaming, it is logical that China’s military would turn to generative AI to enable training exercises to continue. It is the closest you can get to a simulated human working alongside you. Might it also be logical for enterprises to explore these capabilities further?

War is not business, despite the overwrought metaphors. However, militaries and enterprise-scale businesses are both large organizations that deal with competition, fields of engagement, and dynamic circumstances. Simulations are known to be valuable even if they are never exact replicas of reality. They can reveal unexpected activities or responses, proactively identify opportunities and risks, and facilitate learning among leaders, managers, and front-line personnel.

Despite this, simulations are not widely used in business. They are most common in finance, supply chain, and manufacturing. The risks associated with making errors in these disciplines have created a multi-billion-dollar market. However, simulations have typically been restricted to considering scenarios driven by structured data. There has not been a convenient way to inject subjective and unstructured data into these simulations. Scenarios that are heavily influenced by human conversation and decision-making are underserved.

Generative AI presents an opportunity for businesses to embrace simulations that involve these subjective and human elements. This can have positive impacts on the functional areas that are using simulation today and on those that have typically ignored it. For example, business strategy, marketing, sales, hiring, staffing, and many other functions can now take advantage of simulations for leanring and scenario planning.

It is important not to overestimate the ability of simulations to predict the future. They are merely a tool for becoming smarter about what could happen and what could be done. Military leaders may be showing that more pedestrian economic pursuits can also benefit from generative AI beyond the productivity revolution.

Share

Subscribe now

Many thanks to our title sponsor.

• Dabble Lab: AI-powered bespoke software development services

Apple’s Worldwide Developer Conference (WWDC) has been somewhat disappointing in recent years. The company always seemed a bit astray from the cultural zeitgeist and more focused on feature extensions than innovation. WWDC 2024 also leaned heavily on feature enhancements, but it also hit the zeitgeist much better than in past years.

There is a gap between AI interest among enterprises and everyday consumer AI use. Apple is looking to fill that gap with “AI for the rest of us.”

Apple figured out great market positioning for its AI strategy. It doesn’t have to be a world-leading innovator in AI research or advances. Instead, Apple can start by integrating AI into everyday activities. This will help consumers accumulate a lot of small AI wins while enterprises swing for the fences. Still, the features are more limited than they might appear. There are no opportunities for big wins among Apple users since the features are significantly constrained…for now.

Then again, this may be much ado about too little. The long lists of new Apple Intelligence features will only be made available starting from iPhone 15. There is likely some dependency on Apple silicon, but this is going to make many existing users unhappy. They will think it is a trick to entice users into upgrading their phones more frequently.

OpenAI understood that consumer reach and everyday use cases were its Achilles heel. Google has a significant advantage in these areas. By becoming incorporated into Siri and presumably further enhancing the ChatGPT mobile app, OpenAI is attempting to solidify its early lead among consumers while also supporting novel activities that consumers wind up using smartphones for every day.

Apple’s Key AI Announcements

Nearly everything AI at WWDC was baked into Apple Intelligence. It should be made clear that none of this will be available until the fall when the beta is released. In addition, it is unclear whether older iPhone models will support Apple Intelligence. Access may require an upgrade.

• Writing support - The stuff of all generative AI solutions, Apple will enable access to on-device generative AI-enabled writing assistants across Mail, Notes, Messages, and other apps. So, in many ways, this is just Apple beginning to catch up nearly two years after the launch of ChatGPT. This feature, by the way, does rely on OpenAI models.

• Create images - There are many image generation options, including the new Genmoji and an image generator that is limited to sketch, illustration, and animation styles. This is also a positive step and is likely to be employed a lot by users, but it is also considerably behind other options.

• Image search and dditing - Users will soon be able to search for their photos with natural language that takes into account the image content. They will also be able to add or remove elements from photos. Again, this is another set of practical use cases for consumers.

• Type to Siri - This is the most overlooked and significant of Siri’s new features. As Siri becomes more capable, the incidence of requests that will be better suited to typing will rise. Text chats with Siri are a very welcome addition to the feature set.

• App training / help - Ask Siri questions about how to use an app for a particular use case, and step-by-step instructions will be returned.

• Context maintenance - This will happen within Siri conversations and is long overdue. Rival voice assistants have had this for more than six years.

• Personal Context - This is new and largely novel among voice assistants. Grounding is simply personalizing the user experience based on what information exists in other apps.

• Screen awareness - Siri will be able to identify information on the user’s screen and take actions related to it.

• Actions in and across apps - Siri is the original “Do Engine.” While ChatGPT knows many things, it can do few things outside the knowledge domain. Siri has always been about taking action, but it never knew much. So combining the two capabilities in a logical pairing.

• Third-party apps get native generative AI - Developers will also get access to APIs so their apps can take advantage of the features. However, these are limited to the writing API, presumably from OpenAI, the Image Playground, and Genmoji.

• The more controversial element - App Intents are a critical way for users to leverage generative AI through Siri. However, there are only a handful of use cases supported, such as search and integrations with email, photos, and videos. If generative AI doesn’t work, Siri shortcuts might be able to fulfill the request.

This list was underwhelming from the perspective of providing an open and flexible set of generative AI-based services. At the same time, it is strong from the perspective of dozens of incremental benefits to user experience.

The Good and Bad for OpenAI

The Apple alliance is useful for OpenAI and may even be critical to its competitive positioning. However, the “partnership” appeared to be very thin. There are some legitimate positive and negative elements from OpenAI’s perspective.

Positive

• Access and relevance

• Blocked Google from that role

• Potential for access to everyday activities

• Wider use of OpenAI’s API for search, text, and images

Negative

• Siri decides when to call ChatGPT

• The deal is not exclusive

The fact that Siri acts as an intermediary between the user and ChatGPT is not necessarily a good strategy. This means that Siri has total control over the conversation. Eventually, that will create issues where additional Siri upgrades are available only for the wealthy people who upgrade their phones frequently.

Apple’s Craig Frederighi also indicated, “We also intend to add support for other AI models in the future.” This seems to confirm that the OpenAI deal is not exclusive. You might see Gemini or Claude holding a similar position as OpenAI later this year. You are almost certain to see a new Apple-developed model. OpenAI may have got to the table first, but they will not be the least to eat off the Apple table.

Still, this is a big win for OpenAI. The benefits are likely to be more promotional than deeply substantive, but OpenAI kept Google from some generative AI use cases on the most important mobile platform. If OpenAI still fancies itself as a consumer-oriented business, then the deal with Microsoft is the first of several steps it is likely to make over the next two months.

Share

Subscribe now

Many thanks to our title sponsor.

• Dabble Lab: AI-powered bespoke software development services

We’ve heard this all before. AI doomers versus AI boomers. In early 2023, the doomers were ascendant. They generally promote the thesis that AI presents an existential threat. Whether or not it's imminent, the disaster will come so suddenly, they say, that we must act preemptively to save humanity.

AI doomers are often technology industry celebrities or hangers-ons who like to be seen as close to the innovation curve. They are not monolithic, as there is some degree of difference in their perceived threat. However, they come off as monolithic and “aligned” around activities such as “pausing” giant AI and ramping up government regulation.

Elon Musk, Geoffry Hinton, Joshua Bengio, Sam Harris, Gary Marcus, and many others are good for a frightening headline about impending doom on a regular basis. OpenAI’s Ilya Sutskever is at least an honorary member of this group. So are a couple of the former OpenAI board members who were ousted after they fired Sam Altman in November 2023. Some people used to put Sam Altman in this category, but the doomer purity clan has decided he was a boomer in disguise.

The clear leader of the boomer group seems to be Yan LeCun, who, along with Hinton and Bengio, is the Turing Award winner for contributions to the AI field. However, I’m not sure this is even correct. He is the most outspoken and critical of the doomer crowd, but he seems more anti-doomer than a true AI boomer. Microsoft’s Satya Nadella, AI legend Andrew Ng, elite venture capitalist Mark Andreessen, Jeff Bezos, and many venture-funded startup CEOs are generally the staunchest prophets of inevitable AI benefits.

Against this backdrop, we were treated to several stories over the past two weeks about OpenAI’s abandonment of AI safety as a principle. This created a vehicle for doomer lament, but frankly, these incidents seemed more like an opportunity for ousted and marginalized OpenAI employees to repair their reputation or to signal to fellow doomers that the fight was not over.

Helen Toner

Helen Toner became famous as one of the OpenAI board members who voted to oust Sam Altman as CEO in November 2023. As the saga unfolded, her background attracted heightened scrutiny, particularly in terms of her limited professional experience and a paper she had written critical of OpenAI while she was serving as a board member. She suggested in a recent podcast interview that she was unhappy about Sam’s reaction:

The paper came out Sam started lying to other board members in order to try and push me off the board.

Another grievance was that Altman didn’t inform the board before releasing ChatGPT.

When ChatGPT came out in November 2022, the board was not informed in advance. We learned about ChatGPT on Twitter.

Toner also talks about their concerns about Altman’s control over the OpenAI fund and what she considered a pattern of deceit that undermined the board’s trust in the leadership’s approach to AI safety. She also referenced a complaint about a “toxic atmosphere” at OpenAI.

In retrospect, it seems hard for Toner to justify the drastic decision. She says the board came to a conclusion based on a series of infractions that individually may have been innocuous but together raised concerns.

However, these remarks make it harder to rebut the critiques that she was simply unqualified for the responsibility. She accuses Sam of lying about her and working to remove her from the board. This suggests she had a personal interest in removing Altman from the board before he succeeded in removing her.

Toner also seems miffed that the management team didn’t tell them about a beta product release. Sure, ChatGPT became a cultural phenomenon, but there was no way to know that before it was released. Oversight boards typically do not micromanage product releases. ChatGPT is clearly not AGI, even by broad definitions, nor does it pose any existential threat to humanity. What is the issue?

If Toner had personal reasons that shaded her thinking about OpenAI and Sam Altman in November 2023, those were only amplified in the aftermath of the management shake-up debacle. Her podcast comments seem like a campaign to reset the narrative with her as both a hero and a victim for trying to do the right thing.

Jan Leike

Toner’s podcast appearance seemed to set the stage for Jan Leike’s OpenAI departure and his disparaging remarks about the company’s approach to AI safety. Leike wrote on X.

Yesterday was my last day as head of alignment, superalignment lead, and executive @OpenAI… Stepping away from this job has been one of the hardest things I have ever done, because we urgently need to figure out how to steer and control AI systems much smarter than us.

I joined because I thought OpenAI would be the best place in the world to do this research. However, I have been disagreeing with OpenAI leadership about the company's core priorities for quite some time, until we finally reached a breaking point.

I believe much more of our bandwidth should be spent getting ready for the next generations of models, on security, monitoring, preparedness, safety, adversarial robustness, (super)alignment, confidentiality, societal impact, and related topics.

These problems are quite hard to get right, and I am concerned we aren't on a trajectory to get there.

Over the past few months my team has been sailing against the wind. Sometimes we were struggling for compute and it was getting harder and harder to get this crucial research done.

But over the past years, safety culture and processes have taken a backseat to shiny products.

Many employees depart companies, and some make critical remarks about their former employers. However, why should we believe their opinions are akin to facts? His comments were tailor-made for juicy headlines. There is just no evidence other than his opinion that his views represent the entire story. That said, he did get largely favorable news coverage, which raised his profile as someone who was with OpenAI, was fighting for AI safety in opposition to the goals of company management, and is on the side of AI safety.

The difficulty here is that it is hard to know who is right. Employees who aren’t successful in persuading their manager to push a specific objective become enamored of themselves as whistleblowers with truths that must be heard. Sometimes, those truths turn out to be false. Other times, the conflicts are wars of opinion.

Regardless, comments like those of Toner and Leike quickly received backing from the AI doomers. They already believe OpenAI is bad and largely uncontrollable. The latest comments from former insiders are seen as a verification of their existing negative sentiment. Therefore, they must be right.

Tactical Mistakes

The doomers made several tactical mistakes in pushing their AI pause and related concerns. That assumes their actual goal is AI safety more than posturing and headline hunting.

• Exposing self-interest - Many vocal AI doomers have a financial interest in slowing down OpenAI. Others benefit from additional media mentions.

• Attacking open-source - Open-source makes it hard to control access and use. Many doomers embraced this and lost the backing of open-source developers. However, their goal to control AI model access and use demands that open-source projects are curtailed. This stance is unpopular with many people that don’t believe AI should be controlled by a few large technology companies.

• Making claims that frighten without offering empirical evidence - It is also problematic that AI doomers don’t have evidence for their claims of impending disaster. The public eventually wants evidence that the approach and policy are true and practical.

These tactical mistakes have made it harder for doomers to make progress. That means they must look to negative commentary from former employees to serve as new “smoking guns” justifying action now.

The Real AI Safety

It is difficult to say whether the doomer crowd is helping or hurting the AI safety cause. AI safety has many important elements that don’t raise the specter of imminent human extinction but instead pertain to more likely near-term harms. However, it is hard for these perfectly reasonable concerns to generate significant discussion because they pale in comparison with a human extinction event.

There is a view among doomers that they alone are warning the world of AI risks that must be taken seriously, and everyone else is in it for greed. However, we have seen that many doomers also benefit from peer-group prestige, and some are in line to benefit monetarily if their desires around AI pause and regulations are followed. Earnest warnings are undermined by what too often appear to be self-serving interests or over-the-top fear-mongering. Messianic fervor doesn’t guarantee persuasion.

What about considering the likelihood of risk? What about providing empirical evidence of the risks they believe are so certain? What about scenario planning? What about making incremental progress?

Yan LeCun offers an interesting perspective that few doomers seem to grasp. What if the singularity is not likely to be a sudden event and instead could only happen over a period of time with various stages of development? According to LeCun, the history of technology innovation and AI innovation follows that pattern far better than a sudden transition hypothesis. If this is true, then the world should have ample time to enact safeguards as AI progresses and introduces new levels of risk.

Share

Subscribe now

Many thanks to our title sponsor.

• Dabble Lab: AI-powered bespoke software development services

Tektonic announced a new $10 million funding round yesterday, led by Madrona and Point72 Ventures. The company has managed to incorporate generative, neural, and symbolic AI alongside agents into its core messaging. You might come away thinking Tektonic has something for everyone. However, it appears that symbolic AI is the real workhorse of the solution, and maybe a simple rules engine is really at its core.

Agents Everywhere

Many startups are hesitant to talk about assistants and copilots since that places them in the crosshairs of OpenAI and Microsoft. Both companies are looking like the juggernauts of the generative AI era. Then you have Google and its large customer base. If you’d prefer to avoid direct competition with the big three, then AI agents is a good bet. If your leadership has a background in robotic process automation (RPA), agents offer even better odds of landing new funding.

Tektonic’s generative AI and neural elements are focused on intent identification and developing a fulfillment plan. Neural refers to natural language understanding (NLU) or the broader natural language processing (NLP) option when generative is not the preferred option for query intake.

The term Rules is coupled with Symbolic AI to offer up a callback to expert systems and is likely designed to insert an AI sheen on top of RPA. In some ways, it’s a more honest representation of agents than other agent vendors. Many are simply rules engines trying to pass as AI. Take a look at Rabbit, its procedural scripts, and the complete absence of anything resembling an AI large action model (LAM).

I am not saying that Tektonic is devoid of AI or is simply repackaged RPA. Nor am I suggesting RPA coupled with large language models (LLM) is somehow inferior. The coupling is logical and useful in many situations. RPA enables LLMs to execute actions and not just churn out knowledge. However, claims around AI agents have a checkered history. They have typically delivered less than advertised and been underwhelming from a technology perspective.

What is an AI Agent

An AI agent has several important characteristics, including autonomy, adaptability, and decision-making capabilities. Autonomy is the ability to execute a task on behalf of a person or system without their direct supervision. Adaptability speaks to the capability to navigate a new or changing digital environment to execute an objective without explicit instructions on how to fulfill the request. Decision-making refers to the AI agent’s ability to choose between alternatives in order to best fulfill the user or system objective.

RPA can do none of these things. It is a procedural program with imperative instructions. If the target digital services introduce changes to their APIs or user interfaces, the RPA bot typically needs to be rewritten. These bots cannot adapt and do not make decisions. Beware of RPA in AI agent clothing.

Does Tektonic Have Agents

As I mentioned in the article regarding MultiOn’s funding, AI agents are more similar to declarative instructions that state an objective and not how to fulfill it. It could be that MultiOn also uses rules engines and RPA, but it looks like the stated direction is an autonomous, adaptable decision-maker. Tektonic may get there as well. However, the way the company talks about its solution, it is clear that autonomy and delegated decision-making are not priorities.

Tektonic’s approach involves using GenAI Agents to augment employees with contextual information, guidance, and simplified actions through natural interfaces. We believe the best outcomes come from employees working interactively with the Agents, providing feedback, making decisions, and supervising. This contrasts with other approaches that focus on automating tasks using fully autonomous GenAI Agents. But when these Agents are tasked with solving complex problems and left unsupervised, they prove to be generally unreliable, prone to making poor decisions, compound mistakes, and are ultimately unable to guarantee improved outcomes.

To fulfill their promise, GenAI Agents need to work alongside humans, constrained by business rules, and paired with deterministic software. The result is an AI-augmented work environment, where human abilities are amplified, leading to better outcomes and job satisfaction.

Tektonic may be right that the risks and inconsistency of generative AI require an alternative approach. However, its approach appears to lack ambition. Add a natural language interface in front of a rules engine with some procedure-driven RPA bots to fulfill tasks and move along. This is unsurprising as the company founder is a former executive at UiPath, the market-leading RPA vendor. My expectation is that the company’s solutions will be very light on the benefits of generative AI, and its sales messages will balance between saying it is AI but the more reliable kind.

Does RPA Have a Place in the AI Era

As a final point, the intent of this discussion is not to disparage RPA. In fact, I suspect RPA will enjoy a renaissance as a helper to generative AI assistants and AI agents. When you think of function calling, you should think of an RPA bot as a function or tool that can be called to fulfill a task.

AI agents are novel in that they don’t require all the upfront detailed requirements and design to add value. However, there will be times when a quick function call that follows a step-by-step procedure will be useful and potentially optimal. RPA may also be the fastest and least expensive option in many instances. Just don’t confuse RPA and rules engines as AI, even when coupled with terms like Symbolic AI. Symbolic AI is AI, but not when its a rules engine.

Share

Subscribe now

Thank you to Synthedia’s sponsor:

• Dabble Lab: AI-powered bespoke software development services

Synthedia’s regular readers will be familiar with the difference between knowing and doing assistants. ChatGPT, Gemini, Claude, and other generative AI-enabled assistants know many things. This enables them to fulfill knowledge-based tasks at a quality level that did not exist before 2022. By contrast, voice assistants such as Alexa, Google Assistant, and Siri can do a lot of things. They excel at tasks that control applications.

Siri and Viv Labs co-founder Adam Cheyer was the first to propose the knowing-doing divide among assistants. Synthedia has recently added a “connecting” capability to this framework. However, for today, let’s stick with the original knowing-doing segmentation, as these represent the capabilities users most want from digital assistants.

The divide exists because doing assistants have historically been lacking in the knowing department, and the newer knowing assistants can’t do much in the way of controlling applications. Agents may be the most effecting approach to filling this capability gap for knowing assistants, and ultimately enhance what doing assistants can accomplish.

MultiOn Agents

The Information reported that MultiOn will announce a $20 million funding round at a $100 million valuation. That is a rapid valuation rise just 12 months after the company was founded and less than six months after it announced its first product. MultiOn is not a foundation model company. Instead, it customizes open-source large language models (LLM) and small language models (SLM) to execute tasks that require application control.

General Catalyst is leading the round, with participation from Forerunner Ventures and Blitzscaling Ventures. Amazon participated in MultiOn’s 2023 seed round and has contributed to the most recent funding.

According to a person involved in the round, Amazon and MultiOn have held discussions in the past few months about whether MultiOn’s agent tech could make Alexa voice assistant more useful.

From Knowing to Doing

LLMs represent a probabilistic technology that is not necessarily suited to reliably executing tasks in applications. Controlling applications often requires specific steps and data formats to succeed. This is better aligned with deterministic than probabilistic approaches because consistency and repeatability matter. However, MultiOn and other agent development environments are training smaller models to navigate websites independently and, presumably, consistently. What makes AI agents different from procedural automation is that they can negotiate variability.

The strategy to use LLMs with agents is designed to combine the knowing and doing elements that create a well-rounded assistant. The LLM determines user intent and then calls on the AI agent most likely to fulfill the request. The AI agent then performs the “doing” request.

Granted, you could go straight to the specialty agent as a user, but that may mean you need a separate agent for every task or task category and would impose a significant cognitive load. Why? Users would need to remember every agent and what tasks they could perform. This may be acceptable for 5-10 different tasks. It is less practical if tasks reach into the dozens.

LLM-enabled assistants, particularly those backed by frontier AI foundation models, have proven versatile at fulfilling a broad range of knowing tasks for users. Many have also added tool-calling or function-calling capabilities. This approach leverages LLM strengths in determining user intent and then passing off the “doing” request to a tool or agent designed for that purpose. It is likely the fastest way for LLM-backed knowing assistants to accumulate broad access to doing skills.

It is also why a young startup can command a nine-figure valuation so soon after its founding and first product launch. The knowing-doing gap is a meaningful shortcoming that the popular knowing assistants have not yet addressed. Agents also provide the opportunity for traditional doing assistants that rely on coding predetermined procedural steps to adopt goal-seeking agents that figure out how to navigate applications to fulfill a request. If MultiOn can fill this gap for generative AI-based assistants and lands a customer like Amazon for Alexa in the doing assistant category, it is poised for rapid growth.

MultiOn already counts Amazon’s Alexa Fund as an investor. Synthedia also covered recent news that suggests Amazon is working on an Alexa upgrade that goes beyond the announcements from its fall 2023 product event. Adding agents, along with more LLM-enabled knowing features, is a logical strategy move by Amazon. Consider that the Alexa Fund’s Paul Berard commented in January 2024:

MultiOn’s AI Agent is impressive with its ability to connect with virtually any device or interface on the internet. We’re proud to be an investor in MultiOn and fully support their mission of simplifying customers’ lives through AI Agents.”

MultiOn Examples

MultiOn launched a playground in late May 2024 that provides several examples of agent capabilities. Some of these include adding a book to an Amazon shopping cart, looking up the weather on a website, reserving a table at a restaurant, and summarizing the news.

Many of these features existed in the earlier era doing assistants like Alexa and were target use cases for Google Duplex, which was originally demonstrated in 2018. In other words, the use cases are not entirely novel. The approach to fulfilling them is new and long anticipated … provided the agents work as advertised.

I add this note of caution because we have heard many stories about so-called AI agents in the past that turned out to be more brittle, less flexible, and less automated than promised. Google Duplex is a case in point. It’s use case fulfillment for restaurant reservations and salon appointment bookings often involved human intervention.

Rabbit Runs, Agents Negotiate

More recently, Rabbit announced the large action model (LAM) to tackle similar tasks as MultiOn is offering with agents. However, Rabbit appears to use procedural code and a rules engine to execute some basic tasks. Developers have not found evidence of an AI model other than OpenAI models. According to a recent Android Authority article:

Software developers @xyz3va, @schlizzawg, and @MarcelD505 have dug into the servers powering the Rabbit R1. One of the key findings is that the servers running the AI gadget’s interactions apparently aren’t running a Large Action Model (LAM) as previously claimed.

…@xyz3va scoured through the server’s code and claimed that “the LAM is also not an LAM.” She asserts that the company isn’t using AI to figure out app interactions but is instead using a “hardcoded list of locations” to tell the app where to click. This would contradict Rabbit’s claims that Rabbit OS and the Rabbit R1 use an LAM to handle interactions.

Other developers discovered in the source code that Rabbit OS is employing Playwright automation scripts. The software is designed to automate application testing, and it turns out it can also be used to control applications using hardcoded procedures.

The LAM is supposed to be a “super agent” that enables a lot of “doing” tasks in one package and does not require hard-coded scripts to control applications. The ambition of one model to navigate multiple use cases may be a key reason why it has not yet materialized. Or, it may be that the Rabbit team has not yet embarked on LAM development and took a shortcut to bring its first product to market.

Regardless, MultiOn agents take a more practical approach by limiting task scope. This significantly increases the likelihood that an AI model can execute a task that it has been trained on conceptually, even if it has not previously seen the application and its required steps. Narrowing the scope does not ensure success, but it does improve your odds.

Developers might consider what Alexa and Siri do for application control as an imperative programming approach. Rabbit, despite its original claims, is also using an imperative approach to task execution by employing Playwright scripts. The steps are defined and executed in a precise order that is expected to yield the objective. If the application changes, the scripts almost certainly need to change. This makes maintenance a challenge if the objective is to support many apps.

Agents like MultiOn are more similar to a declarative model. The objective is declared, and the solution determines the steps to fulfill the request. The most prominent example of declarative programming for digital assistants is Samsung’s Bixby, which was built on top of Viv Labs technology. The Alexa Conversations developer feature also took this approach. These weren’t agents per se, but MultiOn’s agent approach is a similar to a declarative model. The AI models powering the agents are tasked with determining the best way to achieve the objective.

What’s Next

AI Agents are a key tool that will augment the benefits users already capture from assistants powered by AI foundation models. Agents are the tools that will deliver task execution to knowing assistants and expand the capabilities of existing doing assistants.

MultiOn is among the first wave of AI agents and is benefitting from early market entry and high-profile investors. The AI agent segment will quickly become very crowded, and foundation model providers such as OpenAI and Google will also become competitors. As a result, it will be critical for MultiOn to scale rapidly. The funding round and the presence of Blitzscaling Ventures among its new investors look well-timed.

Share

Subscribe now

Thank you to Synthedia’s sponsors:

• Dabble Lab: AI-powered bespoke software development services

• Loccus.ai: AI for protecting against voice fraud

Reuters reported today that Cohere has closed a $450 million funding round that included investments from NVIDIA, Salesforce, and Cisco. According to Reuters sources, the funding supports a new valuation of $5 billion, more than double its $2.2 billion valuation in June 2023.

The leaked news was not confirmed by Cohere or its leading investors at the time of publication. However, the source indicated that the company hit an annualized revenue run rate of $35 million in March (about $3 million per month), nearly three times as much as the $13 million run rate (just over $1 million per month) at this time last year.

LLM Wars

Founded in 2019, Cohere is among the earliest developers of commercial large language models (LLM). It has been overshadowed by OpenAI, Anthropic, and Google in the proprietary LLM segment. However, while rivals have focused on building general purpose AI foundation models, Cohere has sought differentiation in deploying models customized to specific use cases are technology architectures. For example:

• Command R focuses on retrieving augmented generation (RAG) “at scale.” This improves response accuracy by grounding answer retrieval in known datasets.

• Embed is designed to improve LLM response accuracy by .

• Rerank is a founcaiton model designed to enhance semantic search and deliver cost and latency optimization for RAG architectures.

It is a reasonable strategy to focus on niche solutions when your primary competitors in the general-purpose model segment are tech giants and funded startups with valuations four to twenty times higher. By definition, niche segments represent smaller addressable markets than general-purpose solutions. However, niches are also attractive as they make it easier to align solutions around specific value propositions.

What’s Next

The fact that Cohere is on pace to generate $36 million over the next twelve months shows significant progress. Another tripling of revenue over the next year, will indicate strong momentum. This is well behind OpenAI’s multi-billion dollar revenue engine, but Cohere is competing in the solution optimization market. Cohere’s key risks are:

1. General purpose frontier and open-sourse models continue to improve in response quality, latency, and cost.

2. Adoption of advanced generative AI solutions are delayed.

The first point would result in Cohere’s solution optimizations becoming less differentiated. The second reflects the fact that Cohere is focused on system optimization when much of the enterprise market is oriented around deploying solutions that work and many are not thinking yet about optimization.

Another issue Cohere could face is the rise of the data-centric RAG optimization solutions. Databricks and Snowflake now have LLMs combined with their data products. While their data products are general-purpose, their entry into generative AI is focused on a mix of convenience, performance, and optimization for RAG-related solutions.

Still, the valuation and revenue progress are promising. Generative AI represents a very large and rapidly growing market. There is room for many winners and some of the niches will be sizeable in their own right.

The funding round also shows that investor enthusiasm in generative AI foundation models remains strong.

Share

Subscribe now

Thank you to Synthedia’s sponsors:

• Dabble Lab: AI-powered bespoke software development services

• Loccus.ai: AI for protecting against voice fraud

OpenAI closed three new data and content deals last week with news publishers NewsCorp, Vox, and The Atlantic. This follows earlier deals with The Associated Press, Axel Springer, LeMonde, Prisa, and The Financial Times. Each of the deals enables OpenAI to utilize the publishers’ content catalogs to train their AI foundation models. They also provide access to the current news via OpenAI’s ChatGPT. Publishers receive fees for access to their content and ensure relevance with consumers if generative AI chat assistants become an important channel for news consumption.

News Corp

The NewsCorp agreement is the most substantial in terms of content depth and breadth. According to the announcement:

OpenAI has permission to display content from News Corp mastheads in response to user questions and to enhance its products…OpenAI will receive access to current and archived content from News Corp’s major news and information publications, including The Wall Street Journal, Barron’s, MarketWatch, Investor’s Business Daily, FN, and New York Post; The Times, The Sunday Times and The Sun; The Australian, news.com.au, The Daily Telegraph, The Courier Mail, The Advertiser, and Herald Sun; and others…

In addition to providing content, News Corp will share journalistic expertise to help ensure the highest journalism standards are present across OpenAI’s offering. 

There was no explicit mention of NewsCorp receiving compensation from OpenAI or plans to use OpenAI tools internally. However, those were elements of earlier deals and are likely included in the “multi-year agreement.”

Vox Media

Vox Media positioned its OpenAI agreement as “strategic.” In addition to providing access to content, Vox says it plans to collaborate with OpenAI to launch “innovative products” and integrate the technology with the Forte data platform.

Vox Media’s respected portfolio of properties including Vox, The Verge, Eater, New York Magazine, The Cut, Vulture, and SB Nation, will help inform ChatGPT’s 100 million users, receiving brand attribution and audience referrals. The two companies will also collaborate using OpenAI’s technology to develop innovative products for Vox Media’s consumers and advertising partners…

Additionally, OpenAI will enhance its technology with Vox Media’s archives, which contain a deep well of reliable and accountable information and journalism…Vox Media will also use OpenAI technology to extend the leadership of Forte, its first-party data platform…Advertisers will benefit from the OpenAI partnership through even stronger creative optimization and audience segment targeting, leading to even higher campaign performance.

The Atlantic

The Atlantic also positioned the OpenAI agreement as “strategic.” The publisher intends to launch a new microsite that showcases its AI products, including collaboration with OpenAI.

The Atlantic’s articles will be discoverable within OpenAI’s products, including ChatGPT, and as a partner, The Atlantic will help to shape how news is surfaced and presented in future real-time discovery products. Queries that surface The Atlantic will include attribution and a link to read the full article on theatlantic.com(opens in a new window).

As part of this agreement, The Atlantic and OpenAI are also collaborating on product and tech: The Atlantic’s product team will have privileged access to OpenAI tech, give feedback, and share use-cases to shape and improve future news experiences in ChatGPT and other OpenAI products. The Atlantic is currently developing an experimental microsite, called Atlantic Labs, to figure out how AI can help in the development of new products and features to better serve its journalism and readers––and will pilot OpenAI’s and other emerging tech into this work.

What it Means

OpenAI has three objectives driving these deals. First, it needs high-quality data to train its AI foundation models. The race among AI model developers is transitioning from competition around accumulating more data to the quest for higher-quality data. Data quality has been a key driver in recent benchmark performance improvements of several AI foundation models.

Second, OpenAI would like to provide a better user experience for ChatGPT subscribers. The company believes it can grow the current 100 million weekly active users and increase usage frequency by adding more timely news coverage with attribution links. This puts OpenAI on a collision course with Google and the company’s key patron, Microsoft.

Finally, each licensing deal ensures that OpenAI will not be sued for its previous use of content from the publishers in training its earlier models. Though the company believes it had the right to use that public data to train its models, these agreements serve the purpose of reducing the risk of future lawsuits.

This is also more bad news for The New York Times. Instead of cutting a deal with OpenAI, the publisher filed a lawsuit. News publishers’ content varies in quality, and some have niche reporting that may be unique. However, publishers largely deliver similar news. In addition, OpenAI is primarily interested in high-quality content to train its models. Given the breadth of deals that OpenAI has already cut, the value of a deal with The New York Times has surely diminished. OpenAI may still attempt to strike an agreement, but its incentives to bend to the publisher’s demands lessen with every new publisher deal it inks.

Share

Subscribe now

• Dabble Lab: AI-powered bespoke software development services

• Loccus.ai: AI for protecting against voice fraud

Google is developing a habit of releasing research papers and then simply adding information to the versions hosted in Deepmind servers without versioning. This makes it harder to know when the information was added and what was added. During Google I/O, the company showed off the latest Gemini 1.5 Pro and Flash large language models (LLM) but didn’t mention new research data in its Gemini 1.5 paper. However, you can see the magnitude of the change because the original paper was also uploaded to arXiv.org.

An April 25, 2024, version of Google’s research paper titled “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” hosted on arXiv.org, has a total of 77 pages. The link to the paper with that same name hosted on Google Deepmind servers is 153 pages.

This decision to withhold some of the research from the initial paper publication is understandable. The expanded version includes information about the Gemini 1.5 Flash LLM that Google likely did not want to share publicly before the I/O developer conference. With that said, the paper doubled in length over two weeks and added a lot more than additions for the 1.5 Flash model. Some of these changes are particularly notable and it seems odd that Google isn’t driving more attention to it.

Better Reasoning

One of the most interesting sections is Advancing Mathematical Reasoning. The paper discusses a math-specialized model that includes addition inference time for complex problems.

In training Gemini 1.5 Pro, we want to understand how far we can push the quantitative reasoning capabilities of LLMs, with the goal of solving increasingly challenging and open-ended problems. Solving such problems typically requires extensive domain-specific study. In addition, mathematicians often benefit from extended periods of contemplation while formulating solutions; written mathematics tends to focus on the final result, obscuring the rich thought processes that precede it…We aim to emulate this by training a math-specialized model and providing it additional inference time computation, allowing it to explore a wider range of possibilities.

The result was a significant improvement in math-related benchmarks and tests. It is unclear what proportion of the improvement should be attributed to additional training as opposed to the longer inference time. However, the net result is notable.

The Gemini 1-5 Pro model scored a respectable 67.7 on the MATH benchmark, which landed it halfway between the scores of Anthropic’s Claude 3 Opus and OpenAI’s GPT-4 Turbo. Math-Specialized 1.5 Pro model scores well above both models in the MATH, AIME 2024, Math Odyssey, HiddenMath, and IMO-Bench benchmarks. Google has taken its share of criticism for trailing OpenAI and Anthropic in a technology segment it invented, but we are now seeing the combined Deepmind and Google AI innovation engine showing off its capabilities.

Instruction Following

A topic that was significantly expanded between the paper’s April and May editions was Instruction Following. The first two sentences of the sections are the same. They then diverge around the number of customer prompts developed by human rates regarding “real-world use cases.” The April edition included 406 prompts relating to “generating formal and creative content, providing recommendations, summarizing and rewriting texts, and solving coding and logical tasks.” By May, that list had grown to 1,326 prompts and added several categories.

In addition, they capture enterprise tasks such as information extraction, data/table understanding, and multi-document summarization. These prompts are long, 307 words on average. They have between one to tens of instructions with a mean of about 8. Different from the Gemini 1.0 Technical Report (Gemini-Team et al., 2023), we also use another set of 406 prompts from human raters that covers varied topics and instruction types. These prompts are shorter, 66 words on average, with one to more than a dozen instructions (average count is five).

For evaluation, human annotators were asked to rate whether a response follows (or not) each of the instructions present in the prompt. We aggregate these human judgements into two metrics: per-instruction accuracy (the percentage of instructions over the full evaluation set that are followed) and full-response accuracy (percentage of prompts where every instruction was followed).

Our results for the two prompt sets are shown in Table 13. The Gemini 1.5 models show strong gains on the set of 1326 long and enterprise prompts: the 1.5 Pro model has 32% improvement in response accuracy from the 1.0 Pro model, by fully following 59% of all the long prompts. Even the smaller 1.5 Flash model has a 24% increase here. At instruction-level, the 1.5 Pro model reaches 90% accuracy.

For the set of 406 shorter prompts, the Gemini 1.5 models follow 86-87% of the diverse instructions. 65% of the prompts were fully followed, performing similar to Gemini 1.0 Pro.

Long-tail GenAI Tasks

One more area to highlight is the addition of data around long-tail instructions. These exercises test LLMs on their ability to answer expert-level questions.

In Expertise QA, we engage with in-house experts that have formal training and experience in various domains (e.g., history, literature, psychology). They produced hard, sometimes complex questions (e.g., As films began using sound for dialogue, how did the changing use of visual metaphor affect the ways audiences expected narratives to develop in movies? or How does Vygotsky’s theory of internal speech apply across multilingual language acquisition?).

The same experts then rated and ranked model responses to their respective questions. Models were evaluated according to their ability to answer such questions with a high degree of accuracy, but also, secondarily, completeness and informativeness…Gemini 1.5 models significantly and strongly outperform 1.0 Pro on the this task.

You can see from Figure 18 above that Gemini 1.5 and 1.5 Flash significantly outperformed the Gemini 1.0 Pro and Ultra models. Just as important, instances of “severe inaccuracies,” commonly referred to as hallucinations, declined significantly.

These data help illuminate one of the mysteries revolving around Gemini 1.5. Why did it arrive so quickly after the limited introduction of the 1.0 model family, and why has the Ultra model not been released? Google wanted to release Gemini before year-end, and any inclination to delay until 2024 was set aside after OpenAI had its management shake-up debacle in November 2023.

The 1.5 model release was likely about more than just hitting the million token context window. It was the more accurate and reliable model that Google wanted to release as Gemini’s debut, but their hand was forced to move more quickly.

Google’s Progress

Other additions to the updated research are worth reviewing. Improvements in tool use, STEM topics, translation, and video interpretation are among the new content that Google quietly inserted into the paper, which more than doubled in length between late April and early May 2024.

Google has been widely criticized for its missteps and inadvertent role as a follower in the market. Some of those critiques were warranted, and some of the issues were self-inflicted. However, the issues may be overshadowing substantial progress on the technology innovation front. OpenAI and Anthropic have carved out technology leadership roles in the frontier AI foundation model segment. It looks like Google now has a seat at the table. More choices among leading frontier models will be better for users and incentivize the technology leaders to keep innovating.

Share

Subscribe now

• Dabble Lab: AI-powered bespoke software development services

• Loccus.ai: AI for protecting against voice fraud

AI foundation models, whether they are large language models (LLM) or multimodal models with capabilities extending to audio, video, and images, rely on data to work their magic. Scale AI’s founder, Alexandr Wang, commented in a recent blog:

In this construction, NVIDIA, other chip makers, and the cloud hyperscalers are focused on “compute.” Foundation model developers are responsible for “algorithms.” Companies like Scale live in the world of data.

However, when considering data and AI, it is useful to break that segment down further. One way to segment the solution space in relation to AI is to consider whether the feature focus relates to using, storing, labeling, collecting, or generating data.

Foundation models use data. Data lakes and databases store data. Scale AI and competitors, such as Snorkel, Appen, and TELUS International, label, collect, and generate data. This is a fiercely competitive segment that serves the needs of foundation model developers, teams looking to fine-tune foundation models, automobile manufacturers, application developers, and numerous other buyers. It is also differentiated in that automation is often coupled with humans in the loop.

The Big Round

The rise of AI for machine learning applications and self-driving had already created outsized demand for data labeling, collection, and generation services before 2022. Generative AI’s expansion over the past 18 months has amplified this demand, and Scale’s recent funding round is evidence of the data segment’s momentum.

Scale AI announced last week that it had raised $1 billion in new funding at a $14 billion valuation. That is nearly double the company’s $7.3 billion valuation from 2021 and three times the $3.5 billion in 2020. The round was led by Accel and included a mix of prominent venture capital and private equity firms and tech giants, including Amazon, Meta, NVIDIA, Qualcomm, and ServiceNow.

The funding round and valuation increase were driven by strong revenue growth, particularly in the generative AI segment. A Fortune interview with Wang revealed:

Scale AI provides human workers and software services that help companies label and test data for AI model training, a critical step in getting AI to be effective. For Scale AI, that business is growing quickly as corporate customers race to adopt products related to generative AI—so much so that a whopping 90% of its business is now driven by that spending on that subset of AI.

In an exclusive interview with Fortune, Scale AI CEO Alexandr Wang shared previously undisclosed details about the company’s financials that show just how quickly that growth is happening. The company’s annual recurring revenue—the money paid by businesses for Scale AI's services over extended periods of time—tripled in 2023 to an undisclosed amount and is expected to reach $1.4 billion by the end of 2024, he said.

Wang told Fortune that growth was 200% year-over-year, which suggests that revenue in 2023 was about $465 billion, though some sources report it was $600-$700 million. He also said that Scale AI would be profitable by year-end, and the company was preparing for an IPO, though he did not indicate the timing.

Some of Scale AI’s competitors have already achieved profitability. However, investors have been attracted to the company’s strong growth rates and high-profile customers among generative AI foundation model builders, automakers, and U.S. government agencies.

From Services to Products

Another key shift in Scale AI’s business model is from a completely services-oriented model to a product model. While the company previously focused on collecting and annotating data using human raters, it has been developing curated datasets annotated by domain experts to sell to developers and companies for AI model fine-tuning. This presents an opportunity to drive higher margins from product sales.

Scale AI is also developing tools to evaluate models and applications based on these curated datasets. This will enable the company to provide value beyond serving as a data collection and labeling vendor.

What’s Next

Foundation model quality improvements were initially driven by increasing parameter counts. That was followed by a combination of novel architectures and significantly larger training datasets. More recently, foundational model improvements have been attributed to higher quality datasets that include curated data by knowledge domain and verified by experts in law, medicine, science, mathematics, and other disciplines.

The next phase of development is likely to be a rise in more sophisticated model fine-tuning by enterprise users and developers of domain-optimized models. These data users start with general-purpose foundation models and enhance them with additional training that relies on high quality domain-expert reviewed datasets.

However, these fine-tuning customers often have far smaller training budgets compared to the leading foundation model developers. Many will find it cost-prohibitive to pay for large-scale datasets verified by domain experts.

As a result, a practical business model choice is to develop and sell curated datasets to many customers instead of contracting the work to a single foundation model developer. This approach provides the opportunity to amortize the collection and annotation cost across a larger customer base and could eventually drive higher margins if demand rises as forecasted. By building these domain-curated datasets in advance of that demand increase, Scale AI could be positioned for another round of growth.

Scale AI will not be alone in these efforts. Its competitors have similar capabilities, and some have adopted similar strategies. The data lake wars and foundation model wars are already at a fever pitch and well-recognized by industry observers and the news media. The dataset wars are not as visible but are no less fierce. Scale AI’s latest funding round shines a spotlight on another AI landscape battlefront that is sure to become more visible over the next two years.

Share

Subscribe now

• Dabble Lab: AI-powered bespoke software development services

• Loccus.ai: AI for protecting against voice fraud

Synthedia recently published a story about Elon Musk’s X.ai intent to raise $3 billion at a valuation of $18 billion. Those rumors turned out to be considerably too conservative from a fundraising angle. X.ai announced over the weekend that it had raised $6 billion at an $18 billion pre-money valuation, which translates into $24 billion with the new cash infusion.

Our Series B funding round of $6 billion with participation from key investors including Valor Equity Partners, Vy Capital, Andreessen Horowitz, Sequoia Capital, Fidelity Management & Research Company, Prince Alwaleed Bin Talal and Kingdom Holding, amongst others…

xAI will continue on this steep trajectory of progress over the coming months, with multiple exciting technology updates and products soon to be announced. The funds from the round will be used to take xAI’s first products to market, build advanced infrastructure, and accelerate the research and development of future technologies.

Grok 2.0 Imminent?

X Daily News reported in an early April post that “Grok 2 will be done training in May and will be better than GPT4.” This appears to refer to the Grok 2 AI foundation model as opposed to an update to the generative AI chat assistant of the same name.

On April 1st, X.ai introduced the Grok-1.5 multimodal AI foundation model. It reported a considerable improvement in public benchmark performance compared to the 1.0 version released in November 2023. For the limited number of benchmarks cited, Grok-1.5 was shown as slightly behind OpenAI’s GPT-4 and Anthropic’s Claude 3 Opus. This demonstrated substantial improvements over the 1.0 model.

While the 2.0 model may complete its training this week, we are likely to see an announcement in June. Additional alignment training for use in the Grok chat assistant could delay public disclosure. However, X.ai doesn’t offer independent access to its frontier models, so there is no reason to wait for an announcement. The company can promote its scientific advances and create more buzz around a potential alternative to Meta’s Llama 3 models.

100k NVIDIA GPUs

Musk also revealed in a recent investor presentation that the company intends to build “Gigafactory of Compute.” It is expected to run a cluster of 100,000 NVIDIA H100 GPUs and be online in the fall of 2025, according to The Information.

That could turn out to be a smaller number of NVIDIA’s new Blackwell chips with a comparable capacity. Regardless, it would far exceed the 20,000 H100 GPUs employed to train Grok-2.0 and the reported 25,000 A100 GPUs used for GPT-4 training. The expectation is that the higher GPU capacity will enable faster model training on larger datasets. This may be particularly important for Musk as he intends to use a lot of visual data captured by Tesla automobiles to train the foundation model’s vision system.

At the current $30,000 H100 price tag, the 100,000 GPU cluster would account for $3 billion in revenue for NVIDIA and a comparable cost to X.ai. Those costs will only rise as GPUs are only part of the cost that X.ai will incur in building out a custom X data center. The H100 pricing is expected to fall as the Blackwell GPU accelerators begin shipping, so X.ai may get these at a discount to current costs. However, the multi-billion dollar investment in GPUs and other required computing infrastructure is surely a key reason that Musk sought to increase the funding round size to $6 billion.

This may also be a less risky bet than those made by several of the cloud providers that are counting on rising demand for model training and inference. In addition to training X.ai models, Tesla and X will need increasing amounts of inference capacity at competitive pricing. The “GigaFactory of Compute” could be viewed as a vertical integration strategy for Musk’s existing businesses that brings inference capacity in-house with the added bonus of capacity for training new X.ai models.

What’s Next?

The more significant development around Grok-2.0 would be the open-source release of the Grok-1.5 multimodal foundation model. The 1.0 release is currently available for download and use under the permissive Apache 2.0 license. However, the 1.5 model release will be more competitive with existing open-source models.

In addition, Grok’s open-source release requires users to deploy and manage the model on their own. This is not a task that many developers want to take on, at least until they know they want to use it. X.ai will get a lot of momentum out of a cloud provider offering API access to a pre-deployed model. Given Larry Ellison’s close relationship with Musk and X.ai, Oracle Cloud may be the first to offer this service, though any infrastructure provider could do it.

Another logical candidate to host a Grok LLM endpoint is NVIDIA’s DGX Cloud, given the company’s strong partnership with Tesla, its open-source model garden, and Musk’s intent to order another 100,000 GPUs for the Gigacomputer. Finally, AWS is following the “all of the above” AI model approach. This could be an opportunity for AWS to differentiate among the cloud hyperscalers, as Azure and Google are unlikely to add X.ai models soon.

The announcement also indicated that new products are coming soon. A $6 billion war chest will help drive product development and give X.ai the means to buy a busload of AI computing infrastructure. Momentum around model development, along with unique data sources via Twitter and Telsa, combined with billions in fresh funding, suggest X.ai may become a significant player in the open-source AI foundation model market very soon. Llama and Mistral have a competitor on the rise, even though OpenAI is Musk’s real target.

Share

Subscribe now

• Dabble Lab: AI-powered bespoke software development services

• Loccus.ai: AI for protecting against voice fraud

Pindrop announced a new deepfake warranty for users of its full voice security, biometrics, and deepfake detection product suite. The company will reimburse its users for “synthetic voice fraud losses…that occur in the IVR or contact center when Pindrop Scores do not alert to that risk.”

The reimbursement rate is capped at $1 million for customers with more than 100 million annual calls, requires a three-year subscription, and is only available to companies in the U.S. and Canada. Companies can get $100,000 in coverage for as little as 10 million calls.

As deepfake technology advances, posing new challenges in cybersecurity, Pindrop is dedicated to helping its customers with effective detection strategies. The Pindrop Pulse technology is an integral part of the Pindrop® Product Suite, and it boasts a 99% deepfake detection rate with minimal false positives. The Pulse Deepfake Warranty embodies our confidence in the Pindrop Product Suite’s ability to detect synthetic voice fraud.

Rise of Deepfakes

Deepfake incidents are rising as the technology improves and costs fall. WPP CEO Mark Read was the victim of a recent deepfake scam attempt. According to the Financial Times:

Mark Read, chief executive of WPP, has been the target of a deepfake scam in which criminals used a voice clone and public YouTube footage to set up a video meeting with his executives.

In an email to colleagues seen by the Financial Times, Read said that the scammers used a publicly available photo to set up a fake WhatsApp account under his name. This was used to arrange a Microsoft Teams call between one of his agency heads, another senior executive and the scammers. 

Once on the Teams meeting, a voice clone and YouTube footage of the other executive was used, he said, while scammers impersonated Read off-camera using the chat function. 

The scam was apparently unsuccessful but highlighted the increasingly sophisticated use of voice clones in fraud schemes. However, these scams are not limited to high-profile executives.

A high school physical education teacher created a deepfake of the school’s principal “making racist and antisemitic comments,” according to The Verge. Baltimore police arrested the perpetrator last month.

Families are receiving calls impersonating loved ones asking for money. Banks are fielding calls from fraudsters attempting to bypass voice authentication technologies that verify their customers. Pindrop’s warranty arrives at a time when companies are beginning to consider how best to address the rise in deepfakes hitting their contact centers.

Technology and Assurance

Pindrop has developed deepfake and voice clone detection software based on 22 patents. It claims a detection rate of 99% across all deepfakes and 90% for zero-day attacks by new voice cloning technology. The company added the Pindrop Deepfake Warranty to provide added assurance that its liveness detection technology is effective and offers its customers protection from losses if it fails.

Learn more about Synthedia’s deepfake research here.

Share

Subscribe now

• Dabble Lab: AI-powered bespoke software development services

• Loccus.ai: AI for protecting against voice fraud

Most of the attention following generative AI focuses on large language model (LLM) developers, such as OpenAI and Anthropic, or the big three cloud hyperscalers. New AI foundation models and features drive technology news cycles. However, attention doesn’t necessarily equate to cash or financial returns in those sectors just yet.

OpenAI is raking in subscription fees from ChatGPT, and Microsoft is capturing revenue and new customers through its Azure OpenAI Service. However, when it comes to revenue and profit, no one is in the same league as NVIDIA.

NVIDIA announced this week that its Q1 2025 revenue rose to $26.0 billion, up 18% from the previous quarter and 262% from Q1 2024. Data center revenue, which is predominantly driven by GPU sales for AI workloads, accounted for 89% of total revenue and was up 427% from the same quarter in 2024. Its net income was nearly $15 billion for the quarter. The quarterly revenue was only $974 million less than the full-year 2023 revenue.

It’s Not All Training Anymore

The company estimates that about 40% of its data center revenue was driven by inference. This rise occurred despite the 3x improvement in inference speed, which translated into a comparable price reduction. Inference is scaling in terms of applications, model complexity, users, and queries per user. However, the data also means that three-fifths of data center revenue is still driven by training AI models.

In Q1, NVIDIA worked with over 100 companies to build AI factories, including more than one that exceeded 100,000 GPUs. Automotive is expected to be the largest data center customer segment this year due largely to the rise in training for self-driving and other vision systems. Tesla expanded its NVIDIA cluster to 35,000 H100s in the quarter.

Consumer internet is a strong growth driver, according to NVIDIA CFO Colette Kress. Llama 3 was trained on 24,000 H100s. Sovereign AI is also growing and is expected to reach the “high single-digit billions” in revenue this year.

Rising Developer Adoption

NVIDIA’s secret strength goes beyond the hardware to developer adoption of its CUDA software. Total CUDA developers more than doubled over the past three years, rising from 2.5 to 5.1 million. Startups and GPU-based applications also more than doubled during the period.

The Engine Behind Model Advances

It is hard to overstate NVIDIA’s importance in the generative AI innovation curve. Accelerated computing and the voracious drive to reduce costs have made it possible for new and larger models to arrive in the market faster. This opens up access to new use cases for businesses and consumers and drives both technology trial and adoption.

Since GPUs are a precursor material for generative AI model training, you can look at NVIDIA sales as a leading indicator of industry growth. That will continue to be the case until capacity exceeds demand. When you see NVIDIA sales begin to decline, it will be a sign that the capacity shortage is coming to an end, and AI model training and inference costs are likely to start falling.

Share

Subscribe now

• Dabble Lab: AI-powered bespoke software development services

• Loccus.ai: AI for protecting against voice fraud