ChatGPT Plugins Alpha - Conversational Commerce, Images and UX Issues [VIDEOS]

22 Plugins ranging from Lowe's to Wolfram Alpha

TLDR;

• View three videos walking through ChatGPT plugins from Lowe’s, Expedia, and Wolfram, demonstrating e-commerce, multimodal image and chart display, and access to real-time internet data from within the chat conversation.

• The plugin model is interesting and adds to ChatGPT’s capabilities, but it does introduce several user experience (UX) issues that may inhibit adoption.

• There are 22 plugins currently available for early access Alpha users, but you cannot use two at the same time which serve the same domain. Travel is an example. You can use Kayak or Expedia, but not both simultaneously.

• You cannot use the GPT-4 model for chat and the Plugins model at the same time. The functionality is separated, and that can lead to poor intent fulfillment.

• Plugins are the “doing,” while ChatGPT models are the “knowing.” Siri’s Adam Cheyer and Dag Kittlaus correctly predicted the difficulty in merging the two.

OpenAI has been rolling out an alpha version of ChatGPT plugins to ChatGPT Plus users on the waitlist. There are now 22 plugins available, including DIY home improvement retailer Lowe’s, restaurant booking service OpenTable, quantitative knowledge engine Wolfram Alpha, delivery service Instacart, real estate listing service Zillow, travel booking sites Expedia and Kayak, and several other services.

I have early access, and the video above walks through a really interesting e-commerce use case, complete with a multimodal image display right in the ChatGPT conversation and links to purchase products on Lowes.com. However, it also includes a glaring UX issue.

Two other videos below show other shortcomings, but overall it’s an intriguing experience and a pretty amazing step forward after such a short development cycle. The images immediately below show all of the currently available plugins.

The Flaw in ChatGPT Plugins UX

After enabling plugins and selecting some from the store, ChatGPT will automatically invoke them when you ask a question about their domain. However, there are a few important nuances here.

1. Choose a Model: To use plugins, you must first select the Plugins AI model in the ChatGPT model selection drop-down menu. That means you are not using GPT-3.5 or GPT-4 but a model designed for use with third-party services. This breaks the user experience because you have to decide between models before asking your question, or you must choose a new model and repeat the process. This is precisely the issue that Siri co-founder Adam Cheyer predicted in a recent Voicebot Podcast interview which will publish next week.

2. Intent-Matching Failure: At the end of the video above, you can also see that this causes another problem. Instead of trying to answer the clear intent of my question, the plugins model selected Lowe’s as the best option and then proceeded to tell me about fence products for purchase at Lowe’s. That might be helpful, but I had to go back to the GPT-4 model to get my initial question answered, which was not about building suppliers but rather about the tools required for the project. It would seem logical that Lowe’s could also answer that question but went right for the product sale.

3. Manual Model Deconfliction: A user can select up to three plugins at any time, and ChatGPT will select the one most relevant to your query. However, you cannot have two plugins that service the same domain active simultaneously. The video below shows that I could have either the Expedia or Kayak plugin active, but not both. This helps ChatGPT avoid conflicts between selecting the right plugin. It would be a richer experience if the user could see answers from both side-by-side or some arbitration about which answer is better. OpenAI is forcing the user to manually address deconfliction instead of having the AI do it for you.

4. Cannot Use “Incognito” Mode: OpenAI added “private chat” or “incognito” mode to ChatGPT last week. That feature must be turned off to use the Plugins model. I assume this is because ChatGPT must share information with the plugin; therefore, the chat is not fully private. There might not be a way around this, but it does break the user experience flow. If you want that chat not to be saved by ChatGPT, you must go into your history and delete the conversation. I don’t believe this removes the conversation from potential use for future model training, but it will remove the data from your profile—a little warning for anyone that wants private chat sessions and plans to employ plugins.

Multimodal Upgrade

You can see in each video how images now play a role in ChatGPT responses from plugins. The Lowe’s home improvement example showed images of different fence styles I could consider and provided links to purchase. The Expedia flight search example included thumbnail images designed to entice a clickthrough to one of the flight purchase options. The Wolfram Alpha multimodal response was more impressive.

I asked a question that I expected Wolfram Alpha to have useful information about but might be more difficult for ChatGPT. The images below tell the story. ChatGPT’s GPT-4 model offered useful context but did not fully answer the question, while the Wolfram Alpha response answered precisely and provided a chart!

You can see in the video below the Wolfram plugin also updated the chart based on a follow-on question that requested data from France be incorporated. The images below will make inspecting the differences between the GPT-4 model and plugin output easy, while the video offers additional details.

How Revolutionary are ChatGPT Plugins?

Now that we have seen what plugins can do, how revolutionary are they? I think this will depend on the plugin. However, I don’t expect plugins to be revolutionary or drive significant incremental ChatGPT adoption in the near term. The requirement to select the plugin model and then ensure the plugin you want is active breaks the user experience.

Then again, for some tasks, it will be superior. I will start with the Wolfram plugin to access its knowledge base and then click through on the link if I want additional information. I might look at flight search times in ChatGPT, but I’m not sure that process is materially better than just going to Expedia or an airline website directly. The ChatGPT plugin is superior to the Expedia generative AI solution, which I reviewed recently, but not the website.

The most optimistic view is that the Plugins model demonstrated in the Alpha release is a first step. Future iterations could integrate a seamless handoff from a core model, such as GPT-4, to the plugin model and a response from a third party. That would come much closer to the “super app” experience I wrote about last week.

We can see the vision, but the implementation details and user experience will matter. Voice assistants such as Siri and Alexa help you with “doing,” as Siri’s Adam Cheyer summarized for me recently. You may recall that Dag Kittlaus, another Siri co-founder, originally positioned Siri as the “do engine.” Cheyer says ChatGPT solves the other half of the assistant equation, “knowing.” Large language models were the missing ingredient in earlier voice assistants for knowledge tasks.

Plugins are OpenAI’s approach to bridging the “knowing” and “doing” divide. ChatGPT will handle the knowledge requests and hand over the action-oriented requests to partners through plugins. It’s a little clunky now, but you can see the spark of innovation and a path toward the assistant we’ve always wanted.

The forthcoming ChatGPT feature that may drive more users and more frequent use is the “browsing” model. OpenAI’s Greg Brockman demonstrated the feature at a recent Ted Talk. That model turns ChatGPT into a real-time, internet-connected browser with some features not currently available in Bing Chat. ChatGPT was already valuable. These new models and plugins will make it even more enticing.

Let me know what you think in the comments.