8 Announcements from Microsoft BUILD

Microsoft keeps layering on the generative AI application tooling

The Microsoft BUILD 2024 developer conference offered a new slate of generative AI announcements. However, the tone was very different from what we witnessed last week for OpenAI’s and Google’s announcements. That seemed appropriate, given that the “wow factor” was not working in Microsoft’s favor.

Nearly everyone was impressed at what OpenAI and Google showed last week. Microsoft didn’t have a clear moment or feature ready to outshine its partner or rival, so the company relied more on substance than flash. It also wasn’t shy about repeating announcements from the past several months that were designed to repeat the evidence that the company has a meaningful lead in several segments.

Dozens of new announcements from BUILD showed incremental progress designed to overcome enterprise challenges in transitioning from pilot to production and highlight differences from rivals. While OpenAI and Google seemed to tilt slightly toward consumer applications in their May releases, Microsoft stuck to its enterprise focus.

Synthedia’s criticism of Google I/O was that it offered more futures than presents. While Microsoft talked more about what was available today than its cloud and AI rival, it also provided a significant list of future capabilities expected later this year. Here are Synthedia’s top picks for Microsoft’s AI-related announcements from BUILD.

1. Speak a Copilot into Existence

Microsoft expects copilots to be anywhere. However, they don’t want that process stalled by a lack of skilled designers and developers available to bring copilots to life. The answer? Enable any user to describe what they want their copilot to do and automatically create a version.

This may remind you of GPTs on ChatGPT. It is even simpler. If that is too hard, Microsoft also offers templates that enable you to create a copilot with the click of a button. The announcement states:

You can create your copilot with our brand new conversationally driven experience—simply describe what you want it to do, and what knowledge you want it to have, and Copilot Studio will create your very own copilot. You can then immediately test it out, add additional capabilities, such as your own actions, APIs, and enterprise knowledge—and then publish it live with a few clicks.

2. Copilot Connectors

Connectors are scheduled for availability in June 2024. Microsoft suggests they will enable easier connections with enterprise data and applications, including custom enterprise sources, Microsoft products, and some yet-to-be-named third-party applications.

3. Security Upgrades

BUILD also included a very brief mention of new security features offered through Microsoft Purview. This solution offers “access to more detailed governance tools, including audit logs, inventory capabilities, and sensitivity labels. They will be able to review comprehensive audit logs that cover tenant-wide usage, inventory (with API support), and tenant hygiene (such as data loss prevention violations and inactive copilots)…”

Features that did not receive meaningful airtime were the safety evaluations for Azure AI studio. These were originally announced in March and are currently in a restricted preview. Key features include:

• Prompt Shields to detect and block prompt injection attacks, including a new model for identifying indirect prompt attacks before they impact your model, coming soon and now available in preview in Azure AI Content Safety. 

• Groundedness detection to detect “hallucinations” in model outputs, coming soon. 

• Safety system messages to steer your model’s behavior toward safe, responsible outputs, coming soon.

• Safety evaluations to assess an application’s vulnerability to jailbreak attacks and to generating content risks, now available in preview.  

• Risk and safety monitoring to understand what model inputs, outputs, and end users are triggering content filters to inform mitigations, coming soon, and now available in preview in Azure OpenAI Service.

Security is an under-resourced niche in the generative AI solution stack. So much attention is paid to ensuring accurate LLM responses that few organizations have begun to tackle many of the key security issues. This will become a higher priority as more enterprises take generative AI from pilot to production. Microsoft has a solid start on this, but access to the tools remains limited.

4. Team Copilots

On the surface, Team Copilots appear similar to Google’s AI Teammates, slated for a 2025 release. Google’s Teammate approach creates a virtual worker that has skills, access, and presumably memory. For example, you could create a virtual assistant project manager that can answer questions and execute tasks on behalf of project teammates.

Team Copilots were presented as bots or tools that exist in various workspaces. For example, they could summarize meetings or documents. However, the description was more robust than the demonstrations. Microsoft’s announcement stated:

You will be able to invoke Copilot where you collaborate – in Teams, Loop, Planner and more. Team Copilot can be a meeting facilitator in meetings, managing the agenda, tracking time and taking notes. It can act as a collaborator in chats by surfacing important information, tracking action items and addressing unresolved issues. It can serve as a project manager to help ensure every project runs smoothly and notify the team when their input is needed.

While Team Copilots lacks the persona of the Teammates concept introduced by Google, the capabilities are likely to converge around similar feature sets. The first phase of these Team Copilots is likely to be request-response solutions. Future interactions may offer proactive monitoring and action execution based on a defined role and set of responsibilities. You might recognize this as morphing into an agent. Team Copilots are planned for private preview later this year.

5. Phi Rising

Phi-3 also made an appearance at BUILD. While the Phi-3 Mini and Medium editions were announced in April, Microsoft’s first vision-enabled small language model (SLM) made its BUILD debut. According to the announcement:

Phi-3-vision is the first multimodal model in the Phi-3 family, bringing together text and images, and the ability to reason over real-world images and extract and reason over text from images. It has also been optimized for chart and diagram understanding and can be used to generate insights and answer questions. Phi-3-vision builds on the language capabilities of the Phi-3-mini, continuing to pack strong language and image reasoning quality in a small model.

Microsoft also stressed the performance per parameter metrics that enable the company to demonstrate a meaningful lead over other SLM rivals. Phi is designed to reside on devices or as an inference-efficient alternative to LLMs for many use cases.

Another Phi model also made a debut this week, Phi Silica. This is a purpose built Phi model for implementation with the new NPU’s that power the Copilot +PC devices. A blog on the announcement added, “Windows is the first platform to have a state-of-the-art small language model (SLM) custom-built for the NPU and shipping inbox.”

6. Windows in VR

Apple may have raised some eyebrows with its prediction that the Vision Pro would be an everyday tool for workers and replace laptops. That seems unlikely. However, Microsoft has partnered with Meta to bring Windows and Office applications to Quest. If a user needs 3D spatial awareness applications, Microsoft is planning on fulfilling that request.

We are deepening our partnership with Meta to make Windows a first-class experience on Quest devices. And Windows can take advantage of Quest’s unique capabilities to extend Windows apps into 3D space. We call these Volumetric apps.

7. Khan Academy Copilot

Microsoft has partnered with Khan Academy to offer students more generative AI-enabled learning opportunities. Its investment in computing resources has also allowed Khan Academy to make its Amigo assistant free to users.

By donating access to Azure AI-optimized infrastructure, Microsoft is enabling Khan Academy to offer all K-12 educators in the U.S. free access to the pilot of Khanmigo for Teachers, which will now be powered by Azure OpenAI Service.   

The two companies will also collaborate to explore opportunities to improve AI tools for math tutoring in an affordable, scalable and adaptable way with a new version of Phi-3, a family of small language models (SLMs) developed by Microsoft.

8. GitHub Copilot Extensions

Enterprises can now customize their GitHub Copilot experience by adding integrations with 3rd party services. This means developers can use the natural language interface offered through GitHub Copilot chat to ask questions about information in a variety of other applications and even execute tasks. GitHub SVP of product, Mario Rodriguez, added:

We’re starting with GitHub Copilot Extensions from DataStax, Docker, LambdaTest, LaunchDarkly, McKinsey & Company, Microsoft Azure and Teams, MongoDB, Octopus Deploy, Pangea, Pinecone, Product Science, ReadMe, Sentry, and Stripe. Extensions are supported in GitHub Copilot Chat on GitHub.com, Visual Studio, as well as VS Code.

While the GitHub Marketplace will offer extensions that are open to all, organizations can also create private Copilot Extensions for their homegrown developer tooling, making the capabilities from an internal library of APIs or the knowledge from a custom monitoring system only a conversation away.

This will be helpful for developers to avoid context-switching. Moreover, it is even more impactful if a task requires interaction with several software solutions. GitHub Copilot chat then becomes the central control for multiple apps through a single natural language interface.

What’s Next

Microsoft said that over 50,000 companies are using Azure OpenAI Service, including nearly 60% of the Fortune 500, and several users have more than 10,000 Copilot seat licenses. That breadth of users offers Microsoft significant insight into how enterprises are using generative AI and what they need to manage and scale the solutions. That appears to be what is guiding the many incremental updates across AI models and tooling offerings. It’s less sexy than ChatGPT-4o’s vision and voice interactions, but the features are practical necessities.

The key driver of generative AI activity throughout the next 2-3 years will be the shift to production for many of the promised copilot solutions. It will take some time for the industry to understand, adopt, and absorb the technology, but it seems inevitable at this point that copilots will be ubiquitous.

The next big thing will be agents. Even though we will see more “agents” in 2024, many will simply be RPA-style procedures invoked by applications that also tap into LLMs. With that said, Google and Microsoft both gave considerable airtime to agents that autonomously make decisions and execute tasks on a user’s behalf. Everyone wants this. However, the real world is a bit messier than the pristine theories behind agents often envision. Agents are closer to becoming meaningful AI-enabled solutions, but you should not expect these to have an immediate impact. Still, some things are worth waiting for.

• Dabble Lab: AI-powered bespoke software development services

• Loccus.ai: AI for protecting against voice fraud