How to Weaponize Microsoft Copilot for Cyberattackers

At Black Hat USA, security researcher Michael Bargury released a LOLCopilot ethical hacking module to demonstrate how attackers can exploit Microsoft Copilot — and offered advice for defensive tooling.

BLACK HAT USA – Las Vegas – Thursday, Aug. 8 –
Enterprises are implementing Microsofts Copilot AI-based chatbots at a rapid pace, hoping to transform how employees gather data and organize their time and work. But at the same time, Copilot is also an ideal tool for threat actors.
Security researcher Michael Bargury, a former senior security architect in Microsofts Azure Security CTO office and now co-founder and chief technology officer of Zenity, says
attackers can use Copilot
to search for data, exfiltrate it without producing logs, and socially engineer victims to phishing sites even if they dont open emails or click on links.
Today at Black Hat USA in Las Vegas, Bargury demonstrated how Copilot, like other chatbots, is susceptible to prompt injections that enable hackers to evade its security controls.
The briefing,
Living off Microsoft Copilot
, is the second Black Hat presentation in as many days for Bargury. In his first presentation on Wednesday, Bargury demonstrated how developers could
unwittingly build Copilot chatbots capable of exfiltrating data
or bypassing policies and data loss prevention controls with Microsofts bot creation and management tool, Copilot Studio.
Thursdays follow-up session focused on various risks associated with the actual chatbots, and Bargury released an offensive security toolset for Microsoft 365 on GitHub. The new
LOLCopilot module
, part of powerpwn, is designed for Microsoft Copilot, Copilot Studio, and Power Platform.
Bargury describes it as a red-team hacking tool to show how to change the behavior of a bot, or copilot in Microsoft parlance,
through prompt injection
. There are two types: A direct prompt injection, or jailbreak, is where the attacker manipulates the LLM prompt to alter its output. With indirect prompt injections, attackers modify the data sources accessed by the model.
Using the tool, Bargury can add a direct prompt injection to a copilot, jailbreaking it and modifying a parameter or instruction within the model. For instance, he could embed an HTML tag into an email to replace a correct bank account number with that of the attacker, without changing any of the reference information or altering the model with, say, white text or a very small font.
Im able to manipulate everything that Copilot does on your behalf, including the responses it provides for you, every action that it can perform on your behalf, and how I can personally take full control of the conversation, Bargury tells Dark Reading.
Further, the tool can do all of this undetected. There is no indication here that this comes from a different source, Bargury says. This is still pointing to valid information that this victim actually created, and so this thread looks trustworthy. You dont see any indication of a prompt injection.
Bargury describes Copilot prompt injections as tantamount to remote code-execution (RCE) attacks. While copilots dont run code, they do follow instructions, perform operations, and create compositions from those actions.
I can enter your conversation from the outside and take full control of all of the actions that the copilot does on your behalf and its input, he says. Therefore, Im saying this is the equivalent of
remote code execution in the world of LLM apps
.
During the session, Bargury demoed what he describes as remote Copilot executions (RCEs) where the attacker:
Manipulates a copilot to
alter the banking information
of a victims vendors to steal funds
Exfiltrates data in advance of an
earnings report
to trade on that information
Makes Copilot a malicious insider
that directs users
to a phishing site to harvest credentials
Bargury isnt the only researcher who has studied how threat actors could attack Copilot and other chatbots with prompt injection. In June, Anthropic detailed its
approach to red team testing
of its AI offerings. And for its part, Microsoft has touted
its red team efforts
on AI security for some time.
In recent months, Microsoft has addressed newly surfaced research about prompt injections, which come in direct and indirect forms.
Mark Russinovich, Microsoft Azure’s CTO and technical fellow, recently discussed various AI and Copilot threats at the annual Microsoft Build conference in May. He emphasized the release of Microsofts new Prompt Shields, an API designed to detect direct and indirect prompt injection attacks.
The idea here is that were looking for signs that there are instructions embedded in the context, either the direct user context or the context that is being fed in through the RAG [retrieval-augmented generation], that could cause the model to misbehave, Russinovich said.
Prompt Shields is among a collection of Azure tools
Microsoft recently launched
that are designed for developers to build secure AI applications. Other new tools include Groundedness Detection to detect hallucinations in LLM outputs, and Safety Evaluation to detect an applications susceptibility to jailbreak attacks and creating inappropriate content.
Russinovich also noted two other new tools for security red teams:
PyRIT (Python Risk Identification Toolkit for generative AI)
, an open source framework that discovers risks in generative AI systems. The other, Crescendomation, automates Crescendo attacks, which produce malicious content. Further, he announced Microsoft’s new
partnership with HiddenLayer
, whose Model Scanner is now available to Azure AI to scan commercial and open source models for vulnerabilities, malware or tampering.
While Microsoft says it has addressed those attacks with safety filters, AI models are still susceptible to them, according to Bargury.
He says in specific, theres a need for more tools that scan for what he and other researchers call promptware, i.e., hidden instructions and untrusted data. Im not aware of anything you can use out of the box today [for detection], Bargury says.
Microsoft Defender and Purview dont have those capabilities today, he adds. They have some user behavior analytics, which is helpful. If they find the copilot endpoint having multiple conversations, that could be an indication that theyre trying to do prompt injection. But actually, something like this is very surgical, where somebody has a payload, they send you the payload, and [the defenses] arent going to spot it.
Bargury says he regularly communicates with Microsofts red team and notes they are aware of his presentations at Black Hat. Further, he believes Microsoft has moved aggressively to address the risks associated with AI in general and its own Copilot specifically.
They are working really hard, he says. I can tell you that in this research, we have found 10 different security mechanisms that Microsofts put in place inside of Microsoft Copilot. These are mechanisms that scan everything that goes into Copilot, everything that goes out of Copilot, and a lot of steps in the middle.

Last News
▸ Nigerian scammers now turning into mediocre malware pushers. ◂ Discovered: 23/12/2024 Category: security	▸ Beware EMV may not fully protect against skilled thieves. ◂ Discovered: 23/12/2024 Category: security	▸ Hack Your Hotel Room ◂ Discovered: 23/12/2024 Category: security

**Cyber Security Categories**
Google Dorks Database	Exploits Vulnerability	Exploit Shellcodes

CVE List

Tools/Apps

News/Aarticles

Phishing Database

Deepfake Detection

Trends/Statistics & Live Infos

Tags:
How to Weaponize Microsoft Copilot for Cyberattackers