The National Cyber Security Centre (NCSC) is warning security teams to be on the lookout for AI prompt injection attacks.
These involve an attacker creating apparently innocent inputs to large language models (LLMs) which take advantage of the model's inability to distinguish between developer-defined prompts and user inputs to cause unintended behaviour.
Prompt injection attacks are often seen as just another version of SQL injection attacks, said NCSC technical director for platforms research David C, with data and instructions being handled incorrectly - but this is a mistake.
In SQL, instructions are something the database engine does and data is something that is stored or used in a query; much the same is true in cross-site scripting and buffer overflows.
Mitigations for these issues enforce this separation between data and instructions. For example, the use of parameterized queries in SQL means the database engine can never interpret it as an instruction, regardless of the input. The right mitigation solves the data/instruction conflation at its root, David C pointed out.
"Under the hood of an LLM, there’s no distinction made between ‘data' or ‘instructions'; there is only ever ‘next token’. When you provide an LLM prompt, it doesn’t understand the text it in the way a person does. It is simply predicting the most likely next token from the text so far," he said.
"As there is no inherent distinction between ‘data’ and ‘instruction’, it’s very possible that prompt injection attacks may never be totally mitigated in the way that SQL injection attacks can be."
Security teams should stop treating prompt injection as a form of code injection, but instead view it as an exploitation of an ‘inherently confusable deputy’.
This is where a system can be coerced to perform a function that benefits the attacker, typically where a privileged component is coerced into making a request on behalf of a less-privileged attacker.
"Crucially, a classical confused deputy vulnerability can be mitigated, whilst I’d argue LLMs are ‘inherently confusable’ as the risk can’t be mitigated," said David C.
"Rather than hoping we can apply a mitigation that fixes prompt injection, we instead need to approach it by seeking to reduce the risk and the impact. If the system’s security cannot tolerate the remaining risk, it may not be a good use case for LLMs."
AI prompt injection attacks are rising
Prompt injection attacks have become a recurring talking point over the last three years, with security experts warning about the potential for threat actors to manipulate AI models into producing malicious outputs.
Pete Luban, field CISO at AttackIQ, said the NCSC advice should be taken seriously by enterprises using the technology.
However, just because AI prompt injection attacks can't be mitigated in the same way as SQL injection attacks, Luban said this doesn't mean that lessons can't be learned from SQL injection defense.
"Developers need to build systems around LLMs with the awareness that prompt injection attacks are a threatening class of vulnerability. Since these attacks cannot be handled with a single product or appliance, careful design and operation is paramount to preventing exploitation," he said.
"Security teams should understand that all known methods of prompt injection prevention can only reduce chances of an attack or breach. Updating and strengthening their overall security posture, including continuously monitoring systems for irregularities and testing against common adversarial tactics, can help systems identify earlier stages of an attack and quickly move to mitigate it."
Make sure to follow ITPro on Google News to keep tabs on all our latest news, analysis, and reviews.
MORE FROM ITPRO
-
Licensed mmWave: Opportunity or overhead?Industry Insights Ofcom’s latest mmWave auction unlocks major new capacity for 5G and FWA, offering a faster, more flexible complement to fiber - especially in dense urban areas
-
CISA issues alert as China-linked hackers exploit Brickstorm malware to target VMware serversNews Organizations, particularly in the critical infrastructure, government services, and facilities and IT sectors, need to be wary of Brickstorm
-
Chinese hackers are using ‘stealthy and resilient’ Brickstorm malware to target VMware servers and hide in networks for months at a time
News Organizations, particularly in the critical infrastructure, government services, and facilities and IT sectors, need to be wary of Brickstorm
-
AWS CISO Amy Herzog thinks AI agents will be a ‘boon’ for cyber professionals — and teams at Amazon are already seeing huge gainsNews AWS CISO Amy Herzog thinks AI agents will be a ‘boon’ for cyber professionals, and the company has already unlocked significant benefits from the technology internally.
-
HPE selects CrowdStrike to safeguard high-performance AI workloadsNews The security vendor joins HPE’s Unleash AI partner program, bringing Falcon security capabilities to HPE Private Cloud AI
-
The Scattered Lapsus$ Hunters group is targeting Zendesk customers – here’s what you need to knowNews The group appears to be infecting support and help-desk personnel with remote access trojans and other forms of malware
-
Impact of Asahi cyber attack laid bare as company confirms 1.5 million customers exposedNews No ransom has been paid, said president and group CEO Atsushi Katsuki, and the company is restoring its systems
-
If you're not taking insider threats seriously, then the CrowdStrike incident should be a big wake up callNews CrowdStrike has admitted an insider took screenshots of systems and shared them with hackers, and experts say it should serve as a wake up call for enterprises globally.
-
Shai-Hulud malware is back with a vengeance and has hit more than 19,000 GitHub repositories so far — here's what developers need to knowNews The malware has compromised more than 700 widely-used npm packages, and is spreading fast
-
Security experts claim the CVE Program isn’t up to scratch anymore — inaccurate scores and lengthy delays mean the system needs updatedNews CVE data is vital in combating emerging threats, yet inaccurate ratings and lengthy wait times are placing enterprises at risk
