LLM gateway aims to take the pain out of managing AI security

Brain hovering above a chip on a motherboard, denoting AI and hardware
(Image credit: Getty Images)

An open source Large Language Model (LLM) gateway has been made available as a reference framework as administrators grapple with the security implications of recent generative AI innovations.

The LLM gateway was first developed as an internal tool by eSentire to govern and monitor the corporate usage of platforms such as ChatGPT, Google Bard, and other LLMs.  

Organizations across the world have been grappling with how to deal with the explosion in generative AI and the enthusiasm of users wishing to make use of the technology.

The results of this enthusiasm have often been mixed. Some of Samsung’s employees memorably fed sensitive data in ChatGPT to get outputs, inadvertently causing the data to leak outside the company. The US House of Representatives recently instructed staffers not to paste confidential information into the system. 

However, with employees keen to use the technology, a blanket ban could result in shadow IT systems, where workers find ways to circumvent IT diktats. 

An LLM gateway is therefore required to both monitor and mitigate risk while permitting employees to make use of the technology.

The open source reference framework published by eSentire is designed to create a protective layer between corporate data and tools such as ChatGPT. It logs all LLM interactions and could be extended based on an organization’s LLM security policy. This might include policies around sensitive data – for example, content rejection or masking – or a rejection of malicious interactions.

A docker postgres database is used to store conversations and interactions, although, being open source, administrators could implement their own rules. 

Alexander Feick, vice president eSentire Labs, told ITPro that the LLM gateway is a reference framework intended to allow a security professional to quickly deploy and explore the concept of what an LLM gateway might look like. The experience could then be used to drive discussion around next steps.


A whitepaper from ServiceNow covering how to lay a strategic foundation for cloud security that protects what matters to your business

(Image credit: ServiceNow)

Get started on finding an integrated, automated solution that addresses your key security and compliance concerns


He said: "The goal of making the eSentire LLM Gateway open source is to help security pros and IT teams understand the value of deploying an LLM gateway, as a data capture and monitoring point, what data might be available by default, and where security teams or IT teams might need to expand that in future".

eSentire is by no means the only organization grappling with this issue, although an open source framework is a step in the right direction.

Databricks recently announced a preview of an AI gateway component in MLflow 2.5 to secure LLMs from development through production. Also coming soon is deepchecks’ LLM Gateway, designed to scan inputs and outputs in realtime in order to block content or reroute inputs based on certain conditions.

Feick told ITPro that eSentire had no plans to release a commercial LLM gateway itself, "but we plan on offering our own MDR service for managing LLM data securely."

Making AI secure by design 

The framework follows a recent post by CISA on securing software and, in particular, AI.

The CISA post makes the point: “CEOs, policymakers, and academics are grappling with how to design safe and fair AI systems, and how to establish guardrails for the most powerful AI systems. Whatever the outcome of these conversations, AI software must be Secure by Design”.

Adopting a LLM gateway goes some way to adopting the secure by design principles espoused by CISA and shifts the burden of cyber security away from users without simply applying a global block.

Going further, a secure by default approach can also be applied to generative AI platforms, with an LLM gateway used to monitor access and control access to sensitive information.

Richard Speed
Staff Writer

Richard Speed is an expert in databases, DevOps and IT regulations and governance. He was previously a Staff Writer for ITProCloudPro and ChannelPro, before going freelance. He first joined Future in 2023 having worked as a reporter for The Register. He has also attended numerous domestic and international events, including Microsoft's Build and Ignite conferences and both US and EU KubeCons.

Prior to joining The Register, he spent a number of years working in IT in the pharmaceutical and financial sectors.