Access control and permission management for AI agents: building with security in mind

We're seeing more companies create AI agents (AI companions) — chatbots that answer their users' questions based on the company's data and assist users within the product context. You have probably seen those: Notion AI, Zoom AI companion, Humaans AI Companion, Otter AI Chat, and many others.

AI-powered chatbots and agents are value-adding features for many companies, helping both “internal users” (company employees) and external users, who interact with the company products.

For example, Notion AI has an AI-powered Q&A feature that unblocks users' work using information across all their Notion pages: wikis, projects, knowledge bases, and docs. Such use cases for AI agents should improve product activation and retention, which is a holy grail for every tech company.

The key challenge here is that these chatbots should provide contextually useful responses that would solve users' problems. So how can it be achieved?

Empowered by Retrieval-Augmented Generation

Large Language Models are the foundation of these AI chatbots, but Retrieval-Augment Generation is key to making them useful.

And that is where security concerns start to creep in.

Traditional LLMs are trained on vast datasets so they can learn patterns and relationships in the data. When prompted, they use this reservoir of data to generate a coherent, relevant response. This is powerful from a generalized perspective but can cause ‘hallucinations’ or improper responses, especially when dealing with specific use cases like a nuanced, atypical prompt (user question) not present in the training data set.

In addition, LLMs deal with a knowledge cutoff, which stops them from actively pulling in new information. But, it’s relatively secure because it only pulls information from the dataset it’s trained on. So if you don’t have sensitive information in the training data set, you can sleep well.

Retrieval-Augment Generation (RAG) systems connect the LLM to external data sources, allowing them to dynamically access and process information from outside the data set they were trained on, including databases, APIs, or document repositories. This allows them to provide more current, accurate, and contextually appropriate responses compared to traditional LLMs. Let’s get back to our Notion example and see how it works for them:

However, these RAG-powered AI agents and chatbots are only as good as the data they can access. Without the right data, their ability to help users is limited.

The data access dilemma

It’s totally natural to have the desire to maximise the benefits of these AI agents. As a result, many companies are making bold—and potentially risky—decisions by essentially giving AI agents root access to all data and assuming that sensitive or protected data won’t be leaked.

This approach gives users access to rich data that is more likely to be useful.

A great example is a Humaans AI Companion, which we already mentioned. Their AI companion provides instant answers to queries from Humaans’ users, allowing employees to quickly find information on company policies, documents, and other HR-related matters.

HR is a great example of an industry where the data stored within these systems needs to have strong protections around it for numerous legal, regulatory, and ethical reasons. Imagine if an employee asks a question about their manager's salary or team restructuring plans (aka plans for layoffs).

The AI companion should not reveal such sensitive information. If a company were to build a RAG-based architecture that has access to the information and there is no security stopping the response, the AI companion would answer the question.

As you see RAG introduces a critical security concern: the RAG architectures need to consider the permissions of the user. Access control and permission management is something you can not ignore when building your RAG-based AI agent.

The security risks of unrestricted AI access

Without properly mitigating the various risks associated with RAG-based systems, you end up with an AI agent that can compromise your sensitive data. Either it isn’t as useful as it can be, because you’ve restricted its access too much, or is a security risk because it has too much access.

When you allow your RAG system too much access, you open the door to a variety of ways users can manipulate your AI chatbot, whether maliciously or accidentally. To make sure your AI chatbot is both useful and secure, you need to consider security risks on all sides of your AI companion, as examined in the latest OWASP report.

Risks of data leaks

As we’ve covered above, AI systems without proper permission management could potentially access sensitive data and give it to unauthorized users. This is a significant concern when handling personally identifiable information (PII) or protected health information (PHI), as even the safest systems are vulnerable to human error.

Based on a Verizon report, “65% of data breaches in 2023 involved internal actors.” In 68% of these cases, human error was a main factor.

That makes ensuring the security of your AI agents extremely important, no matter how trust-worthy the users are.

But, even with controls in place, RAG is still vulnerable to data exfiltration and prompt injection. With the right prompts, attackers may be able to bypass security measures and extract sensitive information in connected databases. This is analogous to traditional SQL injection attacks but targets the LLM's input processing.

Risks of data injection

While data leaks are the most obvious vulnerability, malicious actors can go beyond uncovering data they shouldn’t have to inject data into your databases. With context injection, attackers inject malicious code or misleading content into the external knowledge sources used by RAG. This poisoned context could then influence the AI's responses, leading to the generation of inaccurate, inappropriate, or harmful content.

Risks of breaking predetermined rules

Malicious actors can overwhelm the AI chatbot by flooding its context window with irrelevant information. The context window, essentially the working memory of the AI, determines how much preceding context the model can use to generate coherent and relevant responses. If there is too much information overloading it, essential information, including system instructions or contextual data, can get bumped out. This could lead to the AI "forgetting" its limitations and producing unintended or harmful outputs.

A mistake too expensive to make

These security risks will lead to a variety of serious problems, that should make your leadership and legal team very nervous:

• You might inadvertently violate data protection regulations like GDPR or commit data privacy violations when sensitive information is extracted and exposed to unauthorized people.
• Confidential business information could be leaked if the AI companion is compromised, leaving your company at a competitive disadvantage.
• Employees and customers may lose faith in the company's ability to protect their data, or because they’ve been subject to poisoned or injected data that leads them wrong.

Authorization-aware data filtering with Cerbos

You know you need your AI to keep up with the competition. But, you also know racing ahead without addressing security concerns is a recipe for disaster. So, how do you walk that line?

Context-aware data filtering, is a prevention and mitigation strategy mentioned in the latest OWASP report. Using an authorization solution like Cerbos, helps you make the most of RAG-enhanced LLMs while avoiding security pitfalls. Here's how this secure AI interaction process works:

1. A user interacts with the company’s AI agent, asking a work-related question. Cerbos enforces permission policies to ensure the user is authorized to launch an agent.
2. Before retrieving data, Cerbos creates a query plan to apply conditions that ensure the user only accesses records allowed by their role, department, region, or other attributes.
3. Then Cerbos provides an authorization filter to limit the information fetched from the company’s vector database or other data stores. So the database is queried for applicable documents, which are filtered based on authorization.
4. Retrieved documents are injected into the prompt.
5. The LLM processes the prompt and generates an answer, making it relevant and fully compliant with user permissions.

This ensures that all interactions are safe and secure, as the AI agent only accesses data that the user is authorized to see. Take a closer look at how access control and permission management for AI agents works:

AI agents' access control and permission management applied consistently

One of the best parts of this approach is that authorization policies for AI agents and other parts of your product can be stored and managed centrally. That allows you to consistently apply them across the various layers of your technology stack:

• In the data layer, you can control access to raw data in databases and data warehouses.
• At the API level, you can enforce permissions for data accessed programmatically.
• You can enforce access controls within software applications in your application.
• You can also ensure AI agents and chatbots respect user permissions.

By keeping the authorizations centralized, you simplify the constant process of updating and maintaining security policies.

There are many more benefits of making proper access management for AI agents, including:

• Stronger security by auditing your agent's data retrieval and responses.
• Enterprise-ready compliance with data protection regulations to build trust.
• Flexible, scalable authorization that grows alongside your AI agent.
• Dynamic policy adjustments to update logic filtering in real-time, ensuring always up-to-date control.
• Complete control over AI agent response, ensuring all data is authorized to preserve competitive advantages and personal information.
• Quick, easy permission management rollout that allows for faster release times.

If you want to dive deeper into the technical details of implementing permission management for your AI companion, join one of our engineering demos or check out our in-depth documentation.

Until then, keep building useful and secure AI agents!