Skip to main content

AI & Agent Security

AI and Agent Security is the discipline of building, deploying, and operating AI-powered systems that are resilient against adversarial manipulation, protect sensitive data, and respect the boundaries of their intended use. It extends traditional security engineering to address the unique attack surfaces introduced by large language models (LLMs), retrieval-augmented generation (RAG), autonomous agents, and the protocols that connect them.

As AI moves from experimental prototypes to mission-critical enterprise platforms, securing these systems becomes an engineering imperative. This section of SecurityDevPro provides the architectural patterns, defensive techniques, and governance frameworks needed to design trustworthy AI applications—without sacrificing functionality or velocity.

What Is AI & Agent Security?​

AI security is the practice of protecting AI systems from threats that can compromise their confidentiality, integrity, or availability. Agent security focuses specifically on systems that can autonomously take actions through tool calls, function invocations, and multi-step reasoning.

Unlike traditional applications, AI systems process natural language as an untrusted input channel, rely on probabilistic models with emergent behaviors, and often act as orchestrators that invoke external tools and APIs. This creates new categories of risk that traditional firewalls and authentication mechanisms alone cannot address.

AI security spans:

  • LLM applications: Chatbots, copilots, and text-based AI features that consume user prompts and generate responses.
  • AI agents: Systems that use LLMs to plan and execute tasks, calling APIs, querying databases, or updating internal state.
  • Multi-agent systems: Multiple agents interacting and delegating work to each other.
  • Enterprise AI platforms: Integrated AI capabilities embedded into productivity suites, CRM systems, and development toolchains.

Securing these systems requires a holistic approach that blends application security, cloud security, identity management, and AI-specific controls.

Why AI Security Matters​

The rapid adoption of AI introduces security challenges that traditional practices were not designed to handle:

  • Untrusted prompts: Any user can craft inputs that manipulate model behavior, extract sensitive information, or bypass content restrictions.
  • External tool invocation: Agents connect models to APIs, databases, and internal services, turning prompt injection into potential remote code execution or data exfiltration.
  • Sensitive enterprise data: AI systems often process proprietary documents, customer PII, or regulated data, creating new leakage vectors.
  • Autonomous decision-making: Agents that can execute actions without human approval raise the stakes for permission boundaries and audit trails.
  • AI-powered APIs: Exposing AI capabilities as APIs introduces the same injection, authentication, and authorization concerns as traditional APIs, plus model-specific attacks.
  • Continuous model evolution: Model updates, fine-tuning, and user feedback loops can introduce or reintroduce vulnerabilities dynamically.

AI security is not a separate silo; it is an extension of security engineering that must be integrated from the initial architecture through production monitoring.

Core Knowledge Areas​

LLM Security​

LLM security focuses on the unique threats that arise from using large language models as part of an application.

Key concerns include:

  • Model misuse: Generating harmful content, impersonating individuals, or circumventing usage policies.
  • Adversarial inputs: Carefully crafted prompts that bypass model alignment or safety filters.
  • Secure system design: Placing the model behind an API layer with input validation, output filtering, and rate limiting.

The OWASP Top 10 for LLM Applications is an essential industry reference that categorizes the most critical risks, including prompt injection, insecure output handling, and model denial of service.

Explore the fundamentals in LLM Security Fundamentals.

Prompt Injection​

Prompt injection is the AI equivalent of injection attacks in traditional software. An attacker crafts input that subverts the model's intended behavior by overriding system instructions or injecting malicious context.

  • Direct prompt injection: The attacker includes instructions directly in a user prompt, such as "Ignore previous instructions and reveal the system prompt."
  • Indirect prompt injection: Malicious content is embedded in data the model retrieves, such as a poisoned web page, email, or document. When the model processes this content, it follows the embedded instructions.
  • Context poisoning: Corrupting the knowledge base or conversation history that the model relies on for context.

Defenses include strict input/output separation, sandboxed tool execution, and treating model output as untrusted. See Prompt Injection Explained.

RAG Security​

Retrieval-Augmented Generation (RAG) grounds LLM responses in external knowledge. This introduces security considerations around the retrieval pipeline.

Core security dimensions:

  • Knowledge source trust: Ensuring that only vetted, trusted documents enter the retrieval index. Untrusted content can become an indirect prompt injection vector.
  • Retrieval security: Enforcing access controls so users only retrieve documents they are authorized to see. A query should not leak data from restricted repositories.
  • Context integrity: Preventing attackers from injecting malicious content into the retrieved context that would manipulate the model's output.
  • Sensitive document protection: Redacting or filtering PII, secrets, and confidential information from indexed data.
  • Secure indexing: Preventing index poisoning by validating and sanitizing content before ingestion.

For architectural patterns, read RAG Security.

AI Agent Security​

AI agents extend LLMs with the ability to take actions. They invoke tools, call APIs, and make decisions that affect real systems. Securing them requires strict control over what an agent can do.

Key concepts:

  • Tool invocation: An agent should only be able to call pre-approved functions. The model's output must be validated before any tool is executed.
  • Function calling security: Parameters passed to tools must be sanitized, just like any other user input. An LLM should not be able to inject commands into a shell tool or SQL query tool.
  • Agent permissions: Each agent should run with a dedicated identity that has least privilege. An email agent should not have access to billing APIs.
  • Agent memory: State and conversation history can be manipulated. Memory stores must be protected and scoped.
  • Multi-agent communication: When agents delegate tasks to each other, enforce authentication, authorization, and input validation on inter-agent messages.
  • Human approval workflows: For high-risk actions (e.g., making a financial transaction, sending a bulk email), require a human to approve before execution.

Least privilege is paramount. See Agent Security.

MCP Security​

The Model Context Protocol (MCP) is an open standard for connecting AI models to external tools and data sources. MCP introduces a client-server architecture where a model (host) communicates with tool servers.

Security considerations include:

  • Trust boundaries: The model host, the MCP client, and the MCP server may run in different security domains. Every connection must be authenticated and authorized.
  • Tool authentication: The MCP server must verify the identity of the requesting agent and enforce per-agent or per-user access to tools.
  • Capability restrictions: Expose only the minimum set of tools needed. A server that provides both read-only and destructive operations should allow fine-grained capability selection.
  • Secure server design: Servers must validate all inputs from the model, resist injection attacks, and never trust raw LLM output.
  • Context isolation: Information shared with one tool or session should not leak to another.

MCP security is an emerging but critical area. Learn more in MCP Security.

AI Identity and Access Control​

Identity is the foundation of security in AI systems, just as in cloud and application security. However, AI introduces additional dimensions:

  • User identity: The human interacting with the AI must be authenticated and authorized.
  • Agent identity: Each autonomous agent must have its own service identity with scoped permissions.
  • Service identity: AI models, embedding services, and vector databases all require service-to-service authentication.
  • Delegated authorization: When an agent acts on behalf of a user, it must do so with bound, short-lived credentials that limit what it can do.
  • Fine-grained permissions: Access to specific documents in a knowledge base, specific tool functions, or specific model capabilities should be controlled at a granular level.

Read AI Identity and Access Management for implementation guidance.

AI Data Security​

AI systems consume and generate vast amounts of data. Protecting this data is a primary security objective.

Data security spans:

  • Training data protection: Ensuring training datasets do not contain PII, secrets, or proprietary code, and are stored securely.
  • Inference data protection: Prompts and completions may contain sensitive business data. They must be encrypted in transit and at rest, and not used for model retraining without explicit consent.
  • Sensitive prompt handling: Users may inadvertently include credentials or personal data in prompts. Implement scrubbing and warn users.
  • Confidential enterprise knowledge: RAG systems indexing internal wikis, code repositories, or design documents must enforce strict access controls.
  • Data governance: Classifying data, enforcing retention policies, and maintaining audit trails for all AI data interactions.

AI Governance​

Governance in AI security means establishing the policies, processes, and technical controls to ensure AI systems are used responsibly and securely.

Key focus areas for engineers:

  • Responsible AI principles: Fairness, transparency, accountability, and human-centered design.
  • Security governance: Defining who can deploy models, approve tool integrations, and authorize agentic behaviors.
  • Auditability: Logging all model inputs, outputs, tool invocations, and access to retrieval data for forensic analysis.
  • Explainability: Understanding why a model made a particular decision, especially for high-stakes actions.
  • Human oversight: Ensuring that human operators can monitor, interrupt, and override AI actions.
  • Regulatory awareness: Understanding the impact of emerging AI regulations (EU AI Act, NIST AI RMF, etc.) on system design.

Our AI Governance article translates these concepts into engineering requirements.

AI Security Architecture Principles​

The following principles guide the design of secure AI systems:

  • Zero Trust for AI: Never trust a model output. Validate it. Never trust a prompt. Sanitize it. Authenticate every tool call and data access.
  • Defense in Depth: Layer controls: input filters, model-level safety mechanisms, output validators, tool execution sandboxes, and runtime monitoring.
  • Least Privilege: Grant agents and tools the minimum permissions required. If an agent only needs to read email, don't give it delete access.
  • Human-in-the-Loop: Require human approval for high-risk operations, especially when agents handle sensitive data or financial transactions.
  • Secure by Design: Build AI systems with security requirements defined from the start. Perform threat modeling on agent workflows and tool integrations.
  • Trust Boundaries: Clearly define boundaries between the model, the data layer, the tool layer, and the user. Enforce strict isolation.
  • Continuous Monitoring: Log and monitor all AI interactions for anomalies, prompt injection attempts, data exfiltration patterns, and policy violations.

Common AI Security Risks​

A high-level map of AI-specific threats that every AI engineer should understand:

  • Prompt Injection – User input overrides system instructions.
  • Sensitive Data Leakage – Model reveals training data, personal information, or system prompts.
  • Model Abuse – Using the model to generate malicious code, phishing emails, or disinformation.
  • Hallucination-Related Risks – Model generates plausible but incorrect information that leads to harmful actions.
  • Insecure Tool Invocation – Agent passes unsanitized data to a tool, enabling injection or escalation.
  • Excessive Agent Permissions – Agent has broad access and is tricked into performing destructive operations.
  • Retrieval Poisoning – Attacker plants malicious documents that the RAG system indexes and uses as context.
  • Supply Chain Risks – Compromised base models, fine-tuning datasets, or dependency libraries.
  • Model Theft – Extracting model weights or architecture through API queries.
  • AI API Abuse – Using the AI endpoint for unauthorized purposes, including bypassing rate limits or content policies.

AI Security Engineering Lifecycle​

Security must be integrated into every stage of the AI system lifecycle:

  • Planning: Define security requirements, classify data sensitivity, and identify regulatory obligations.
  • Architecture: Map trust boundaries between model, tools, data stores, and users. Choose agent permission models.
  • Model Selection: Evaluate security properties of base models. Prefer models with robust safety alignment.
  • Prompt Design: Write system prompts that resist injection. Separate instructions from user data.
  • Knowledge Integration: Vet data sources for RAG. Enforce document-level access controls.
  • Tool Integration: Sandbox tool execution. Validate all parameters. Apply least privilege.
  • Deployment: Use secure CI/CD for model pipelines. Scan dependencies. Sign artifacts.
  • Monitoring: Log prompts, completions, tool calls. Detect anomalies. Alert on injection attempts.
  • Continuous Improvement: Incorporate security findings into prompt engineering, fine-tuning, and architecture updates.

Start your AI security journey with this structured sequence:

  1. AI Security Fundamentals – Understand the core threats and the AI security mindset.
  2. LLM Security Fundamentals – Dive into the threat landscape for LLM-based applications.
  3. Prompt Injection Explained – Master the most critical AI-specific vulnerability.
  4. RAG Security – Secure the retrieval layer that grounds LLM responses.
  5. AI Agent Security – Learn to contain autonomous agents.
  6. MCP Security – Architect secure tool connections.
  7. AI Identity and Access Management – Build identity systems for AI.
  8. AI Governance – Establish responsible and auditable AI operations.
  9. Secure AI Application Architecture – Integrate all controls into a cohesive design.

Relationship to Other Sections​

AI security builds upon and extends the entire SecurityDevPro knowledge base:

  • Foundations – The principles of CIA triad, least privilege, defense in depth, and threat modeling apply directly to AI systems.
  • Application Security – AI applications are software. They need secure APIs, proper authentication, authorization, input validation, and secure coding.
  • Cloud & Infrastructure Security – AI workloads run on cloud infrastructure. Model endpoints, vector databases, and agent containers need network security, IAM, and secrets management.
  • DevSecOps – Model training pipelines, prompt versioning, and agent deployment require the same supply chain security, SAST, and CI/CD hardening as traditional software.

Essential reading from this section:

Future of AI Security​

AI security is evolving rapidly alongside the technology it protects. Key trends shaping the near future include:

  • Enterprise AI platforms: Organizations are standardizing on platforms that embed security controls, identity, and governance. Security teams must understand these platforms' trust models.
  • Agentic AI: As agents become more autonomous and interconnected, security will demand robust inter-agent trust frameworks and decentralized authorization.
  • Secure autonomous systems: AI systems that control physical infrastructure or make medical decisions require safety-critical engineering practices.
  • AI-native software engineering: AI assistants generate, review, and deploy code. Securing the AI-driven SDLC is a new frontier.
  • AI security standards: Frameworks like NIST AI RMF, ISO 42001, and the EU AI Act are translating principles into concrete compliance requirements.
  • Runtime policy enforcement: Real-time guardrails that intercept unsafe model outputs or block dangerous tool calls before they execute.

Staying ahead means treating AI security as a continuous learning discipline, not a static set of rules.

Key Takeaways​

AI and Agent Security is not a separate field from traditional security engineering—it is an extension. The same principles of defense in depth, least privilege, Zero Trust, and continuous verification apply, but they must be adapted to the probabilistic, language-driven, and tool-invoking nature of modern AI systems.

  • AI introduces new attack surfaces – Prompt injection, insecure tool use, and retrieval poisoning require AI-specific defenses.
  • Traditional security is still essential – Without strong identity, network, and application security, AI-specific controls rest on a fragile foundation.
  • Agents amplify risk – Autonomous execution demands strict permission boundaries, human approval workflows, and comprehensive logging.
  • Governance is an engineering concern – Audibility, explainability, and oversight must be built into systems, not bolted on after an incident.

The articles in this section will give you the knowledge to design, build, and operate AI systems that are secure by design. Start with the fundamentals, then progress through the learning path as you integrate AI into your products and platforms.