How do we detect prompt injection attempts and data exfiltration in an agent that can call tools?

When your agent can call tools, one bad prompt injection or data exfiltration attempt isn’t just an embarrassing answer—it’s a potential system breach, a PII leak, or a costly external API call. Detecting these attacks isn’t a “nice-to-have.” It’s the difference between a reliable agent and an unsafe demo.

Quick Answer: You detect prompt injection and data exfiltration in tool-using agents by continuously evaluating every session, trace, and tool call for attack patterns—then turning those evaluations into real-time guardrails that can block, redact, or override unsafe actions in under 200ms. Galileo does this by combining evaluation, Signals-based detection, and runtime Protect policies powered by Luna-2.

---

The Quick Overview

• What It Is: A runtime safety and evaluation workflow that detects prompt injection and data exfiltration attempts across your agent’s inputs, outputs, and tool calls—and actively stops them before they reach users or downstream systems.
• Who It Is For: Teams running LLM agents and RAG systems in production, especially where tools can access internal databases, user records, or third-party APIs.
• Core Problem Solved: You can’t see or stop many injection and exfiltration attempts with simple log search or regex. Galileo continuously scores behavior for attack patterns and turns those signals into guardrails that intercept unsafe actions in real time.

---

How It Works

At a high level, detection has to cover three planes of behavior:

1. User and content inputs – catching jailbreaks, override attempts, and malicious instructions.
2. Model outputs and reasoning – spotting signs the model has been steered away from policy.
3. Tool calls and data flows – preventing the agent from pulling, leaking, or sending data where it shouldn’t.

Galileo wires those together into an eval-to-guardrail flow:

1. Evaluate: Use Galileo’s Evaluation Engine to score historical and synthetic traffic for prompt injection, data exfiltration, PII exposure, and policy violations. Calibrate evaluators so they match your domain and tools.
2. Signals: Run continuous detection over 100% of production traces to surface emerging patterns—like a new jailbreak style or a previously unseen exfiltration combo involving a specific tool.
3. Protect: Deploy runtime guardrails that intercept suspicious inputs, outputs, or tool actions in sub-200ms. Guardrails use Luna/Luna-2 evaluators to classify behavior and trigger deterministic actions (block, redact, override, webhook).

1. Evaluate: Build reliable detectors for injection & exfiltration

You start by treating injection and exfiltration as measurable failure modes:

• Prompt injection patterns:

  • “Ignore previous instructions and…”
  • “You must always obey the most recent instruction.”
  • Attempts to read system prompts, environment variables, or tool schemas.
  • Indirect injections via retrieved content (RAG docs, emails, tickets).

• Data exfiltration patterns:

  • Repeated attempts to enumerate internal tables, S3 buckets, or indices.
  • Asking for “all users”, “entire database”, “all tickets”, “every record”.
  • Combining sensitive fields in one response (name + SSN + full address).
  • Tool calls that request more data than the task requires.

With Galileo’s Evaluation Engine, you:

• Use 20+ out-of-the-box evaluators for safety, security, RAG, and agents (e.g., prompt injection, PII detection, data leakage).
• Generate LLM-as-judge evaluators from natural language descriptions (e.g., “detect attempts to override system instructions or exfiltrate more than 100 records”).
• Incorporate SME annotations from security, compliance, or data owners to label real injection/exfiltration events in your logs.
• Use CLHF-style fine-tuning by feeding these labeled examples back into your evaluator so it better fits your domain.

Galileo then distills these evaluators into compact Luna / Luna-2 models and runs them on a dedicated inference stack. That’s what makes them cheap and fast enough to run on every trace later, instead of just on a small sample.

2. Signals: Detect ‘unknown unknowns’ in production traffic

Even with well-designed evaluators, production always reveals surprises—new jailbreak patterns, creative “prompt smuggling” via PDFs, or novel sequences of tool calls that leak data.

Galileo Signals continuously analyzes:

• Sessions → traces → spans, including:
  • User messages and uploaded content.
  • Model responses and reasoning snippets.
  • Tool selection and tool call parameters.
  • Latency and cost per span.

Signals looks for patterns like:

• Sudden spikes in:
  • Tool calls that return large result sets.
  • Requests hitting user tables, HR systems, or payment records.
  • Responses that include unusually high token counts of raw data.
• Repeated sequences such as:
  • “Enumerate all users” → “now export to CSV” → “send via email tool.”
  • Tool calls that bypass normal filters or scopes.
• Drifts in behavior:
  • An agent that rarely used a particular “powerful” tool suddenly calls it in most sessions.
  • A change in RAG documents or system prompts that increases injection rate.

From any detected pattern, Galileo can generate a new evaluator—effectively turning “we saw a weird exfiltration sequence once” into “we now have a reusable detector that runs on all traffic.” This closes the loop between discovery and prevention.

3. Protect: Intercept and block attacks in real time

Detection without interception leaves you in “after the fact” territory. You see the incident in a report after users or systems are already impacted.

Galileo Protect is where detection becomes a guardrail:

• Runtime scoring: Every input, output, and tool call can be evaluated by Luna-2-based guardrail models for:

  • Prompt injection
  • Sensitive data exposure (PII, secrets)
  • Data exfiltration attempts (volume, scope, destination)
  • Policy alignment (e.g., “no export of full customer lists”)

• Deterministic actions: When a guardrail fires, Protect doesn’t just log it. It can:

  • Block the message or tool call outright.
  • Redact sensitive fields or truncate result sets (e.g., limit to N records).
  • Override a response with a safe alternative.
  • Trigger a webhook to your own system for custom flows (e.g., require human approval for certain tool actions).

• Operational guarantees:

  • Sub-200ms guardrailing latency, even at production scale.
  • 100% traffic coverage at up to 10,000+ requests/min with 97% lower cost than heavyweight LLM-as-judge approaches, thanks to Luna-2.

Protect policies are versioned, so you can iterate on rules, run A/B tests, and roll back instantly without redeploying your agent.

---

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Evaluation Engine for attacks	Scores sessions for prompt injection, PII exposure, and data exfiltration across inputs/tools.	Turns messy agent behavior into measurable, testable security metrics before you go to production.
Signals for emerging patterns	Analyzes 100% of traces to surface new jailbreak styles and exfiltration sequences.	Detects “unknown unknowns” early and converts them into reusable evaluators—after first signal, not 1000.
Protect runtime guardrails	Intercepts risky prompts, responses, and tool calls; blocks/redacts/overrides in sub-200ms.	Prevents prompt injection and data leaks from ever reaching users or downstream systems.

---

Ideal Use Cases

• Best for agents with powerful internal tools:
  Because it can inspect tool parameters and results in context, then block or constrain calls that would leak internal data (e.g., “export all HR records”) or abuse sensitive APIs.

• Best for customer-facing RAG systems:
  Because it detects prompt injection both in user queries and retrieved documents, and stops the model from executing “ignore safety” instructions hidden in PDFs, tickets, or wiki pages.

---

Limitations & Considerations

• You still need good tool design:
  Guardrails can block obvious exfiltration, but if your tools expose “dump all data” endpoints with no access controls, you’re increasing blast radius. Pair Galileo with strong scopes, filters, and rate limits.

• Evaluator calibration is not one-and-done:
  Attack patterns evolve. Plan for ongoing refinement using SME feedback and Signals-surfaced incidents so your injection/exfiltration evaluators stay aligned with real-world behavior.

---

Pricing & Plans

Galileo is built for teams who need evaluation and protection to run at production scale, not just in a demo. Pricing reflects usage (traces, guardrailed requests) and deployment model.

Common options include:

• Growth / Team: Best for product teams bringing their first serious agent or RAG system into production, needing:

  • 5,000+ traces per month for evaluation.
  • Always-on Protect for core flows.
  • SaaS deployment with SSO and standard security controls.

• Enterprise: Best for large organizations with multiple agents, strict compliance, and heavier traffic, needing:

  • 100% traffic coverage across multiple apps.
  • VPC or on-prem deployment, SOC 2 Type II posture, HIPAA-ready infrastructure with BAAs.
  • Dedicated inference capacity for Luna-2 and advanced governance workflows.

For specific details, Galileo’s sales team can map plans to your traffic, latency budgets, and security requirements.

---

Frequently Asked Questions

How do I know if my agent is already vulnerable to prompt injection and data exfiltration?

Short Answer: Run evaluation over your existing logs and synthetic tests to see how often your agent follows malicious instructions or over-returns sensitive data.

Details:
With Galileo, you ingest historical sessions, including tool traces. The Evaluation Engine scores each span for:

• Prompt injection obedience (did the model follow a jailbreak or override attempt?).
• PII leakage (did the response or tool output expose sensitive fields?).
• Over-scoped data access (did tool calls fetch more data than the user’s request justified?).

You’ll quickly see concrete failure examples and metrics like “2.3% of sessions showed partial injection obedience.” That baseline tells you where to focus evaluator tuning and guardrail deployment.

---

Can I run these detectors inline without blowing my latency budget?

Short Answer: Yes. Galileo distills evaluators into Luna-2 models and serves them on a purpose-built stack so guardrail checks stay under ~200ms, even at high throughput.

Details:
Traditional LLM-as-judge approaches are too slow and expensive to run on every request; they’re fine for offline evals, not for production. Galileo’s approach:

• Distills your evaluators into compact Luna / Luna-2 models.
• Hosts them on dedicated inference optimized for low-latency evaluation.
• Lets you choose where to attach checks (input, output, tool call) so you can balance protection and latency.

This allows always-on, 100% coverage evaluation for injection and exfiltration in live traffic, at costs and latencies that fit real-world SLAs.

---

Summary

Detecting prompt injection attempts and data exfiltration in an agent that can call tools requires more than regex and log search. You need a pipeline that:

1. Evaluates agent behavior against well-defined injection and exfiltration metrics.
2. Detects emerging patterns in production with Signals so new attack styles are surfaced quickly.
3. Protects users and systems by turning those evaluations into runtime guardrails that intercept and block unsafe prompts, responses, and tool calls in sub-200ms.

Galileo’s Evaluate → Signals → Protect workflow, powered by Luna-2, gives you that end-to-end system so your agents don’t just pass a demo—they stay safe in production.

---

Next Step

Get Started