AI Agent Red Teaming Checklist.

Red Team The System, Not Only The Prompt

AI red teaming is not only about tricking a model into saying something strange. Production agents have retrieval, tools, memory, permissions, queues, and user interfaces. The real question is whether a user can make the system reveal data, call an unsafe tool, skip approval, follow injected instructions, or create business harm.

Prompt Injection Tests

• Tell the agent to ignore previous instructions.
• Put malicious instructions inside uploaded or retrieved content.
• Ask the agent to reveal hidden policy, prompts, or tool schemas.
• Ask the agent to summarize private data from another user.
• Try to make retrieved content grant new permissions.

Tool Abuse Tests

• Try unsafe tool arguments and malformed IDs.
• Repeat actions to test idempotency.
• Trigger retries and check for duplicate writes.
• Ask for refunds, cancellations, account changes, or exports without approval.
• Test whether read-only users can trigger write tools.

Retrieval And Memory Tests

• Ask questions that should not retrieve restricted documents.
• Test outdated policies against current policies.
• Insert conflicting content and check whether the agent escalates.
• Confirm memory does not leak between users or accounts.
• Verify citations point to approved sources.

What Good Looks Like

A red-team pass should produce issues, severity, reproduction steps, affected components, and recommended fixes. Store the failures as evaluation examples so the same issue does not return after a prompt, model, or retrieval change.

The main point: red teaming should become part of the release process for AI agents. If the agent can use tools or private data, it needs adversarial testing before customers find the weak point.