Course Outline
Module 1 — Failure Modes in AI Applications
Lab: None — Architecture walkthrough & discussion
Establishing a developer-centric mental model of the attack surface.
Topics:
- LLM, RAG, and agent architectures from a developer’s perspective
- The request/response lifecycle of AI features
- Prompt flow: system, developer, user, and tool messages
- Entry points for untrusted data into the model (and re-entry points)
- Trust boundaries owned versus inherited by the developer
- The semantic rather than syntactic nature of AI attacks
- Mapping the OWASP LLM Top 10 to code implementations
Key Insight: Any point where untrusted text reaches the model, or where model output enters application code, represents a trust boundary under the developer’s control.
Module 2 — Prompt Injection for Builders
Lab: Lab 01 — 01-Prompt-Injection
The “SQL injection equivalent” for AI, where complete prevention is impossible.
Topics:
- Direct versus indirect prompt injection
- Hidden instructions embedded in documents, web pages, or tool outputs
- Jailbreak techniques and role confusion
- The importance of separating instructions from data
- Defensive prompt design (using delimiters, structure, and minimal authority)
- The limits of prevention and the need for containment strategies
Hands-on Activities:
- Simulating attacks on one’s own chatbot
- Bypassing naive filter mechanisms
- Restructuring prompts to minimize the blast radius
Module 3 — Treating Model Output as Untrusted
Lab: Lab 02 — 02-Output-Handling
The most frequently underestimated bug class.
Topics:
- Viewing model output as untrusted input for the application
- Insecure output handling (LLM02): downstream XSS, SSRF, command/SQL injection
- Prohibiting eval/exec/rendering of raw model output
- Structured outputs and schema validation
- Output encoding and allowlists
- Safe rendering in web/UI contexts
Hands-on Activities:
- Identifying and remediating insecure output handling vulnerabilities
- Enforcing JSON schemas on model responses
Module 4 — RAG Security
Lab: Lab 03 — 03-RAG-Security
A significant new attack surface that developers must construct securely.
Topics:
- Threats to vector databases and retrieval processes
- Ingestion sanitization techniques
- Document provenance and trust scoring
- Retrieval scoping and metadata isolation
- Hidden instructions in retrieved content (indirect injection)
- Data exfiltration via retrieval mechanisms
Hands-on Activities: - Inject malicious documents into a RAG pipeline - Implement ingestion sanitization and retrieval scoping to defend against it
Module 5 — Agent & Tool Safety
Lab: Lab 04 — 04-Agent-Safety
The point where logical bugs translate into executed actions.
Topics:
- Excessive agency (LLM06) and tool abuse
- Least-privilege principles for agents
- Tool allowlists and argument validation
- Approval gates and human-in-the-loop mechanisms
- Sandboxing tool execution
- Scoped, short-lived credentials for agents
- Limiting autonomous loops and chaining
Hands-on Activities:
- Restricting over-permissioned agents
- Implementing allowlists and approval gates for dangerous tools
Module 6 — Secrets, Identity & Cost
Lab: Lab 05 — 05-Secrets-and-Cost
Operational errors that cause the most rapid harm.
Topics:
- API key and secret management (excluded from prompts, code, and logs)
- Per-user authentication and authorization for AI features
- Propagating user identity to tools and retrieval systems
- Denial-of-wallet: unbounded token/cost consumption
- Rate limits, token budgets, and timeouts
- Logging without exposing secrets or PII
Hands-on Activities:
- Removing secrets from the prompt/code path
- Implementing per-user rate limits and token/cost budgets
Module 7 — Guardrail Libraries
Lab: Lab 06 — 06-Guardrails
Deciding between buying vs. building for input/output safety.
Topics:
- Capabilities and limitations of guardrail frameworks
- Input guardrails: injection/PII/topic classifiers
- Output guardrails: validation, filtering, and grounding checks
- Determining when guardrails are appropriate versus using custom deterministic checks
- Integrating guardrails with controls from previous modules
- Performance, false positives, and failure modes
Hands-on Activities:
- Integrating input/output guardrail layers into AI features
- Measuring detection accuracy and identifying gaps
Module 8 — Red-Teaming Your Own App
Lab: Lab 07 — 07-Red-Teaming
Deploying with the assumption that attackers already have access.
Topics:
- Building abuse/test suites for AI features
- Automated prompt injection and jailbreak testing
- Regression testing of guardrails and policies
- Incorporating AI security checks into CI pipelines
- Model and dependency supply chain (provenance, pinning)
- Pre-deployment security checklists for AI features
Hands-on Activities:
- Writing automated red-team tests for AI features
- Integrating these tests into CI checks
Module 9 — Scoring AI Security: The SAIS-100 Framework
Lab: None — Scoring exercise (utilizing the Capstone app)
Converting built features into a repeatable security score.
Topics:
- The AI Security Hexagon: six questions beyond “is it secure?”
- The six scored categories (Data, Prompt, Agent, Supply Chain, Detection, Governance)
- The 100-point rubric and its weightings
- Verdict bands and single-category override rules
- The Elephant Scale Secure AI Score (SAIS-100) as a branded, repeatable framework
- Scoring before and after hardening as a performance metric
Hands-on Activities:
- Scoring the Capstone app on the 100-point scale
- Identifying the single change that most improves the score
Key Insight: The three highest-weighted categories correspond to the trust boundaries owned by the developer, ensuring the score accurately reflects the course’s core teachings.
Capstone
Students harden a deliberately vulnerable AI application end-to-end.
The starter application includes:
- An injectable prompt
- Insecure output handling
- An unscoped RAG pipeline
- An over-permissioned agent
- Secrets within the prompt path
- Lack of cost limits
Students apply course concepts to:
- Restructure prompts for containment
- Validate and encode model output
- Sanitize and scope retrieval processes
- Apply least privilege and approval gates to the agent
- Remove secrets and add cost/rate limits
- Implement guardrails and automated red-team tests
Deliverable: A hardened application and a concise OWASP LLM Top 10 self-assessment.
Module - Lab Map
Labs are executed in sequence, following the module order. The course comprises 9 modules and 7 labs: Module 1 is an architecture walkthrough/discussion, and Module 9 is a scoring exercise; neither has a dedicated lab folder.
- Lab 01 - 01-Prompt-Injection: Attacking the chatbot & designing for containment (Module 2)
- Lab 02 - 02-Output-Handling: Fixing insecure output handling bugs (Module 3)
- Lab 03 - 03-RAG-Security: Poisoning then defending a RAG pipeline (Module 4)
- Lab 04 - 04-Agent-Safety: Restricting over-permissioned agents (Module 5)
- Lab 05 - 05-Secrets-and-Cost: Securing keys + adding cost guardrails (Module 6)
- Lab 06 - 06-Guardrails: Adding input/output guardrail layers (Module 7)
- Lab 07 - 07-Red-Teaming: Automated red-team tests in CI (Module 8)
Module 1 (Failure Modes in AI Applications) has no lab — it is conducted as an architecture walkthrough and discussion. Module 9 (Scoring AI Security) has no lab folder — it is conducted as a scoring exercise against the Capstone app.
Requirements
- Intermediate skill level.
- Proficiency in building and consuming REST APIs, a scripting language (labs utilize Python), basic application authentication, Git, and command-line interfaces is expected.
- No machine learning background is required; this is an application security course for developers building with LLMs, not training them.
Target Audience
- Software / Backend engineers developing LLM features
- Full-stack and API developers
- AI/ML application engineers
- Platform engineers deploying copilots and agents
- Tech leads and senior engineers responsible for AI features
Testimonials (3)
inventory and identifying the different risk exposures within AI
Gary Cook - Cybersecurity and Information Technology Risk Division
Course - Introduction to AI Trust, Risk, and Security Management (AI TRiSM)
I really enjoyed learning about AI attacks and the tools out there to begin practicing and actively using for security testing. I took a lot of knowledge away which I didn't have at the beginning and the course met what I hoped it would be. My favorite part shown from the training was Comet Browser and was amazed at what it could do. Definitely something will be looking into more. Overall it was a great course and enjoyed learning all OWASP GenAI Top 10.
Patrick Collins - Optum
Course - OWASP GenAI Security
The profesional knolage and the way how he presented it before us