Certified AI Security Engineer

Participants should have:

• A foundational understanding of AI concepts such as neural networks and model lifecycle stages
• Basic familiarity with cybersecurity principles and common attack types
• Experience working with applications that use AI or LLM functionality (recommended)
• Access to a development environment suitable for practising AI integration (recommended)

Target audience

This course is designed for:

• Technology professionals responsible for deploying, integrating, or securing AI solutions
• Security practitioners seeking a deeper understanding of AI-specific threats
• Developers building applications that use large language models or generative AI
• Organisations aiming to enhance their resilience against AI-driven risks

By the end of this course, learners will be able to:

• Describe different types of AI systems and explain their security vulnerabilities
• Identify and mitigate attacks such as prompt injection, model jailbreaks, visual prompt manipulation, and denial of service
• Apply defensive methods and security API tooling to strengthen AI systems
• Assess and protect training data sources, model integrity, and supply chain dependencies
• Integrate large language models securely within applications, respecting trust boundaries and common best practices
• Evaluate ethical considerations, responsible AI principles, and techniques to improve reliability and explainability
• Investigate model behaviour, detect potential misuse, and apply structured threat modelling for AI-driven workflows
• Build secure human-AI interaction patterns that minimise hallucinations, misuse, and exposure of sensitive information

Introduction to AI security

• Defining AI and defining security
• Scope of AI security and the boundaries of this course
• Types of AI systems: neural networks, models, integrated systems
• How AI systems are used across organisational contexts
• What secure AI means: responsible, reliable, explainable, and aligned models
• Human-AI interactions and risks of uncensored or malicious models
• Real-world examples of misuse including deepfakes, voice cloning, and social engineering
• How misinformation spreads through AI-generated content
• Exercises exploring uncensored models and image watermarking

The AI security landscape

• Attack surfaces of AI systems across the model lifecycle
• Components of AI pipelines and why supply-chain security matters
• Models accessed via APIs and APIs accessed by models
• Non-AI attack vectors that remain relevant
• OWASP ML Top 10, OWASP LLM Top 10, and how they apply to modern AI
• Threat modelling approaches for AI-integrated applications
• Sample AI-powered workflows and common security findings
• Exercise: threat modelling an LLM-integrated application using a realistic data flow

Prompt injection

• Overview of prompt injection attacks and their impact
• Direct and indirect prompt injection
• Social engineering through prompts and phishing opportunities
• SudoLang for representing attack logic
• How LLM integration choices influence vulnerabilities
• Exercises translating prompts into SudoLang and retrieving passwords across levels 1 and 2

Model jailbreaks

• How jailbreaks work and common techniques
• Case studies including DAN prompts and AutoDAN
• Tree of Attacks with Pruning (TAP)
• Exercises retrieving restricted information across levels 3, 4, and 5

Prompt extraction

• Extracting system prompts, private data, and boundaries
• Techniques used in challenges and real applications
• Exercises retrieving prompts and boundaries at levels 6 and 7

Defending AI systems

• Intermediate and advanced defence strategies
• Security APIs including ReBuff, Llama Guard, Lakera, and similar tools
• Example exploits seen in public challenges
• Exercise: defeating protections in levels 8 and 9
• Other injection methods including reverse psychology and manipulation techniques
• Categorising attacks and implementing robust protections
• Additional defensive models and structured methods such as the Bergeron method

Visual prompt injection

• How visual prompts manipulate multimodal models
• Trivial examples and advanced adversarial attacks
• Examples affecting self-driving systems and image classifiers
• Exercises using OpenAI vision capabilities and creating adversarial samples
• Protections against visual attacks and dataset considerations

Denial of service

• How DoS attacks manifest in LLMs and chatbots
• Prompt routing challenges and resource exhaustion
• Practical defence strategies and system-level mitigations
• Exercise: designing prompts that halt or degrade model behaviour

Model theft

• Threat landscape for model extraction
• Risks of dataset exploration and query-based stealing
• How fine-tuned models can be cloned
• Exercises using API parameters to replicate model behaviour
• Protections for model confidentiality, from simple rate limits to advanced monitoring

LLM integration

• Understanding the LLM trust boundary
• Classical integration challenges in novel AI workflows
• Treating LLM output as untrusted user input
• Exchange formats and secure function calling
• Risks of custom GPTs, identity flow, and cross-application access
• Exercises on SQL injection, XSS payload generation, invalid parameter passing, and privilege escalation
• Principles of secure coding applied to AI systems including Bishop, Saltzer, and Schroeder
• Designing privilege boundaries for AI components
• Exercise: breaking out of an AI sandbox

Training data manipulation

• Importance of dataset integrity and reliability
• How attackers poison training data
• Using dataset cards and model cards for assurance
• Analysing datasets and reviewing dataset objectives
• Exercises constructing and analysing malicious datasets

Secure supply chain

• Proving model integrity and emerging cryptographic methods
• Hardware-assisted attestation and verification
• Risks across the model building and deployment lifecycle

Human-AI interaction

• Overreliance on LLM output and what can go wrong
• Countering hallucinations and validating information
• Sandboxing and safe API patterns
• Exercise: verifying LLM output in realistic scenarios

Secure AI infrastructure

• Requirements of secure AI infrastructure including monitoring, observability, and traceability
• Confidentiality, integrity, availability, and privacy considerations
• Case studies such as the Samsung data leak
• Tools and frameworks including LangSmith
• Exercise: experimenting with LangSmith for safe evaluation
• BlindLlama and emerging evaluation tools

Exams and assessments

• The independent APMG Certified AI Security Engineer exam is taken post class, using an exam voucher code via the APMG proctor platform.
• If you experience any issues, please contact the APMG technical help desk on 01494 4520450.
• Duration: 60 Minutes
• Questions: 60, multiple choice (4 multiple choice answers only 1 of which is correct)
• Pass Mark: 50%

Hands-on learning

This course provides extensive practical experience through:

• Interactive labs exploring AI-based attacks and defences
• Real-world scenarios simulating risks across predictive, generative, and multimodal systems
• Guided exercises using security APIs, structured defences, and dataset analysis
• Instructor-led walkthroughs that reinforce secure design, coding, and integration behaviours