Certified Tester AI Testing (CT-AI)

This three-day course provides a comprehensive introduction to Artificial Intelligence (AI) and its application in modern systems. Participants will explore the foundational concepts of AI, including its types, technologies, and development frameworks, as well as the unique quality characteristics that distinguish AI-based systems—such as autonomy, adaptability, ethics, and transparency. The course also covers the essentials of Machine Learning (ML), from algorithm selection and data preparation to performance metrics and neural networks, equipping learners with a solid understanding of how ML models are developed and evaluated.

Building on this foundation, the course delves into the challenges and methodologies of testing AI-based systems. Learners will examine test strategies for AI-specific traits like bias, non-determinism, and concept drift, and gain hands-on insight into techniques such as adversarial testing, metamorphic testing, and A/B testing. The final sessions focus on test environments and the use of AI to enhance software testing processes, including defect analysis and regression optimization. By the end of the course, participants will be equipped to critically assess, test, and apply AI technologies in real-world scenarios.

Prerequisites

The entry criterion for taking the Certified Tester AI Testing exam is that candidates have acquired the ISTQB® Certified Tester Foundation Level certification.

Target audience

The Certified Tester AI Testing is suitable for anyone who is involved in testing as well as anyone interested in AI-based systems. This includes people performing activities such as test analysis, test consulting and software development.

The syllabus provides testing knowledge for anyone working with Agile or sequential software development lifecycles.

1. Introduction to Artificial Intelligence

• Overview of AI and its significance in modern systems
• Comparison between AI-based and conventional systems
• Explanation of narrow AI, general AI, and super AI
• Survey of different types of AI technologies
• Introduction to generative AI and its applications
• Overview of hardware requirements for machine learning systems
• Processes for developing and hosting AI models
• Introduction to machine learning development frameworks
• Overview of regulations and standards relevant to AI

2. Quality Characteristics for AI-Based Systems

• Identification of quality characteristics unique to AI-based systems
• Discussion of AI-specific quality attributes and their implications
• Consideration of safety aspects in AI-based systems
• Establishing acceptance criteria tailored for AI-based solutions

3. Machine Learning Fundamentals

• Introduction to machine learning concepts and terminology
• Exploration of different forms of machine learning
• Understanding the machine learning workflow from data to model
• Practical exercise: Creating a machine learning model
• Use of pretrained models, fine-tuning, and retrieval-augmented generation

4. Data for Machine Learning

• Key activities in data preparation for machine learning
• Practical exercise: Preparing data to support model creation

5. Evaluating Machine Learning Models

• Functional performance metrics for classification tasks
• Calculation and interpretation of machine learning performance metrics
• Practical exercise: Evaluating a machine learning model using selected metrics
• Demonstrating the impact of different models and dataset combinations

6. Neural Networks

• Structure and operation of deep neural networks
• Practical exercise: Implementing a perceptron
• Coverage measures for neural networks

7. Testing AI-Based Systems

• Introduction to the challenges of testing AI-based systems
• Differences between locked and adaptive AI-based systems
• Rationale for using statistical approaches in AI testing
• Role and design of test oracles for AI-based systems

8. Testing Generative AI and Large Language Models

• Approaches to testing generative AI systems
• Introduction to red teaming in the context of AI
• Practical exercise: Exploratory testing of a large language model

9. Test Levels and Risk-Based Testing for Machine Learning Systems

• Test levels applicable to machine learning systems
• Principles of risk-based testing for AI and machine learning

10. Input Data Testing for Machine Learning Systems

• Identification and mitigation of input data risks
• Techniques for testing bias in datasets
• Data pipeline testing strategies
• Assessing data representativeness and dataset constraints
• Label correctness testing methods
• Practical exercise: Input data testing

11. Model Testing for Machine Learning Systems

• Identification and mitigation of machine learning model risks
• Documentation and review processes for machine learning models
• Functional performance testing for probabilistic models
• Adversarial and metamorphic testing techniques
• Practical exercise: Applying metamorphic testing
• Drift testing, and testing for overfitting and underfitting
• A/B testing and back-to-back testing approaches

12. Machine Learning Development Testing

• Risks and mitigations in machine learning development
• Deployment testing for machine learning systems

Exams and Assessments

Your course fee includes an iSQI voucher for the examination which you will book at a later date.

The format of the exam is multiple choice.

• Exam duration is 60 minutes. If the candidate’s native language is not the examination language, the candidate is allowed an additional 25% (exam duration = 75 minutes).
• There are 40 questions.
• To pass the exam, at least 65% of the total sum of points must be answered correctly.
• The total number of points for this exam should be set at 47 points. Therefore, a minimum of 31 points is required to achieve a passing score.

Hands-On Learning

Hands-on Machine Learning Concepts:

Learners engage in exercises that illustrate key ML concepts such as overfitting and underfitting. Activities include creating simulated datasets, training simple models (like linear regression), and visualizing model performance under different data conditions (e.g., limited data, weak feature-target correlations). Participants analyze results using metrics like Mean Squared Error (MSE) and R², and interpret graphical outputs to understand model behavior.

Test Design and Reduction Techniques:

One exercise focuses on combinatorial test design. Learners are tasked with defining a model with multiple parameters (e.g., model type, number of estimators, training rate, etc.), generating a large set of possible parameter combinations, and then applying pairwise testing to reduce the number of test cases. This introduces practical skills in test optimization and the use of tools (such as Microsoft PICT) for efficient test coverage.

By the end of this course, learners will be able to:

• Understand the foundational concepts of Artificial Intelligence, including its types, technologies, and development frameworks.
• Explore the quality characteristics specific to AI-based systems, such as adaptability, autonomy, ethics, and transparency.
• Gain a comprehensive overview of Machine Learning (ML), including its forms, workflows, and algorithm selection criteria.
• Learn the importance of data in ML, including data preparation, dataset types, and the impact of data quality on model performance.
• Evaluate ML performance using functional metrics and benchmark suites for classification, regression, and clustering tasks.
• Understand neural networks and their testing methodologies, including coverage measures and concept drift.
• Apply testing strategies tailored to AI-based systems, addressing specification, test levels, and automation bias.
• Examine challenges in testing AI-specific quality traits, such as bias, non-determinism, and explainability.
• Explore various testing techniques for AI systems, including adversarial testing, metamorphic testing, and A/B testing.
• Discover how AI can be leveraged to enhance software testing processes, including defect analysis, test case generation, and regression optimization.