3.4: Tool Qualification for AI

What You'll Learn

By the end of this chapter, you will be able to:

• Explain the ISO 26262 tool qualification framework and its requirements
• Classify AI tools using TI/TD/TCL and determine qualification effort
• Compare ISO 26262 and DO-178C tool qualification approaches
• Select appropriate qualification methods (1a, 1b, 1c, 1d)
• Address AI-specific challenges: non-determinism, model updates, versioning
• Apply governance frameworks and build tool qualification plans
• Execute a practical step-by-step tool qualification workflow

Why Tool Qualification?

For safety-critical systems, tools that can introduce errors affecting safety must be qualified. AI tools present unique qualification challenges:

1. Non-determinism: Same input may produce different outputs
2. Opacity: Internal decision-making is not transparent
3. Continuous learning: Behavior may change over time
4. Error patterns: Fail in unexpected ways

ISO 26262 Tool Classification

Tool Impact (TI)

Tool Error Detection (TD)

Tool Confidence Level (TCL)

The TCL matrix combines TI and TD:

TCL Classification in Detail

Each Tool Confidence Level carries specific implications for the qualification effort required before the tool may be used in a safety-critical project.

TCL 1 --- No Qualification Required

TCL 1 applies when the tool either has no impact on safety-related outputs (TI1) or when any errors the tool might introduce will be detected with high confidence by downstream processes (TI2 + TD1). No formal tool qualification activity is needed, though the tool should still be listed in the project tool inventory.

TCL 2 --- Partial Qualification Required

TCL 2 applies when the tool has some potential to introduce errors that might affect safety, but mitigating factors exist. Qualification focuses on demonstrating that the tool is suitable for its intended use through one or more methods defined in ISO 26262-8, Clause 11.4.6.

TCL 3 --- Full Qualification Required

TCL 3 applies when the tool has high potential to introduce undetected errors into safety-related outputs. This is the most demanding qualification level and requires rigorous evidence that the tool functions correctly for its intended use cases.

Important: The TCL assignment is not a property of the tool alone. It depends on the combination of the tool and its usage context. The same AI code generator might be TCL 1 in one project (where comprehensive verification catches all errors) and TCL 3 in another (where its output is trusted without independent checking).

Qualification Strategies for AI Tools

Strategy 1: Non-Critical Path

Approach: Use AI tools only for non-safety outputs.

Examples:

• AI generates documentation (not safety documentation)
• AI suggests refactoring (human reviews before safety code)
• AI assists with non-safety test creation

Result: TI1 - TCL1 (No qualification required)

When to Use:

• AI can provide value outside safety-critical path
• Safety-critical work done with qualified tools
• Clear separation between paths

Strategy 2: Verification Overlay

Approach: Use qualified verification to catch AI tool errors.

Examples:

• AI generates code - qualified static analysis + review
• AI generates tests - human review + qualified test execution
• AI suggests design - qualified review process

Result: TI2/TI3 + TD1 - TCL1/TCL2 (Reduced qualification)

When to Use:

• AI provides significant efficiency gain
• Qualified verification processes exist
• Verification can detect AI errors

Strategy 3: Secondary Check

Approach: AI supplements but does not replace qualified tools.

Examples:

• Qualified static analysis + AI additional pattern detection
• Qualified review + AI first-pass screening
• Qualified test tool + AI coverage suggestions

Result: AI is TI1 (no safety impact) - TCL1

When to Use:

• Qualified tools already in place
• AI can add value without replacing
• Clear delineation of responsibilities

AI Tool Governance

Policy Level

Process Level

Technical Level

Monitoring Level

Tool Qualification Methods

ISO 26262-8, Clause 11.4.6 defines four methods for qualifying tools. These methods can be used individually or in combination, depending on the TCL and the nature of the tool.

Method 1a: Increased Confidence from Use

Demonstrate that the tool has been used successfully in similar contexts and that its error history is acceptable.

Method 1b: Evaluation of the Tool Development Process

Evaluate whether the tool was developed according to a recognized quality or safety standard.

Method 1c: Validation of the Tool

Validate the tool by running a test suite that exercises the tool's features as used in the project.

Method 1d: Development According to a Safety Standard

The tool itself is developed in compliance with a safety standard (e.g., ISO 26262 for the tool's own development).

Tip: For most AI tools in safety-critical projects, Method 1c (Validation of the Tool) combined with a Verification Overlay strategy (see Strategy 2 above) provides the most practical qualification path. Build a validation suite that covers the specific use cases, and complement it with qualified downstream verification.

Qualification Evidence

For TCL2/TCL3, qualification evidence may include:

Evidence Requirements by TCL

The depth and breadth of evidence scales with the TCL assignment.

AI Tool Classification

AI tools used in safety-critical development span a wide range of capabilities and risk profiles. Proper classification is the first step toward determining the appropriate qualification effort.

Classification by Function

Note: The classifications above assume the mitigation strategies described in this chapter (verification overlays, human review, qualified downstream tools). Without those mitigations, the same tools would typically receive higher TI ratings and consequently higher TCL assignments.

Classification Decision Process

To classify an AI tool for a specific project, answer these questions in order:

1. Does the tool output directly contribute to a safety-related work product? If no, the tool is TI1 regardless of other factors.
2. If yes, could an error in the tool output propagate to the final safety-related deliverable? If not (because other processes catch it), the tool is TI2 at most.
3. If yes, what is the confidence that downstream processes will detect the error? This determines TD1, TD2, or TD3.
4. Combine TI and TD using the TCL matrix to arrive at the tool's TCL for this specific use context.

ISO 26262 Part 8: Tool Qualification Requirements

ISO 26262 Part 8 (Supporting Processes), Clause 11 defines the normative requirements for software tool qualification in automotive functional safety. This section summarizes the key requirements relevant to AI tool qualification.

Scope and Applicability

Tool qualification under ISO 26262 applies to any software tool that is used in the development of safety-related systems and that could potentially introduce or fail to detect errors in safety-related work products. This includes AI tools when they are used in the safety-critical path.

Key Principles

Principle 1: Tool qualification effort is proportional to the risk the tool introduces. A tool that cannot affect safety needs no qualification. A tool that can introduce undetected safety errors requires full qualification.

Principle 2: Tool qualification is use-case-specific. A tool qualified for one project and one set of use cases is not automatically qualified for a different project or different use cases.

Principle 3: Downstream verification can reduce the qualification effort. If errors introduced by the tool are reliably caught by subsequent qualified activities, the tool's effective TCL is lowered.

DO-178C Tool Qualification: Aviation Comparison

DO-178C (Software Considerations in Airborne Systems and Equipment Certification) uses a parallel but distinct tool qualification framework. Comparing the two approaches helps teams working across automotive and aerospace domains.

DO-178C Tool Qualification Levels (TQL)

TQL assignment depends on the software level (DAL) of the software the tool supports and whether it is a development tool (Criteria 1: output becomes airborne software without independent verification) or a verification tool (Criteria 2: may fail to detect errors; Criteria 3: automates verification activities).

Important: There is no DO-330 equivalent to ISO 26262 TCL 1 (no qualification required). All tools that meet the TQL assignment criteria must be qualified to the applicable TQL level. TQL-5 is the minimum rigor level, not a "no qualification" category.

Key Differences Between ISO 26262 and DO-178C

Note: DO-178C explicitly categorizes tools into development tools (whose output becomes part of the software) and verification tools (that could fail to detect errors). This distinction maps approximately to TI3 (development tools) and TI2 (verification tools) in ISO 26262.

AI-Specific Challenges

AI tools present qualification challenges that go beyond those of traditional deterministic software tools. These challenges must be addressed explicitly in any tool qualification plan for AI.

Detailed Challenge Analysis

Non-Determinism and Reproducibility

Traditional tool qualification assumes that a tool produces the same output for the same input. AI tools, particularly LLM-based tools, violate this assumption. This means that a validation test suite cannot simply compare outputs against fixed expected results.

Model Updates and Version Control

AI tools receive model updates that can fundamentally change their behavior without changing their version number or user interface. A model update is functionally equivalent to replacing the tool with a different tool.

Versioning and Configuration Management

AI tool versioning is more complex than traditional tool versioning because the "tool" consists of multiple components: the client software, the model weights, the system prompt, the temperature and sampling parameters, and any fine-tuning data.

Mitigation Approaches

Practical Qualification Process

This section provides a step-by-step workflow for qualifying an AI tool in a safety-critical project.

Step 1: Inventory and Identify

List all AI tools used in the project. For each tool, record:

Step 2: Classify Each Tool-Use-Case Pair

For each use case identified in Step 1, determine TI, TD, and TCL.

Step 3: Create Tool Qualification Plan (TCL 2 and TCL 3)

For each tool with TCL 2 or TCL 3, create a tool qualification plan. See the template in the next section.

Step 4: Execute Qualification Activities

Execute the qualification methods selected in the plan (Method 1a, 1b, 1c, or 1d).

Step 5: Document Results and Known Limitations

Compile all qualification evidence into a tool qualification report.

Step 6: Establish Monitoring and Re-Qualification Triggers

Define the conditions under which the tool must be re-qualified. Monitor these conditions continuously.

Tool Qualification Plan Template

A tool qualification plan for an AI tool should contain the following elements.

Tip: Maintain the tool qualification plan as a living document under configuration management. Update it whenever the tool, its configuration, or its use context changes.

Industry Examples

The following examples illustrate how specific AI tools might be classified and qualified in a safety-critical automotive project.

Example 1: AI Code Generator for ASIL B ECU Software

Example 2: AI-Assisted Static Analyzer

Example 3: AI Requirements Drafting Assistant

Implementation Checklist

Use this checklist to ensure complete coverage of tool qualification activities for AI tools in a safety-critical project.

Tool Inventory and Classification

• All AI tools used in the project are listed in the tool inventory
• Each tool entry includes: name, version, vendor, model ID, configuration
• Each tool-use-case pair has a documented TI/TD/TCL classification
• Classification rationale references ISO 26262-8 Clause 11.4
• Assumptions about downstream verification are documented and validated

Qualification Planning (TCL 2 and TCL 3)

• Tool qualification plan created for each TCL 2 and TCL 3 tool
• Qualification method(s) selected and justified
• Validation suite designed covering project-specific use cases
• Pass/fail criteria defined (semantic equivalence for non-deterministic tools)
• Known anomalies and limitations documented
• Operating constraints specified (model version, prompt version, parameters)
• User guidelines written and distributed to tool users
• Re-qualification triggers defined

Qualification Execution

• Validation suite executed and results documented
• All test failures analyzed and dispositioned
• Tool qualification report completed and approved
• Qualification evidence archived under configuration management

AI-Specific Controls

• Model version pinned and recorded in configuration management
• Prompt templates version-controlled alongside project artifacts
• Temperature and sampling parameters locked in project configuration
• Automated regression suite established for re-validation after model updates
• Model update monitoring process in place (vendor notifications, periodic checks)
• Non-determinism addressed in validation approach (multiple runs, semantic checks)

Governance and Monitoring

• AI tool governance policy established (approved tools, permitted use cases, restrictions)
• HITL patterns defined for each AI tool use case (see 3.2 HITL Patterns)
• Audit trail requirements implemented (input/output logging)
• Tool performance metrics collected (acceptance rate, error rate, override rate)
• Periodic review schedule established (quarterly minimum)
• Escalation procedures defined for tool failures or qualification concerns

Summary

Tool qualification for AI requires a systematic approach spanning classification, strategy, governance, and ongoing monitoring:

1. Classification: Determine TI, TD, and TCL for each tool-use-case pair. The same tool may have different TCL assignments depending on its usage context and downstream verification.
2. Strategy Selection: Choose from non-critical path (avoid safety impact), verification overlay (catch errors downstream), or secondary check (supplement qualified tools).
3. Qualification Methods: Apply ISO 26262-8 Methods 1a-1d proportional to the TCL. For AI tools, Method 1c (validation suite) is typically the most practical primary method.
4. Evidence: Scale documentation effort to the TCL level --- from simple inventory entries for TCL 1 to comprehensive qualification reports for TCL 3.
5. AI-Specific Controls: Address non-determinism through semantic validation, manage model updates through version pinning and automated regression, and version-control all configuration components including prompts and parameters.
6. Cross-Standard Awareness: Understand how ISO 26262 TCL maps to DO-178C TQL when working across automotive and aerospace domains.
7. Governance and Monitoring: Maintain ongoing oversight through tool performance metrics, periodic re-validation, and defined re-qualification triggers.

The goal is not to avoid AI but to use it safely with appropriate qualification. A well-designed qualification approach enables organizations to capture the productivity benefits of AI tools while maintaining the safety integrity that standards demand.