4.4: Tool Qualification

Introduction

Safety-critical software development relies on tools (compilers, static analyzers, coverage tools). If these tools introduce errors or fail to detect defects, they can compromise safety. This section provides practical guidance on tool qualification across ISO 26262, IEC 61508, and DO-178C standards.

Why Tool Qualification Matters

The Tool Risk Problem

Scenario: A compiler has a bug that generates incorrect machine code.

// Source Code (Intent)
if (temperature > 100) {
    activateCooling();
}

// Compiler Bug: Generates incorrect condition
// Assembly (Actual):
if (temperature > 200) {  // BUG: Wrong threshold!
    activateCooling();
}

Impact:

• Temperature reaches 150°C (dangerous)
• Cooling NOT activated (compiler bug)
• Overheating causes failure → Safety hazard

Problem: Testing won't catch this because tests verify source code behavior, not compiled code.

Solution: Qualify the compiler to ensure it reliably translates source to machine code.

Tool Qualification Standards Comparison

Key Insight: DO-330 is MOST stringent (all tools must be qualified). ISO 26262 is more pragmatic (only high-risk tools).

ISO 26262 Tool Confidence Level (TCL)

TCL Classification Matrix

Step 1: Tool Impact (TI)

Step 2: Tool Error Detection (TD)

Step 3: TCL Determination

TCL Matrix (ISO 26262-8 Table 4):

TCL Decision:

• TCL 1: No qualification required
• TCL 2: Increased confidence from use OR qualification required
• TCL 3: Qualification required

Tool Classification Examples (ISO 26262)

Tool Qualification Process (ISO 26262-8:11)

Qualification Methods

ISO 26262-8:11.4.6: Four methods to achieve tool qualification:

Method 1a: Increased Confidence from Use

When Applicable: Tool has extensive usage history in similar safety applications.

Evidence Required:

• Tool usage statistics (e.g., GCC used in millions of projects)
• Known defect history (public bug database)
• Test suite coverage (GCC has 10,000+ regression tests)

Example: GCC Compiler (ASIL-B)

# Tool Qualification Argument - GCC v11.3.0 (ASIL-B Project)

## Tool Information
- **Tool**: GCC (GNU Compiler Collection) v11.3.0 ARM Embedded
- **Purpose**: C compiler for ARM Cortex-M4 (STM32F407)
- **Classification**: TI2, TD1 → TCL 1
- **Justification for TD1**: Extensive testing catches compiler errors

## Evidence for "Increased Confidence from Use"

### 1. Usage History
- GCC is the most widely used C compiler globally (>90% of embedded Linux)
- Used in automotive ASIL-D projects (e.g., brake systems by Tier-1 suppliers)
- Over 30 years of continuous development and refinement

### 2. Test Suite Coverage
- GCC includes 10,000+ regression tests (gcc/testsuite/)
- Test coverage: 95% statement coverage of compiler itself
- Continuous integration: Tests run on every commit (gitlab.com/gcc/gcc)

### 3. Defect History
- Known defects tracked publicly (gcc.gnu.org/bugzilla)
- Critical defects (code generation errors) are rare (<5 per year)
- ARM Cortex-M4 backend is mature (minimal changes since v8.0)

### 4. Project-Specific Validation
- Compiled our software and ran 500+ unit tests → All passed
- Back-to-back testing (GCC vs IAR) on sample functions → Identical results
- Code review of generated assembly for safety-critical functions → Correct

## Conclusion
**TCL 1 achieved** via Method 1a (Increased confidence from use).
No additional qualification required for ASIL-B.

**Approved by**: [Safety Manager], [Date]

Method 1c: Validation of Tool

When Applicable: In-house tools, or commercial tools without qualification data.

Validation Strategy:

1. Test the tool with known inputs/outputs
2. Inject faults to verify tool detects them
3. Cross-check with independent oracle

Example: Coverage Tool Validation (ASIL-C)

# Tool Validation Plan - Gcov Coverage Tool (ASIL-C)

## Tool Information
- **Tool**: Gcov v11.3.0 (part of GCC)
- **Purpose**: Measure MC/DC coverage for unit tests
- **Classification**: TI3, TD3 → TCL 3 (Qualification Required)

## Validation Approach: Method 1c (Validation of Tool)

### Validation Test Cases

#### TC-VAL-001: Statement Coverage - Positive Test

**Input**: C function with 10 statements
```c
void testFunction() {
    int x = 1;  // Line 1
    int y = 2;  // Line 2
    int z = x + y;  // Line 3
    if (z > 0) {  // Line 4
        printf("Positive\n");  // Line 5
    }
}

Expected Output: Gcov reports 5/5 statements covered (100%)

Actual Output: Gcov reports 100% statement coverage [PASS]

Result: PASS

TC-VAL-002: Branch Coverage - Negative Test (Detect Miss)

Input: Function with untested branch

void branchTest(int x) {
    if (x > 0) {
        // Branch A (tested)
        doSomething();
    } else {
        // Branch B (NOT tested - intentional)
        doSomethingElse();
    }
}

// Test: Only call branchTest(5) - never tests else branch

Expected Output: Gcov reports 50% branch coverage (1/2 branches)

Actual Output: Gcov reports 50% branch coverage [PASS]

Result: PASS (Tool correctly detects missing branch)

TC-VAL-003: MC/DC Coverage - Complex Decision

Input: Decision with 3 conditions

bool complexDecision(bool a, bool b, bool c) {
    return (a && b) || c;
}

// MC/DC Test Cases (4 required for full MC/DC):
// TC1: a=T, b=T, c=F → T
// TC2: a=F, b=T, c=F → F  (toggle a)
// TC3: a=T, b=F, c=F → F  (toggle b)
// TC4: a=T, b=T, c=T → T  (toggle c)

Expected Output: Gcov reports 100% MC/DC (all conditions independently affect decision)

Actual Output: Gcov (with --branch-probabilities) reports 100% MC/DC [PASS]

Result: PASS

Validation Summary

Total: 5/5 tests passed

Conclusion

Gcov has been validated for MC/DC coverage measurement for ASIL-C application.

Qualification Status: [PASS] QUALIFIED (Method 1c: Validation of Tool)

Approved by: [Tool Validation Engineer], [Safety Manager], [Date]

---

## DO-330 Tool Qualification (Aviation)

### DO-330 Tool Qualification Levels (TQL)

TQL is determined by two factors: the software level (DAL) of the software the tool supports, and the tool's criteria (development or verification). Development tools (Criteria 1) whose output is not independently verified receive the highest TQL numbers; verification tools (Criteria 2 and 3) receive lower TQL numbers.

| TQL | DAL | Tool Criteria | Description | Example | Rigor |
|-----|-----|---------------|-------------|---------|-------|
| **TQL-1** | A | 1 (Development) | Output directly becomes airborne software at DAL A without verification | Code generator for DAL A without back-to-back testing | Highest |
| **TQL-2** | B | 1 (Development) | Output directly becomes airborne software at DAL B without verification | Code generator for DAL B | High |
| **TQL-3** | C | 1 (Development) | Output directly becomes airborne software at DAL C without verification | Code generator for DAL C | Medium |
| **TQL-4** | A/B | 2 or 3 (Verification) | Verification tool used at DAL A/B that may fail to detect errors or automates verification | Coverage analyzer for DAL A/B | Medium |
| **TQL-5** | C/D (or A/B/C/D for Criteria 3) | 2 or 3 (Verification) | Verification tool used at DAL C/D, or Criteria 3 verification tool at any level | Coverage analyzer for DAL C; formal proof tool | Low |

**Key Principle**: ALL TQL levels require qualification under DO-330 — there is no "no qualification needed" category. ISO 26262 TCL 1 tools would typically still require operational requirements definition and Tool Accomplishment Summary under DO-330.

**Key Difference from ISO 26262**: DO-330 requires qualification for ALL TQL levels. ISO 26262 TCL 1 tools do NOT require formal qualification.

---

### DO-330 Qualification Process

**Required Artifacts** (DO-330 Section 5):

1. **Tool Operational Requirements (TOR)**: What the tool is supposed to do
2. **Tool Qualification Plan (TQP)**: How to qualify the tool
3. **Tool Accomplishment Summary (TAS)**: Evidence of qualification

**Example: Coverage Tool Qualification (DO-330 TQL-3)**

```markdown
# Tool Qualification Plan (TQP)
## LDRA TBvision v10.5.2 - MC/DC Coverage Analyzer

### 1. Tool Information

| Attribute | Value |
|-----------|-------|
| Tool Name | LDRA TBvision |
| Version | 10.5.2 |
| Vendor | LDRA Ltd |
| TQL | TQL-3 (Automates verification - structural coverage) |

### 2. Tool Operational Requirements (TOR)

**TOR-001**: Tool shall accurately measure statement coverage
- **Input**: Instrumented C source code + test execution trace
- **Output**: Statement coverage percentage (0-100%)
- **Accuracy**: ±0.1% (verified via comparison with manual counting)

**TOR-002**: Tool shall accurately measure MC/DC coverage
- **Input**: Instrumented C source with decisions + test cases
- **Output**: MC/DC coverage percentage + uncovered conditions
- **Accuracy**: 100% (no false positives, no false negatives)

**TOR-003**: Tool shall generate traceable coverage reports
- **Output**: HTML/PDF report with line-by-line coverage annotation
- **Traceability**: Each uncovered line identified with source file + line number

### 3. Verification of TOR

| TOR ID | Verification Method | Test Cases | Result |
|--------|---------------------|------------|--------|
| TOR-001 | Validation testing | 50 test programs with known coverage | [PASS] 100% match |
| TOR-002 | MC/DC validation | 20 complex decisions, manual MC/DC calculation vs tool | [PASS] 100% match |
| TOR-003 | Report inspection | Manual review of 10 coverage reports | [PASS] All traceable |

### 4. Tool Configuration Management

**Baseline**: LDRA TBvision v10.5.2 (Build 10520)
**Configuration Index**: TQL3-LDRA-v10.5.2-CI001
**Change Control**: Any tool update requires re-qualification

### 5. Tool Qualification Completion Criteria

- [x] All TORs verified (100% pass rate)
- [x] Tool Accomplishment Summary (TAS) created
- [x] Independent review completed
- [x] Approval from Certification Authority (if required)

### 6. Tool Accomplishment Summary (TAS)

**Conclusion**: LDRA TBvision v10.5.2 is **QUALIFIED** for TQL-3 (MC/DC coverage analysis) per DO-330.

**Approved by**:
- Tool Qualification Engineer: [Name], [Date]
- IV&V Lead: [Name], [Date]
- DER (Designated Engineering Representative): [Name], [Date]

Vendor-Provided Tool Qualification

Pre-Qualified Tools

Many commercial tool vendors provide qualification kits:

Advantages:

• Time Savings: No need to qualify in-house (saves 6-12 months)
• Authority Acceptance: Pre-qualified tools widely accepted by certification bodies
• Maintenance: Vendor handles qualification for new versions

Usage:

1. Purchase tool + qualification package
2. Review qualification data (ensure it matches your use case)
3. Reference vendor qualification in your safety case

Example Reference:

# Tool Qualification - VectorCAST/C++ v24.0 (ASIL-D Project)

## Tool Information
- **Tool**: VectorCAST/C++ v24.0
- **Vendor**: Vector Software
- **Classification**: TCL 3 (Code coverage analysis)

## Qualification Approach
**Method**: Vendor-provided qualification package (Method 1b: Evaluation of tool development process)

## Vendor Qualification Data
- **Qualification Report**: VectorCAST_ISO26262_Qualification_v24.0.pdf
- **Tool Confidence**: Certified to TCL 3 per ISO 26262-8:2018
- **Certification Body**: TÜV SÜD (Certificate #ISO26262-TC3-2024-089)
- **Valid for ASIL**: ASIL-D (highest level)

## Project-Specific Validation
- Verified tool version matches qualified version (v24.0) [PASS]
- Ran vendor-provided validation suite on target hardware [PASS]
- Cross-checked coverage results on sample function (manual count vs tool) [PASS]

## Conclusion
**VectorCAST v24.0 is qualified** for MC/DC coverage measurement in ASIL-D application.

No additional in-house qualification required.

**Approved by**: [Safety Manager], [Date]

Open-Source Tool Qualification Challenge

The GCC Problem

Challenge: GCC is open-source and free, but NOT pre-qualified for safety standards.

Options:

1. Qualify GCC yourself (Method 1c: Validation of Tool)

   • Effort: 6-12 person-months
   • Cost: ~$100k-200k (internal effort)
   • Reusable across projects? Yes (if same GCC version)

2. Use pre-qualified GCC variant

   • Example: HighTec TriCore Compiler (GCC-based, ISO 26262 qualified)
   • Cost: ~$10k/license
   • Effort: Minimal (review vendor docs)

3. Use commercial alternative

   • Example: IAR EWARM (proprietary, pre-qualified)
   • Cost: ~$15k/license
   • Effort: Minimal

4. Demonstrate "Increased Confidence from Use" (Method 1a)

   • Applicable for: All ASIL levels in principle; most practical for ASIL-A/B where evidence requirements are proportionate
   • Effort: Low to medium (document GCC's proven track record, defect history, test suite)
   • Limitation for ASIL-C/D: The evidence burden is substantial; most ASIL-C/D projects use pre-qualified tools instead
   • Not applicable to: DO-178C DAL A (DO-330 requires full TQL-1 qualification lifecycle for development tools)

Recommendation for ASIL-B:

• Development: Use GCC (free) + demonstrate increased confidence
• Production: Switch to IAR EWARM (pre-qualified) for final release

Tool Qualification Decision Tree

Use this decision tree to determine whether a tool requires qualification, and if so, at what level. The tree evaluates tool impact on safety-critical outputs, error detection capability, and applicable safety standard requirements.

Summary

Tool Qualification for Safety-Critical Development:

• Why Qualify: Tools can introduce errors or fail to detect defects
• ISO 26262: TCL 1-3 based on Tool Impact + Error Detection
• DO-330: TQL 1-5, ALL levels require qualification (most stringent)
• Qualification Methods (ISO 26262-8:11.4.6):
  • Method 1a: Increased confidence from use (low effort)
  • Method 1b/1d: Vendor-provided qualification or development per safety standard (medium cost, low effort)
  • Method 1c: Validation testing (high effort)
• Open-Source Challenge: GCC requires qualification for ASIL-C/D
• Vendor-Qualified Tools: IAR, VectorCAST, LDRA (pre-qualified, $10k-25k)

Practical Advice:

• ASIL-A/B: Use GCC + demonstrate confidence via testing
• ASIL-C/D: Buy pre-qualified tools (IAR, VectorCAST) to save time
• DO-178C DAL A: Only use DO-330 qualified tools (mandatory)