3.2: Testing Workflow Instructions

Testing Workflow for AI Agents

Test Levels in ASPICE

The following diagram maps ASPICE test levels (SWE.4 unit, SWE.5 integration, SWE.6 qualification) to the AI agent's scope of automated test generation and execution.

Testing Workflow

AI Agent Scope: Unit tests (SWE.4) - automated, fast, per-commit

CI/CD Pipeline Integration

GitLab CI Example

Pipeline Configuration (.gitlab-ci.yml):

# .gitlab-ci.yml
stages:
  - build
  - unit_test
  - static_analysis
  - integration_test
  - deploy

# Stage 1: Build
build_firmware:
  stage: build
  image: gcc:11.3
  script:
    - make clean
    - make all CFLAGS="-Wall -Werror"
  artifacts:
    paths:
      - build/firmware.elf
      - build/*.o
    expire_in: 1 hour

# Stage 2: Unit Tests (AI-generated tests)
unit_tests:
  stage: unit_test
  image: gcc:11.3
  dependencies:
    - build_firmware
  script:
    # Compile unit tests
    - make test
    # Run tests with coverage
    - ./build/test_runner --gtest_output=xml:test_results.xml
    # Generate coverage report
    - gcov src/*.c
    - lcov --capture --directory . --output-file coverage.info
    - genhtml coverage.info --output-directory coverage_html
  coverage: '/Lines executed: (\d+\.\d+)%/'
  artifacts:
    reports:
      junit: test_results.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
    paths:
      - coverage_html/
      - test_results.xml
    when: always

# Stage 3: Static Analysis
misra_check:
  stage: static_analysis
  image: cppcheck:latest
  script:
    - cppcheck --addon=misra.json --xml --enable=all src/*.c 2> misra_report.xml
    - python3 scripts/parse_misra.py misra_report.xml
  artifacts:
    paths:
      - misra_report.xml
      - misra_summary.md
  allow_failure: false  # Block PR if MISRA Required violations

complexity_analysis:
  stage: static_analysis
  image: python:3.9
  script:
    - pip install radon
    - radon cc src/*.c --min B --json > complexity.json
    - python3 scripts/complexity_report.py complexity.json
  artifacts:
    paths:
      - complexity.json
      - complexity_report.md

# Stage 4: Integration Tests (if applicable)
integration_tests:
  stage: integration_test
  image: dspace-hil:latest
  script:
    - python3 tests/hil/run_integration_tests.py --xml-output=hil_results.xml
  artifacts:
    reports:
      junit: hil_results.xml
  only:
    - main  # Only run on main branch (time-consuming)

# Stage 5: Deploy (if tests pass)
deploy_to_dev:
  stage: deploy
  script:
    - scp build/firmware.elf devserver:/firmware/latest/
  only:
    - main

Unit Test Execution

Local Testing (Before Commit)

AI Agent Pre-Commit Workflow:

# Step 1: Build code
make clean && make all

# Step 2: Run unit tests
make test

# Expected output:
# [==========] Running 24 tests from 5 test suites.
# [----------] Global test environment set-up.
# [----------] 6 tests from ACC_ControllerTest
# [ RUN      ] ACC_ControllerTest.GetObstacleDistance_TypicalValue_5m
# [       OK ] ACC_ControllerTest.GetObstacleDistance_TypicalValue_5m (0 ms)
# ...
# [----------] 6 tests from ACC_ControllerTest (2 ms total)
# [==========] 24 tests from 5 test suites ran. (10 ms total)
# [  PASSED  ] 24 tests.

# Step 3: Check coverage
gcov src/*.c
# Output:
# File 'src/acc_controller.c'
# Lines executed:95.24% of 126
# Branches executed:88.46% of 26

# Step 4: If all pass, commit
if [ $? -eq 0 ]; then
    git add src/ tests/
    git commit -m "feat(acc): Implement distance calculation [ASPICE-1234]"
else
    echo "[FAIL] Tests failed, fix before committing"
fi

CI/CD Testing (After Push)

Automated Testing on Push:

# After pushing to remote:
git push origin feature/ASPICE-1234-acc-controller

# GitLab CI automatically triggers pipeline:
# ┌────────────────────────────────────────────────────┐
# │ Pipeline #142 (feature/ASPICE-1234-acc-controller) │
# ├────────────────────────────────────────────────────┤
# │ [OK] build_firmware (35s)                          │
# │ [OK] unit_tests (12s) - Coverage: 95.2%            │
# │ [OK] misra_check (8s) - 0 Required violations      │
# │ [OK] complexity_analysis (5s)                      │
# │ [--] integration_tests (skipped, not main branch)  │
# │ [--] deploy_to_dev (skipped, not main branch)      │
# └────────────────────────────────────────────────────┘
# Overall: PASSED (60s total)

# View pipeline: https://gitlab.com/company/acc-ecu/-/pipelines/142

If Pipeline Fails:

# Pipeline #142 (feature/ASPICE-1234-acc-controller)
# FAILED
#
# Stage: unit_tests
# Job: unit_tests
# Error:
# [ RUN      ] ACC_ControllerTest.GetObstacleDistance_TypicalValue_5m
# Expected: 5.0
# Actual: 5.12
# [  FAILED  ] ACC_ControllerTest.GetObstacleDistance_TypicalValue_5m (0 ms)
#
# 1 test failed out of 24

# AI Agent Action:
# 1. Analyze test failure log
# 2. Fix code or test
# 3. Commit fix, push to same branch
# 4. Pipeline re-runs automatically

Test Artifacts

What Gets Generated

1. Test Results (JUnit XML):

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="24" failures="0" errors="0" time="0.010">
  <testsuite name="ACC_ControllerTest" tests="6" failures="0" errors="0" time="0.002">
    <testcase name="GetObstacleDistance_TypicalValue_5m" time="0.000" classname="ACC_ControllerTest"/>
    <testcase name="GetObstacleDistance_BoundaryValue_0m" time="0.000" classname="ACC_ControllerTest"/>
    <testcase name="GetObstacleDistance_BoundaryValue_MaxRange" time="0.000" classname="ACC_ControllerTest"/>
    <testcase name="GetObstacleDistance_InvalidSensorData" time="0.000" classname="ACC_ControllerTest"/>
    <testcase name="GetObstacleDistance_NullPointer" time="0.000" classname="ACC_ControllerTest"/>
    <testcase name="GetObstacleDistance_CANReadFailure" time="0.000" classname="ACC_ControllerTest"/>
  </testsuite>
  <!-- More test suites... -->
</testsuites>

Usage: GitLab/GitHub parses XML, displays test results in UI

2. Coverage Report (HTML):

coverage_html/
├── index.html         # Overview (95.2% coverage)
├── acc_controller.c.html  # Line-by-line coverage (green/red highlighting)
├── can_driver.c.html
└── sensor_fusion.c.html

View Coverage:

Local: firefox coverage_html/index.html
CI/CD: Artifact link in pipeline (e.g., https://gitlab.com/.../jobs/142/artifacts/browse/coverage_html/)

3. MISRA Report (Markdown):

# MISRA C:2012 Compliance Report

**Date**: 2025-12-17
**Analyzed Files**: 5 files (1,200 LOC)

## Summary
| Category | Violations | Status |
|----------|------------|--------|
| **Required Rules** | 0 | PASS |
| **Advisory Rules** | 2 | WARNING |

## Required Violations (BLOCKING)
None (compliant)

## Advisory Violations (NON-BLOCKING)
1. **Rule 2.1** (Line 89, diagnostics.c): Unreachable code
   - Justification: Dead code after 'return', cleanup required
2. **Rule 8.13** (Line 156, can_driver.c): Pointer parameter should be const
   - Justification: Parameter is read-only, add const qualifier

## Recommendation
Fix Advisory violations in follow-up PR (non-blocking for merge)

Test Failure Handling

AI Agent Workflow for Test Failures

Step 1: Analyze Failure

# Read test failure log
cat test_results.xml | grep "failure"

# Example:
# <testcase name="GetObstacleDistance_TypicalValue_5m" time="0.000">
#   <failure message="Expected: 5.0, Actual: 5.12">
#     Expected: 5.0
#     Actual: 5.12
#     Test failed at tests/test_acc_controller.cpp:45
#   </failure>
# </testcase>

Step 2: Determine Root Cause

def analyze_test_failure(test_name, expected, actual):
    """
    AI agent analyzes test failure, suggests fix
    """
    difference = abs(expected - actual)
    difference_pct = (difference / expected) * 100

    if difference_pct < 1.0:
        # Small difference (<1%), likely floating-point precision issue
        return "Floating-point precision issue. Fix: Use EXPECT_NEAR with tolerance 0.1"
    elif difference_pct < 10.0:
        # Moderate difference (1-10%), likely algorithm issue
        return "Algorithm error. Check conversion formula (mm → m)"
    else:
        # Large difference (>10%), likely logic error
        return "Critical logic error. Review function implementation"

# Example
root_cause = analyze_test_failure("GetObstacleDistance_TypicalValue_5m", expected=5.0, actual=5.12)
# Output: "Algorithm error. Check conversion formula (mm → m)"

Step 3: Fix and Re-Test

# Fix code (e.g., correct formula in src/acc_controller.c)
# Original (incorrect): distance_m = radar_mm / 900.0;  // Wrong!
# Fixed: distance_m = radar_mm / 1000.0;  // Correct

# Re-run tests locally
make test

# If pass:
git add src/acc_controller.c
git commit -m "fix(acc): Correct mm-to-m conversion formula [ASPICE-1234]

- Fix: radar_mm / 1000.0 (was / 900.0)
- Resolves test failure: TC-SWE-045-1-1"

git push origin feature/ASPICE-1234-acc-controller

Step 4: Escalate if Cannot Fix

ESCALATION: Unit Test Failure (Cannot Debug)

Test: TC-SWE-045-1-1 (GetObstacleDistance_TypicalValue_5m)
Expected: 5.0 ± 0.01
Actual: 5.12 (off by 2.4%)

AI Analysis:
- Conversion formula verified: radar_mm / 1000.0 (correct)
- Mock data verified: 5000 mm (correct)
- Function logic reviewed: No obvious error

Attempts:
1. Checked formula (correct)
2. Verified test data (correct)
3. Inspected function (no issues found)

Recommendation: Human debugging required (possible floating-point issue, hardware-specific behavior)

Assignee: @senior_engineer

Test Coverage Goals

ASPICE/ISO 26262 Requirements

Safety Class	Statement Coverage	Branch Coverage	MC/DC Coverage
QM (Non-safety)	≥80%	≥70%	Not required
ASIL-A	≥90%	≥80%	Not required
ASIL-B	≥100%*	≥100%*	Recommended
ASIL-C/D	100%	100%	Required

*Justification required for uncovered lines (hardware-dependent, proven unreachable)

AI Agent Target: Generate tests achieving at least 80% coverage (human adds edge cases to reach 100%)

Summary

Testing Workflow Key Actions:

Local Testing: Run tests before commit (make test, gcov)
CI/CD Pipeline: Automated testing on push (build, test, MISRA, coverage)
Test Artifacts: JUnit XML, coverage HTML, MISRA report
Failure Handling: Analyze, fix, re-test, escalate if needed
Coverage Goals: At least 80% (AI), 100% (human completes)

Pipeline Stages: Build → Unit Test → Static Analysis → (Integration Test) → Deploy

Quality Gate: Unit tests must pass before human review

Next: Release Workflow (31.03) - Versioning, tagging, release notes