2.4: Verification Agent Instructions
Role Definition
Verification Agent (SWE.4)
Primary Responsibility: Generate unit tests, execute tests, achieve code coverage, report results
ASPICE Process: SWE.4 Software Unit Verification
Success Metrics:
- Test Generation: 80-85% coverage from AI (human adds edge cases to reach 100%)
- Test Quality: 90-95% pass rate on first run
- Coverage: Achieve ≥100% statement coverage (target for ASIL-B/SIL 3)
Input Work Products
Required Inputs (Must Exist Before Starting)
1. Source Code (SWE.3 Output)
- Content: Implemented functions (.c files), header files (.h)
- Quality Criteria: Code compiles without errors, MISRA compliant
2. Software Requirements (SWE.1 Output)
- Content: Requirements with test criteria, acceptance criteria
- Purpose: Define test cases (input values, expected outputs)
3. Unit Test Framework
- C Projects: Google Test (C++), Unity Test Framework (C), CppUTest
- Build System: CMake, Make, or vendor-specific (Vector VectorCAST for ISO 26262)
Execution Steps
Step-by-Step Workflow
Step 1: Analyze Function Under Test
Action: Extract function signature, requirements, test criteria
Example Function (from SWE.3):
/**
* @brief Get current obstacle distance from radar sensor
* @implements [SWE-045-1] Obstacle Distance Calculation
* @param[out] distance_m Pointer to distance output (meters)
* @return 0 = success, -1 = invalid sensor data
*/
int ACC_GetObstacleDistance(float* distance_m);
Test Criteria (from requirement):
- Typical Values: Input = 5000 mm → Output = 5.0 m
- Boundary Values: Input = 0 mm → Output = 0.0 m, Input = 65535 mm → Output = 65.535 m
- Invalid Data: Input = 0xFFFF (invalid) → Return -1
- Null Pointer: Input pointer = NULL → Return -1
Step 2: Generate Unit Test Cases
Action: Write test cases covering typical values, boundaries, edge cases, error conditions
Test Case Types:
- Typical Values (happy path)
- Boundary Values (min, max, off-by-one)
- Invalid Inputs (null pointers, out-of-range)
- Error Conditions (sensor fault, CAN timeout)
Generated Unit Tests (Google Test):
/**
* @file test_acc_controller.cpp
* @brief Unit tests for ACC Controller
* @verified_by [SWE-045-1, SWE-045-2, SWE-045-3]
*/
#include <gtest/gtest.h>
extern "C" {
#include "acc_controller.h"
}
/* Mock CAN functions for testing */
static uint16_t g_mock_radar_data = 0;
static int g_mock_can_result = 0;
int Mock_CAN_ReadRadarDistance(uint16_t* data) {
*data = g_mock_radar_data;
return g_mock_can_result;
}
/* Test Fixture: Setup/teardown for ACC tests */
class ACC_ControllerTest : public ::testing::Test {
protected:
void SetUp() override {
/* Inject mock CAN function */
ACC_SetCANReadFunc(Mock_CAN_ReadRadarDistance);
g_mock_radar_data = 0;
g_mock_can_result = 0;
}
void TearDown() override {
/* Cleanup (if needed) */
}
};
/**
* @test TC-SWE-045-1-1: Typical value (5 meters)
* @verified_by [SWE-045-1]
*/
TEST_F(ACC_ControllerTest, GetObstacleDistance_TypicalValue_5m) {
float distance_m;
/* Setup: Radar reports 5000 mm (5 meters) */
g_mock_radar_data = 5000;
g_mock_can_result = 0; /* CAN read success */
/* Execute */
int result = ACC_GetObstacleDistance(&distance_m);
/* Verify */
ASSERT_EQ(result, 0); /* Success */
EXPECT_NEAR(distance_m, 5.0f, 0.01f); /* 5.0 m ± 0.01 */
}
/**
* @test TC-SWE-045-1-2: Boundary value (0 meters)
* @verified_by [SWE-045-1]
*/
TEST_F(ACC_ControllerTest, GetObstacleDistance_BoundaryValue_0m) {
float distance_m;
/* Setup: Radar reports 0 mm (0 meters, minimum valid value) */
g_mock_radar_data = 0;
g_mock_can_result = 0;
/* Execute */
int result = ACC_GetObstacleDistance(&distance_m);
/* Verify */
ASSERT_EQ(result, 0);
EXPECT_NEAR(distance_m, 0.0f, 0.01f);
}
/**
* @test TC-SWE-045-1-3: Boundary value (65.535 meters, max)
* @verified_by [SWE-045-1]
*/
TEST_F(ACC_ControllerTest, GetObstacleDistance_BoundaryValue_MaxRange) {
float distance_m;
/* Setup: Radar reports 65535 mm (max uint16_t) */
g_mock_radar_data = 65535;
g_mock_can_result = 0;
/* Execute */
int result = ACC_GetObstacleDistance(&distance_m);
/* Verify */
ASSERT_EQ(result, 0);
EXPECT_NEAR(distance_m, 65.535f, 0.01f);
}
/**
* @test TC-SWE-045-1-4: Invalid sensor data (0xFFFF)
* @verified_by [SWE-045-1]
*/
TEST_F(ACC_ControllerTest, GetObstacleDistance_InvalidSensorData) {
float distance_m;
/* Setup: Radar reports 0xFFFF (invalid data marker) */
g_mock_radar_data = 0xFFFF;
g_mock_can_result = 0;
/* Execute */
int result = ACC_GetObstacleDistance(&distance_m);
/* Verify: Function should return error */
ASSERT_EQ(result, -1);
}
/**
* @test TC-SWE-045-1-5: Null pointer input
* @verified_by [SWE-045-1]
*/
TEST_F(ACC_ControllerTest, GetObstacleDistance_NullPointer) {
/* Execute: Pass NULL pointer */
int result = ACC_GetObstacleDistance(NULL);
/* Verify: Function should reject null pointer */
ASSERT_EQ(result, -1);
}
/**
* @test TC-SWE-045-1-6: CAN read failure
* @verified_by [SWE-045-1]
*/
TEST_F(ACC_ControllerTest, GetObstacleDistance_CANReadFailure) {
float distance_m;
/* Setup: CAN driver returns error */
g_mock_radar_data = 5000;
g_mock_can_result = -1; /* CAN read failed */
/* Execute */
int result = ACC_GetObstacleDistance(&distance_m);
/* Verify: Function should propagate error */
ASSERT_EQ(result, -1);
}
/* Repeat for other functions: ACC_GetClosingSpeed, ACC_ControlLoop_10ms, etc. */
Quality Check:
- [OK] Test cases cover typical, boundary, invalid, error conditions
- [OK] Each test has Doxygen comment (@test, @verified_by)
- [OK] Traceability to requirements ([SWE-045-1])
- [OK] Assertions use appropriate tolerance (EXPECT_NEAR for floats)
Step 3: Execute Unit Tests
Action: Compile and run unit tests, capture results
Build and Run (CMake + Google Test):
# Build tests
mkdir build && cd build
cmake ..
make
# Run tests with XML output (for CI/CD)
./test_acc_controller --gtest_output=xml:test_results.xml
# Example output:
# [==========] Running 6 tests from 1 test suite.
# [----------] Global test environment set-up.
# [----------] 6 tests from ACC_ControllerTest
# [ RUN ] ACC_ControllerTest.GetObstacleDistance_TypicalValue_5m
# [ OK ] ACC_ControllerTest.GetObstacleDistance_TypicalValue_5m (0 ms)
# [ RUN ] ACC_ControllerTest.GetObstacleDistance_BoundaryValue_0m
# [ OK ] ACC_ControllerTest.GetObstacleDistance_BoundaryValue_0m (0 ms)
# [ RUN ] ACC_ControllerTest.GetObstacleDistance_BoundaryValue_MaxRange
# [ OK ] ACC_ControllerTest.GetObstacleDistance_BoundaryValue_MaxRange (0 ms)
# [ RUN ] ACC_ControllerTest.GetObstacleDistance_InvalidSensorData
# [ OK ] ACC_ControllerTest.GetObstacleDistance_InvalidSensorData (0 ms)
# [ RUN ] ACC_ControllerTest.GetObstacleDistance_NullPointer
# [ OK ] ACC_ControllerTest.GetObstacleDistance_NullPointer (0 ms)
# [ RUN ] ACC_ControllerTest.GetObstacleDistance_CANReadFailure
# [ OK ] ACC_ControllerTest.GetObstacleDistance_CANReadFailure (0 ms)
# [----------] 6 tests from ACC_ControllerTest (0 ms total)
# [==========] 6 tests from 1 test suite ran. (0 ms total)
# [ PASSED ] 6 tests.
Escalation: If any tests fail, escalate to the Implementation Agent or a human engineer
[ESCALATION] ESCALATION: Unit Test Failure
Test: TC-SWE-045-1-1 (GetObstacleDistance_TypicalValue_5m)
Expected: distance_m = 5.0 ± 0.01
Actual: distance_m = 5.12
Root Cause: Possible issue in conversion formula
Assignee: @implementation_agent (re-check code) or @senior_engineer
Step 4: Measure Code Coverage
Action: Use coverage tool (gcov, lcov) to measure statement/branch coverage
Coverage Analysis (gcov + lcov):
# Compile with coverage flags
gcc -fprofile-arcs -ftest-coverage -o test_acc_controller test_acc_controller.cpp acc_controller.c
# Run tests
./test_acc_controller
# Generate coverage report
gcov acc_controller.c
lcov --capture --directory . --output-file coverage.info
genhtml coverage.info --output-directory coverage_html
# View report
firefox coverage_html/index.html
# Example coverage report:
# File: acc_controller.c
# Lines executed: 95.2% (120 of 126)
# Branches executed: 88.5% (23 of 26)
# Functions executed: 100% (5 of 5)
Coverage Gaps (lines not covered):
Line 145: Error handling for integer overflow (rare edge case)
Line 178: Diagnostic logging (disabled in unit test environment)
Line 203: Watchdog kick (hardware-dependent, not mockable)
AI Agent Action:
- If coverage < 80%, generate additional test cases
- If coverage 80-100%, document justification for uncovered lines
- If coverage = 100%, report success [PASS]
Coverage Report (Markdown):
## Code Coverage Report
| Metric | Coverage | Target | Status |
|--------|----------|--------|--------|
| **Statement Coverage** | 95.2% (120/126) | ≥100% (ASIL-B) | [WARN] Below target |
| **Branch Coverage** | 88.5% (23/26) | ≥100% (ASIL-B) | [WARN] Below target |
| **Function Coverage** | 100% (5/5) | 100% | [PASS] Met |
### Uncovered Lines (Justification)
| Line | Code | Justification |
|------|------|---------------|
| 145 | Integer overflow check | Unreachable (input range validated by CAN protocol) |
| 178 | Diagnostic logging | Hardware-dependent (requires real ECU, not unit-testable) |
| 203 | Watchdog kick | Hardware-dependent (tested in integration tests) |
### Recommendation
- Add integration tests for lines 178, 203 (hardware-dependent)
- Integer overflow check (line 145) can be excluded with safety justification (ISO 26262 allows if proven unreachable)
Step 5: Generate Test Report
Action: Create ASPICE-compliant test report (SWE.4 BP6)
Test Report Template:
# Software Unit Verification Report (SWE.4)
## Test Summary
- **Test Date**: 2025-12-17
- **Tested Component**: ACC_Controller
- **Test Framework**: Google Test 1.14.0
- **Test Environment**: Ubuntu 22.04, gcc 11.3
- **Tester**: AI Verification Agent (reviewed by @test_engineer)
## Test Results
| Test Case ID | Description | Requirement | Status | Notes |
|--------------|-------------|-------------|--------|-------|
| TC-SWE-045-1-1 | Typical value (5m) | SWE-045-1 | [PASS] | |
| TC-SWE-045-1-2 | Boundary (0m) | SWE-045-1 | [PASS] | |
| TC-SWE-045-1-3 | Boundary (max) | SWE-045-1 | [PASS] | |
| TC-SWE-045-1-4 | Invalid sensor | SWE-045-1 | [PASS] | |
| TC-SWE-045-1-5 | Null pointer | SWE-045-1 | [PASS] | |
| TC-SWE-045-1-6 | CAN failure | SWE-045-1 | [PASS] | |
**Total Tests**: 6
**Passed**: 6 (100%)
**Failed**: 0 (0%)
**Blocked**: 0 (0%)
## Coverage Results
- **Statement Coverage**: 95.2% (120/126 lines)
- **Branch Coverage**: 88.5% (23/26 branches)
- **Function Coverage**: 100% (5/5 functions)
**Target**: 100% statement/branch (ASIL-B requirement)
**Gap**: 4.8% statement, 11.5% branch (justification documented)
## Traceability
- **Requirements Verified**: SWE-045-1 (100% coverage)
- **Functions Tested**: ACC_GetObstacleDistance (100% coverage)
- **Test-to-Requirement Matrix**: See Appendix A
## Defects Found
None (all tests passed)
## Recommendations
1. Add integration tests for hardware-dependent code (lines 178, 203)
2. Achieve 100% coverage or justify exclusions (ISO 26262 safety argument)
## Approval
- **Test Engineer**: [TBD - Human Review Required]
- **Date**: [TBD]
Output Work Products
What Verification Agent Must Generate
1. Unit Test Code (.cpp files)
- Content: Test cases for all functions (typical, boundary, invalid, error)
- Format: Google Test (C++), Unity (C), or VectorCAST (ISO 26262 certified)
2. Test Execution Results (XML)
- Format: JUnit XML (for CI/CD integration)
- Content: Test names, pass/fail status, execution time
3. Coverage Report (HTML + Markdown)
- Content: Statement/branch/function coverage, gap analysis
- Format: lcov HTML report + Markdown summary
4. Test Report (Markdown/PDF)
- Content: ASPICE SWE.4 test report (see template above)
- Format: Markdown (convertible to PDF via Pandoc)
5. Traceability Matrix (Test → Requirement)
| Test Case | Requirement | Function | Status |
|-----------|-------------|----------|--------|
| TC-SWE-045-1-1 | SWE-045-1 | ACC_GetObstacleDistance | [PASS] |
| TC-SWE-045-1-2 | SWE-045-1 | ACC_GetObstacleDistance | [PASS] |
| ... | ... | ... | ... |
6. Pull Request Summary
## Summary
- Generated 24 unit test cases for ACC_Controller component
- Test coverage: 95.2% statement, 88.5% branch, 100% function
- All 24 tests passed [PASS] (0 failures)
- Verified requirements: [SWE-045-1, SWE-045-2, SWE-045-3]
## AI Confidence
- High confidence: Typical/boundary test cases (90% coverage)
- Medium confidence: Edge cases (human should add safety-specific tests)
## Test Quality Metrics
- Test Execution Time: 0.8 seconds (fast feedback)
- Mock Coverage: 100% (all CAN dependencies mocked)
- Assertion Quality: 100% (all tests have expected values)
## Coverage Gaps (4.8%)
- Lines 145, 178, 203 not covered (hardware-dependent)
- Justification: Tested in integration tests (HIL test bench)
## Human Action Required
1. Review coverage gaps (approve justification or add tests)
2. Approve test report (SWE.4 compliance)
3. Sign off on test execution results
## Traceability
- Requirements verified: 100% (3/3 requirements have tests)
- Test-to-requirement matrix: See test_traceability_matrix.xlsx
Quality Criteria
Acceptance Criteria for Verification Agent Output
Verification Agent Quality Checklist:
──────────────────────────────────────────────────────
☐ Test Coverage
☐ Typical values covered (happy path)
☐ Boundary values covered (min, max, off-by-one)
☐ Invalid inputs covered (null, out-of-range)
☐ Error conditions covered (failures, timeouts)
☐ Test Quality
☐ All tests pass (100% pass rate)
☐ Tests have clear names (descriptive)
☐ Assertions use appropriate tolerance (EXPECT_NEAR for floats)
☐ Traceability tags present (@verified_by)
☐ Code Coverage
☐ Statement coverage ≥ 80% (AI-generated, target 100%)
☐ Branch coverage ≥ 80%
☐ Function coverage = 100%
☐ Coverage gaps justified (hardware-dependent, unreachable)
☐ Traceability
☐ 100% requirements have test cases
☐ Test-to-requirement matrix generated
☐ Reporting
☐ Test report complete (SWE.4 format)
☐ Coverage report generated (HTML + Markdown)
☐ Test results exported (JUnit XML for CI/CD)
Verdict:
[PASS]: Submit test suite for human review
[FAIL]: Fix issues, re-run tests
Escalation Triggers
When Verification Agent Must Escalate
1. Test Failure (Cannot Debug) [ESCALATION]
- Trigger: Unit test fails, AI cannot determine root cause after 3 attempts
- Action: Escalate to Implementation Agent or senior engineer
- Example:
[ESCALATION] ESCALATION: Unit Test Failure Test: TC-SWE-045-1-1 (GetObstacleDistance_TypicalValue_5m) Expected: 5.0 ± 0.01 Actual: 5.12 (off by 2.4%) AI Attempted Fixes: 1. Checked conversion formula (correct: mm/1000) 2. Verified mock data (correct: 5000 mm) 3. Inspected function logic (no obvious error) Recommendation: Human debugging required (potential floating-point precision issue?) Assignee: @senior_engineer
2. Coverage Gap Cannot Be Resolved [ESCALATION]
- Trigger: Coverage < 80%, AI cannot generate additional tests
- Action: Escalate to test engineer
- Example:
[ESCALATION] ESCALATION: Coverage Gap Current Coverage: 72% statement (target: 80%) Uncovered Code: Lines 145-162 (hardware watchdog, no mock available) Issue: Cannot mock hardware watchdog in unit test environment Recommendation: Test in integration tests (HIL) or justify exclusion Assignee: @test_engineer
3. Requirement Not Testable [ESCALATION]
- Trigger: Requirement lacks measurable criteria (cannot write test)
- Action: Escalate to requirements engineer
- Example:
[ESCALATION] ESCALATION: Requirement Not Testable Requirement: [SWE-089] "System shall be robust" Issue: "Robust" not quantified (no pass/fail criteria) Question: Define testable criteria (e.g., MTBF >10,000 hours, fault recovery <1 second) Assignee: @requirements_lead
4. Safety-Critical Test Missing [ESCALATION]
- Trigger: ASIL-B function has no tests for safety-critical behavior
- Action: Escalate to safety engineer
- Example:
[ESCALATION] ESCALATION: Safety Test Missing Function: ACC_ControlLoop_10ms (ASIL-B) Issue: No test case for sensor fusion failure (both radar and camera invalid) ISO 26262 requires: Testing of all safety-critical failure modes Recommendation: Add test case for dual-sensor failure (expected: safe state transition) Assignee: @safety_engineer
Examples
Complete Verification Agent Workflow
Input: Source code (ACC_Controller, 285 LOC)
Output 1: Unit Tests
- 24 test cases generated (6 per function × 4 functions)
- Test types: Typical (6), boundary (8), invalid (6), error (4)
Output 2: Test Execution Results
- All 24 tests passed [PASS] (0 failures)
- Execution time: 0.8 seconds
Output 3: Coverage Report
- Statement: 95.2% (120/126)
- Branch: 88.5% (23/26)
- Function: 100% (5/5)
- Gaps justified (hardware-dependent code)
Output 4: Test Report
- ASPICE SWE.4 compliant report (Markdown + PDF)
- Traceability matrix: 100% requirements verified
Output 5: Pull Request
- Branch:
feature/swe4-unit-tests - Files:
test_acc_controller.cpp,test_report.md,coverage_report.html - Assignee: @test_engineer (human review)
- Status: Awaiting approval
Human Review Time: 1 hour (baseline: 4 hours manual) → 75% time savings
Summary
Verification Agent Key Responsibilities:
- Generate Unit Tests: Typical, boundary, invalid, error cases (80-85% coverage)
- Execute Tests: Run tests, capture results (90-95% pass rate)
- Measure Coverage: gcov/lcov for statement/branch coverage (target ≥100% for ASIL-B)
- Generate Test Report: ASPICE SWE.4 compliant report with traceability
- Justify Coverage Gaps: Document why certain lines are not covered (hardware-dependent)
Escalation: Test failures, coverage gaps, untestable requirements, missing safety tests
Success Metrics: 80-85% AI-generated coverage, 90-95% test pass rate, 100% requirement verification