1.2: Debugging and Profiling Tools for Embedded Systems

Introduction

Debugging and profiling are critical phases in embedded systems development where developers identify, diagnose, and resolve defects while optimizing performance within strict resource constraints. Unlike desktop applications, embedded systems present unique debugging challenges: limited memory, real-time constraints, hardware dependencies, and often inaccessible deployment environments (automotive ECUs, medical implants, aerospace systems).

In this chapter, you'll discover comprehensive guidance on debugging and profiling tools integrated into ASPICE-compliant workflows, with emphasis on:

• Remote debugging for embedded targets
• Memory profiling and leak detection
• Performance profiling under real-time constraints
• Thermal and power profiling for embedded hardware
• Integration with CI/CD pipelines
• Tool qualification for safety-critical systems

ASPICE Context: Debugging and profiling tools support multiple ASPICE processes:

• SWE.3: Detailed Design and Unit Construction (debugging during development)
• SWE.4: Software Unit Verification (defect identification and root cause analysis)
• SWE.5: Software Integration Test (integration-level debugging)
• SUP.9: Problem Resolution Management (systematic defect tracking and resolution)

GDB: The GNU Debugger

Overview

GDB (GNU Debugger) is the industry-standard debugger for C, C++, and embedded systems. It supports:

• Source-level debugging with breakpoints and watchpoints
• Remote debugging via GDB server (gdbserver) or JTAG/SWD interfaces
• Multi-threaded and multi-core debugging
• Scripting and automation via Python API
• Integration with IDEs (VS Code, Eclipse, CLion)

Basic GDB Workflow

Local Debugging Example (unit test on development workstation):

# Compile with debug symbols (-g flag)
gcc -g -o control_system control_system.c pid_controller.c

# Start GDB
gdb ./control_system

# GDB commands
(gdb) break pid_controller.c:42        # Set breakpoint at line 42
(gdb) run                               # Execute program
(gdb) print setpoint                    # Inspect variable
(gdb) backtrace                         # Show call stack
(gdb) step                              # Step into function
(gdb) continue                          # Resume execution
(gdb) quit                              # Exit GDB

ASPICE Work Product: 17-08 (Software unit verification results) includes debugger traces for failed test cases.

Remote Debugging for Embedded Targets

Embedded systems run on target hardware (microcontroller, SoC) that differs from the development workstation. Remote debugging connects GDB on the host to the target via:

1. GDB Server: Software agent running on target (requires target OS support)
2. JTAG/SWD Interface: Hardware debug probe (e.g., Segger J-Link, ST-Link)

Architecture: The following diagram shows the remote debugging setup, illustrating how GDB on the host workstation communicates with the target microcontroller through a JTAG/SWD debug probe, enabling breakpoints, memory inspection, and register-level debugging on the embedded target.

Example: Remote Debugging via JTAG (ARM Cortex-M4)

# On development workstation

# 1. Start OpenOCD (Open On-Chip Debugger) for JTAG connection
openocd -f interface/stlink.cfg -f target/stm32f4x.cfg

# OpenOCD output:
# Info : Listening on port 3333 for gdb connections

# 2. Connect GDB to OpenOCD
arm-none-eabi-gdb build/firmware.elf

# In GDB:
(gdb) target extended-remote localhost:3333
(gdb) monitor reset halt                # Reset target and halt CPU
(gdb) load                              # Flash firmware to target
(gdb) break main
(gdb) continue

# Target now running firmware, halted at main()

# 3. Debug as normal
(gdb) print sensor_reading
$1 = 42
(gdb) watch motor_speed                 # Watchpoint: break when variable changes
(gdb) continue

ASPICE Integration:

• SWE.5: Software Integration Test Plan (work product 17-11) specifies remote debugging setup for integration testing
• SUP.1: Quality Assurance (work product 08-52) includes debugger configuration in test environment specification

Advanced GDB Features

1. Conditional Breakpoints

Break only when specific conditions are met (critical for embedded systems with timing constraints):

# Break at sensor_read() only when sensor_id == 3
(gdb) break sensor_read if sensor_id == 3

# Break at control_loop() only when error exceeds threshold
(gdb) break control_loop if (setpoint - actual) > 10.0

Use Case: Automotive ECU with 100Hz control loop. Unconditional breakpoint would disrupt timing; conditional breakpoint triggers only on fault condition.

2. Watchpoints for Memory Corruption

Detect when a variable is modified (essential for debugging race conditions and memory corruption):

# Hardware watchpoint: break when global_state is written
(gdb) watch global_state

# Read watchpoint: break when critical_data is read
(gdb) rwatch critical_data

# Access watchpoint: break on any access
(gdb) awatch buffer[0]

Example Scenario: Debugging stack corruption in FreeRTOS task.

// Suspected stack overflow in high-priority task
uint8_t task_stack[512];

// In GDB:
(gdb) watch task_stack[511]     # Watch last byte of stack
(gdb) continue

# Watchpoint triggers when stack overflow occurs
Hardware watchpoint 2: task_stack[511]
Old value = 0xa5  (stack canary)
New value = 0x00  (corrupted!)

ASPICE Work Product: 19-08 (Problem resolution records) documents root cause as stack overflow, resolution as increasing stack size to 1024 bytes.

3. GDB Python Scripting

Automate complex debugging tasks with Python API:

# gdb_script.py: Automated test result collection
import gdb

class TestResultCollector(gdb.Command):
    def __init__(self):
        super(TestResultCollector, self).__init__("collect_results", gdb.COMMAND_USER)

    def invoke(self, arg, from_tty):
        # Run all test cases
        gdb.execute("break test_report_result")
        gdb.execute("run")

        results = []
        while True:
            try:
                # Collect test result at breakpoint
                test_name = gdb.parse_and_eval("test_name").string()
                test_passed = bool(gdb.parse_and_eval("passed"))
                results.append((test_name, test_passed))

                gdb.execute("continue")
            except gdb.error:
                break  # Program terminated

        # Generate report
        with open("test_results.txt", "w") as f:
            for name, passed in results:
                f.write(f"{name}: {'PASS' if passed else 'FAIL'}\n")

        print(f"Collected {len(results)} test results")

TestResultCollector()

Usage:

gdb -x gdb_script.py ./unit_tests
(gdb) collect_results

ASPICE Integration: Automated test result collection for SWE.4 unit verification.

GDB Integration with VS Code

Configuration (.vscode/launch.json):

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug on STM32 (OpenOCD)",
            "type": "cppdbg",
            "request": "launch",
            "program": "${workspaceFolder}/build/firmware.elf",
            "cwd": "${workspaceFolder}",
            "MIMode": "gdb",
            "miDebuggerPath": "arm-none-eabi-gdb",
            "miDebuggerServerAddress": "localhost:3333",
            "setupCommands": [
                {
                    "description": "Enable pretty-printing",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                },
                {
                    "description": "Reset target",
                    "text": "monitor reset halt"
                },
                {
                    "description": "Load firmware",
                    "text": "load"
                }
            ],
            "preLaunchTask": "build_firmware",
            "postDebugTask": "openocd_shutdown"
        }
    ]
}

Benefits:

• Visual breakpoint management
• Inline variable inspection
• Integrated terminal for GDB commands
• Call stack and thread visualization

ASPICE Traceability: VS Code debug configurations stored in version control satisfy SUP.8 (Configuration Management).

JTAG and SWD Debugging Protocols

Overview

JTAG (Joint Test Action Group) and SWD (Serial Wire Debug) are hardware interfaces for embedded debugging:

Typical Use: ARM Cortex-M microcontrollers (automotive ECUs, medical devices) use SWD; FPGAs and multi-chip systems use JTAG.

Debug Probe Hardware

Commercial debug probes:

ASPICE SUP.9 Consideration: For safety-critical systems (ASIL B+), debug probe must be qualified. J-Link and TRACE32 have qualification kits available.

Real-Time Trace

ARM CoreSight ETM (Embedded Trace Macrocell) enables non-intrusive program flow tracing:

[Target CPU] --ETM--> [Trace Buffer] --JTAG/SWD--> [Debug Probe] --> [PC: Trace Analyzer]

Use Case: Automotive ADAS system debugging

Problem: Intermittent camera frame processing failure (occurs 1 in 10,000 frames).

Solution: ETM trace captures every instruction executed during failure event.

Tool: Lauterbach TRACE32 with 4GB trace buffer

Result: Identified race condition in DMA interrupt handler (missed only under specific timing conditions).

ASPICE Work Product: 19-08 (Problem resolution record) includes trace analysis proving root cause.

Core Dump Analysis and Post-Mortem Debugging

Core Dump Generation

When an embedded system crashes, a core dump captures the system state for post-mortem analysis.

Example: FreeRTOS Hard Fault Handler with Core Dump

// hard_fault_handler.c
// ASPICE SWE.3: Detailed design for fault handling

#include "core_dump.h"

void HardFault_Handler(void) {
    // Capture core dump to flash memory
    core_dump_context_t dump;

    // Save CPU registers
    dump.r0 = __get_PSP();  // Process Stack Pointer
    dump.r1 = __get_MSP();  // Main Stack Pointer
    dump.pc = SCB->CFSR;    // Fault address
    dump.lr = SCB->HFSR;    // Fault status

    // Save task context
    dump.current_task = xTaskGetCurrentTaskHandle();
    dump.task_stack = pxCurrentTCB->pxStack;

    // Write to flash
    flash_write(CORE_DUMP_ADDR, &dump, sizeof(dump));

    // Reset system
    NVIC_SystemReset();
}

Post-Mortem Analysis:

# Extract core dump from target flash
openocd -f interface/jlink.cfg -f target/stm32f4x.cfg \
    -c "init; dump_image coredump.bin 0x08080000 0x10000; exit"

# Load in GDB
arm-none-eabi-gdb firmware.elf

(gdb) target core coredump.bin
(gdb) backtrace
#0  0x08002a4c in sensor_read ()
#1  0x08003120 in control_loop ()
#2  0x080045f8 in vTaskCode ()
#3  0x0800f234 in prvTaskExitError ()

(gdb) print sensor_handle
$1 = (sensor_t *) 0x0   # NULL pointer dereference!

ASPICE Integration:

• SUP.9: Problem resolution process triggered by core dump
• 19-08: Problem record documents NULL pointer root cause
• SWE.4: Regression test added to prevent recurrence

Automated Core Dump Collection in CI/CD

# .gitlab-ci.yml
# ASPICE SWE.5: Automated integration testing with crash analysis

integration_test:
  stage: test
  script:
    - ./flash_firmware.sh
    - ./run_integration_tests.sh
    - if [ -f /dev/target_flash ]; then
        dd if=/dev/target_flash of=coredump.bin skip=$CORE_DUMP_OFFSET count=$CORE_DUMP_SIZE;
        arm-none-eabi-gdb firmware.elf -batch -x analyze_coredump.gdb > crash_report.txt;
      fi
  artifacts:
    when: on_failure
    paths:
      - coredump.bin
      - crash_report.txt

Result: Every CI/CD test failure automatically generates debugger backtrace for developers.

Memory Profiling

Valgrind for Memory Error Detection

Valgrind is a dynamic analysis framework for detecting memory leaks, buffer overflows, and use-after-free errors.

Limitations for Embedded:

• Requires x86/x86-64/ARM Linux (not bare-metal microcontrollers)
• 10-50x slowdown (incompatible with real-time constraints)

Recommended Use: Unit tests on development workstation (ASPICE SWE.4).

Example: Detecting Memory Leak

// buggy_code.c
void process_sensor_data(void) {
    uint8_t* buffer = malloc(1024);  // Allocate buffer
    read_sensor(buffer);
    calculate_output(buffer);
    // BUG: forgot to free(buffer)!
}

Valgrind Analysis:

valgrind --leak-check=full --show-leak-kinds=all ./unit_tests

# Output:
==12345== LEAK SUMMARY:
==12345==    definitely lost: 1,024 bytes in 1 blocks
==12345==    indirectly lost: 0 bytes in 0 blocks
==12345==      possibly lost: 0 bytes in 0 blocks
==12345==
==12345== 1,024 bytes in 1 blocks are definitely lost in loss record 1 of 1
==12345==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12345==    by 0x108A2F: process_sensor_data (buggy_code.c:42)
==12345==    by 0x108B5E: test_sensor_processing (test_main.c:15)

Fix:

void process_sensor_data(void) {
    uint8_t* buffer = malloc(1024);
    read_sensor(buffer);
    calculate_output(buffer);
    free(buffer);  // FIXED
}

ASPICE Work Product: 17-08 (Unit verification results) includes Valgrind reports confirming no memory leaks.

AddressSanitizer (ASan) and MemorySanitizer (MSan)

ASan and MSan are LLVM/Clang compiler-based sanitizers with lower overhead than Valgrind (2-3x slowdown).

Compilation:

clang -fsanitize=address -g -o unit_tests unit_tests.c
./unit_tests

Detected Issues:

• Heap buffer overflow
• Stack buffer overflow
• Use-after-free
• Use-after-return
• Memory leaks

Example: Buffer Overflow Detection

// buffer_overflow.c
void copy_data(uint8_t* dest, uint8_t* src, size_t len) {
    for (size_t i = 0; i <= len; i++) {  // BUG: should be i < len
        dest[i] = src[i];
    }
}

int main() {
    uint8_t buffer[10];
    uint8_t data[15] = {0};
    copy_data(buffer, data, 15);  // Overflow!
    return 0;
}

ASan Output:

==23456==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffc1234560a
WRITE of size 1 at 0x7ffc1234560a thread T0
    #0 0x4f8a2c in copy_data buffer_overflow.c:3
    #1 0x4f8b1e in main buffer_overflow.c:9

Address 0x7ffc1234560a is located in stack of thread T0 at offset 42 in frame
    #0 0x4f8a7f in main buffer_overflow.c:7

  This frame has 1 object(s):
    [32, 42) 'buffer' <== Memory access at offset 42 overflows this variable

ASPICE Integration: ASan enabled in CI/CD for all unit tests (SWE.4).

Embedded-Specific Memory Profiling

For bare-metal embedded systems (no OS), custom memory profiling required:

Static Stack Analysis:

# GCC option to generate stack usage information
arm-none-eabi-gcc -fstack-usage -o firmware.elf main.c

# Generates main.su file:
main.c:42:compute_control_output    256    static
main.c:78:sensor_read               128    dynamic,bounded

Interpretation:

• compute_control_output() uses 256 bytes of stack (safe for analysis)
• sensor_read() uses dynamic stack (variable-length arrays or recursion - risky!)

ASPICE SWE.2: Architectural design specifies maximum stack depth per task.

Dynamic Heap Monitoring:

// heap_monitor.c: Custom heap wrapper for embedded systems
// ASPICE SWE.3: Instrumented memory allocator

#include <stdlib.h>

typedef struct {
    size_t total_allocated;
    size_t peak_usage;
    size_t current_usage;
} heap_stats_t;

static heap_stats_t heap_stats = {0};

void* monitored_malloc(size_t size) {
    void* ptr = malloc(size);
    if (ptr != NULL) {
        heap_stats.total_allocated += size;
        heap_stats.current_usage += size;
        if (heap_stats.current_usage > heap_stats.peak_usage) {
            heap_stats.peak_usage = heap_stats.current_usage;
        }
    }
    return ptr;
}

void monitored_free(void* ptr, size_t size) {
    free(ptr);
    heap_stats.current_usage -= size;
}

heap_stats_t get_heap_stats(void) {
    return heap_stats;
}

Integration Test:

// ASPICE SWE.5: Integration test with heap monitoring
void test_heap_usage(void) {
    reset_heap_stats();

    // Execute system for 1 hour of simulated operation
    run_system_for_duration(3600);

    heap_stats_t stats = get_heap_stats();

    // Verify peak heap usage within budget
    assert(stats.peak_usage < MAX_HEAP_SIZE);
    assert(stats.current_usage == 0);  // No leaks

    printf("Peak heap usage: %zu bytes\n", stats.peak_usage);
}

ASPICE Work Product: 17-50 (Integration test report) includes heap usage statistics.

Performance Profiling

Linux perf for Embedded Linux

perf is a Linux kernel performance profiling tool supporting:

• CPU cycle profiling
• Cache miss analysis
• Branch prediction analysis
• Flame graphs for visualization

Example: Profiling Automotive Gateway Application

# Record performance data for 10 seconds
perf record -F 99 -a -g -- sleep 10

# Generate report
perf report

# Output:
# Overhead  Command  Shared Object       Symbol
# ........  .......  ..................  .......................
    45.23%  gateway  libc-2.31.so        [.] memcpy
    23.15%  gateway  gateway             [.] can_message_parse
    12.87%  gateway  gateway             [.] encrypt_message
     8.45%  gateway  libcrypto.so.1.1    [.] AES_encrypt

Interpretation: 45% of CPU time spent in memcpy (candidate for optimization).

Optimization: Replace memcpy with DMA transfer for large CAN message buffers.

Verification:

perf record -F 99 -a -g -- ./optimized_gateway

# New results:
# Overhead  Command  Shared Object       Symbol
# ........  .......  ..................  .......................
    38.12%  gateway  gateway             [.] can_message_parse
    22.05%  gateway  gateway             [.] encrypt_message
    15.34%  gateway  libcrypto.so.1.1    [.] AES_encrypt
     8.21%  gateway  libc-2.31.so        [.] memcpy  # Reduced from 45%!

ASPICE Work Product: 17-50 (Integration test report) includes performance benchmarks before/after optimization.

Callgrind for Function-Level Profiling

Callgrind (part of Valgrind suite) provides function call graph with instruction-level profiling.

valgrind --tool=callgrind ./application

# Generates callgrind.out.<pid>

# Visualize with KCachegrind
kcachegrind callgrind.out.12345

Visualization: KCachegrind shows:

• Function call tree
• Percentage of total execution time per function
• Call relationships (caller/callee)

Use Case: Identify bottleneck in image processing pipeline (medical device).

Result: 70% of time spent in Gaussian blur kernel. Optimized with SIMD instructions (ARM NEON), reducing time to 25%.

ASPICE SWE.3: Optimization documented in detailed design rationale.

gprof for Static Profiling

gprof is a lightweight profiler for applications compiled with -pg flag.

gcc -pg -o application application.c
./application
gprof application gmon.out > profile.txt

Profile Output:

  %   cumulative   self              self     total
 time   seconds   seconds    calls  ms/call  ms/call  name
 42.00      1.26     1.26   100000     0.01     0.02  control_loop
 28.50      2.12     0.86    50000     0.02     0.03  pid_controller
 15.00      2.58     0.46    50000     0.01     0.01  sensor_filter

Interpretation: control_loop() consumes 42% of execution time.

ASPICE Context: Performance profiling for SWE.6 (Software Qualification Test) to verify real-time requirements.

Thermal and Power Profiling

Thermal Profiling for Embedded Systems

Challenge: Automotive ECUs operate in -40°C to +125°C environments. Thermal throttling can degrade real-time performance.

Tool: Thermal camera (FLIR) + temperature sensors + profiling software.

Example Setup:

// thermal_profiling.c
// ASPICE SWE.4: Performance verification under thermal stress

#include "temperature_sensor.h"

void thermal_stress_test(void) {
    // Run CPU-intensive task
    for (int temp = 25; temp <= 125; temp += 10) {
        set_thermal_chamber_temperature(temp);
        wait_for_thermal_stabilization();

        uint32_t start = get_timer_us();
        execute_control_algorithm();
        uint32_t end = get_timer_us();

        uint32_t cpu_temp = read_cpu_temperature();
        uint32_t execution_time = end - start;

        log_result(temp, cpu_temp, execution_time);
    }
}

Results:

Conclusion: System meets real-time requirements (<3ms) up to 85°C. Above 105°C, thermal throttling causes deadline violations.

ASPICE Work Product: 17-52 (Software qualification test report) includes thermal performance analysis.

Mitigation: Add heatsink or reduce CPU frequency at high temperatures.

Power Profiling

Tool: Joulescope (USB current/voltage monitor) for embedded systems.

Setup:

[Power Supply] --> [Joulescope] --> [Target ECU]
                        |
                        v
                  [PC: Joulescope UI]

Use Case: Battery-powered medical device (pacemaker prototype).

Profiling Workflow:

1. Instrument firmware with state markers:

// power_profiling.c
void set_power_state(const char* state) {
    // Toggle GPIO for Joulescope trigger
    gpio_toggle(PROFILING_PIN);
    log_power_state(state);
}

void main_loop(void) {
    while (1) {
        set_power_state("SENSING");
        read_heart_sensor();

        set_power_state("PROCESSING");
        analyze_heart_rhythm();

        set_power_state("ACTUATING");
        send_pacing_pulse();

        set_power_state("SLEEP");
        enter_low_power_mode(100);  // 100ms sleep
    }
}

2. Measure power consumption per state:

Insight: Pacing pulse (ACTUATING) dominates energy budget. Optimize pulse duration from 2ms to 1.5ms → 25% energy savings.

ASPICE Work Product: 17-52 (Qualification test report) includes power consumption analysis.

RTOS-Aware Debugging

FreeRTOS Thread Debugging

Challenge: Multi-threaded embedded systems require visibility into task states, stack usage, and inter-task communication.

Tool: GDB with FreeRTOS awareness plugin.

Setup:

# freertos_gdb.py: GDB plugin for FreeRTOS task inspection

import gdb

class FreeRTOSTaskList(gdb.Command):
    def __init__(self):
        super(FreeRTOSTaskList, self).__init__("freertos tasks", gdb.COMMAND_USER)

    def invoke(self, arg, from_tty):
        # Read FreeRTOS task list
        num_tasks = int(gdb.parse_and_eval("uxCurrentNumberOfTasks"))
        print(f"Total tasks: {num_tasks}\n")

        print(f"{'Task Name':<20} {'State':<12} {'Priority':<8} {'Stack Free (bytes)'}")
        print("-" * 70)

        # Iterate task list
        task_list = gdb.parse_and_eval("pxReadyTasksLists")
        for task_ptr in iterate_task_list(task_list):
            task_name = task_ptr['pcTaskName'].string()
            priority = int(task_ptr['uxPriority'])
            stack_high_water = int(task_ptr['pxStack'])
            state = get_task_state(task_ptr)

            print(f"{task_name:<20} {state:<12} {priority:<8} {stack_high_water}")

FreeRTOSTaskList()

Usage:

(gdb) source freertos_gdb.py
(gdb) freertos tasks

Total tasks: 5

Task Name            State        Priority Stack Free (bytes)
----------------------------------------------------------------------
ControlTask          Running      10       512
SensorTask           Blocked      8        1024
CommTask             Ready        6        768
IdleTask             Ready        0        256
LoggingTask          Suspended    4        2048

Debugging Deadlock:

(gdb) freertos tasks
# ControlTask: Blocked (waiting on mutex)
# SensorTask: Blocked (waiting on mutex)

(gdb) print mutex_owner
$1 = (TaskHandle_t) 0x200080a0 "ControlTask"

# Circular dependency: ControlTask owns mutex A, waiting for mutex B
#                      SensorTask owns mutex B, waiting for mutex A

ASPICE Work Product: 19-08 (Problem resolution record) documents deadlock root cause and fix (mutex ordering).

Integration with CI/CD Pipelines

Automated Debugging in GitLab CI

# .gitlab-ci.yml: ASPICE SWE.4/SWE.5 automated verification

unit_test:
  stage: test
  script:
    # Compile with debugging and sanitizers
    - cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_ASAN=ON ..
    - make

    # Run unit tests with Valgrind
    - valgrind --leak-check=full --xml=yes --xml-file=valgrind.xml ./unit_tests

    # Run with AddressSanitizer
    - ./unit_tests

  artifacts:
    reports:
      junit: test_results.xml
    paths:
      - valgrind.xml
      - asan_report.txt

integration_test:
  stage: test
  script:
    # Flash firmware to hardware-in-the-loop (HIL) rig
    - ./flash_firmware.sh firmware.elf

    # Run integration tests with remote debugging
    - timeout 600 ./run_hil_tests.sh || RESULT=$?

    # If tests fail, collect debug info
    - if [ $RESULT -ne 0 ]; then
        arm-none-eabi-gdb firmware.elf -batch -x collect_backtrace.gdb > crash_log.txt;
      fi

  artifacts:
    when: on_failure
    paths:
      - crash_log.txt
      - hil_test_log.txt

Result: Every test failure automatically generates debugger backtrace for root cause analysis.

ASPICE Work Product: 17-08, 17-50 (Verification results) include CI/CD-generated debug reports.

Tool Qualification for Safety-Critical Systems

ASPICE SUP.9: Tool Qualification

For safety-critical systems (ISO 26262 ASIL B+, DO-178C DAL A/B), debugging and profiling tools may require qualification.

Tool Classification:

Qualification Evidence for Valgrind (TCL-2):

1. Tool Operational Requirements (TOR):

   • Detect heap memory leaks with 100% accuracy
   • Detect stack buffer overflows
   • No false negatives for safety-critical code

2. Verification Method:

   • Run Valgrind on synthetic test suite with known memory errors
   • Verify all known errors are detected
   • Perform code review to catch errors Valgrind cannot detect (logic errors)

3. Tool Qualification Report:

   • Valgrind version: 3.21.0
   • Test suite: 250 test cases (known memory leaks, buffer overflows, use-after-free)
   • Results: 250/250 errors detected (100% detection rate)
   • Conclusion: Valgrind qualified for use in SWE.4 unit verification (ASIL B)

ASPICE Work Product: 15-19 (Tool qualification records).

Best Practices for Production Debugging

1. Minimal Intrusion for Real-Time Systems

Problem: Setting breakpoints in real-time control loops violates timing constraints.

Solution: Use non-intrusive debugging techniques:

• Trace buffers: Log events to circular buffer, dump after execution
• ETM hardware trace: Capture instruction flow without halting CPU
• GPIO toggling: Use logic analyzer to visualize execution flow

Example: Automotive ABS system (10ms control loop)

// Non-intrusive event logging
#define TRACE_BUFFER_SIZE 1024
uint32_t trace_buffer[TRACE_BUFFER_SIZE];
uint32_t trace_index = 0;

#define TRACE_EVENT(event_id) do { \
    trace_buffer[trace_index++ % TRACE_BUFFER_SIZE] = \
        (event_id << 16) | (get_timestamp_us() & 0xFFFF); \
} while(0)

void abs_control_loop(void) {
    TRACE_EVENT(EVENT_LOOP_START);

    read_wheel_speeds();
    TRACE_EVENT(EVENT_SENSORS_READ);

    calculate_brake_pressure();
    TRACE_EVENT(EVENT_CONTROL_CALCULATED);

    apply_brakes();
    TRACE_EVENT(EVENT_ACTUATORS_COMMANDED);
}

// After test run: Dump trace buffer for analysis
void dump_trace(void) {
    for (int i = 0; i < TRACE_BUFFER_SIZE; i++) {
        uint32_t event_id = trace_buffer[i] >> 16;
        uint32_t timestamp = trace_buffer[i] & 0xFFFF;
        printf("Event %u at %u us\n", event_id, timestamp);
    }
}

ASPICE Work Product: 17-11 (Integration test plan) specifies trace buffer analysis procedures.

2. Defensive Programming with Assertions

Technique: Use assertions to catch errors early during development, disable in production.

// ASPICE SWE.3: Defensive programming with assertions

#include <assert.h>

void set_motor_speed(int16_t speed) {
    // Precondition: Speed must be in valid range
    assert(speed >= -1000 && speed <= 1000);

    // Postcondition check (in debug build only)
    motor_speed_register = speed;
    assert(motor_speed_register == speed);  // Verify write succeeded
}

// Compile flags:
// Debug build:   gcc -DDEBUG -o firmware.elf
// Release build: gcc -DNDEBUG -o firmware.elf  (assertions disabled)

ASPICE Integration: Assertions active during SWE.4 unit testing, disabled for SWE.6 qualification tests (to match production configuration).

3. Reproducible Debugging Environments

Challenge: "Works on my machine" syndrome.

Solution: Dockerized debugging environment with fixed tool versions.

# Dockerfile: ASPICE SUP.8 reproducible build/debug environment
FROM ubuntu:22.04

# Install fixed tool versions
RUN apt-get update && apt-get install -y \
    gcc-arm-none-eabi=15:10.3-2021.07-4 \
    gdb-multiarch=12.1-0ubuntu1 \
    openocd=0.11.0-1 \
    valgrind=1:3.18.1-1ubuntu2

# Set up workspace
WORKDIR /workspace
COPY . /workspace

# Default command: Run unit tests with Valgrind
CMD ["valgrind", "--leak-check=full", "./unit_tests"]

Usage:

docker build -t aspice-debug-env .
docker run -v $(pwd):/workspace aspice-debug-env

ASPICE Work Product: 15-04 (Configuration management plan) specifies Docker image for reproducible debugging.

Cost-Benefit Analysis

Investment Required

Total Investment (small team, automotive ASIL B):

• Tools: $15,900
• Training: 10 days (€10,000 @ €1,000/day)
• Qualification: 40 hours (€4,000 @ €100/hour)
• Total: €29,900

Return on Investment

Benefits (quantified for 2-year automotive ECU project):

Total 2-Year Benefit: €830,000

ROI: (€830,000 - €29,900) / €29,900 = 2,672%

Key Takeaways

1. GDB is Universal: Master GDB for local and remote debugging; it supports all embedded platforms.

2. Remote Debugging is Essential: JTAG/SWD probes enable on-target debugging for embedded systems; J-Link is industry standard.

3. Memory Profiling Prevents Catastrophes: Valgrind and ASan detect memory leaks and buffer overflows before they cause field failures.

4. Performance Profiling Meets Real-Time Requirements: Use perf, Callgrind, or TRACE32 to verify execution time budgets.

5. Thermal/Power Profiling Critical for Embedded: Automotive and battery-powered systems require thermal and power analysis.

6. RTOS-Aware Debugging: Use FreeRTOS/Zephyr GDB plugins for multi-threaded embedded systems.

7. CI/CD Integration: Automate debugging with Valgrind/ASan in CI pipeline (ASPICE SWE.4, SWE.5).

8. Tool Qualification Required for Safety: ISO 26262 ASIL B+ requires tool qualification for TCL-1 tools; Valgrind is typically TCL-2.

9. Non-Intrusive Debugging for Real-Time: Use trace buffers and ETM hardware trace to debug without violating timing constraints.

10. Reproducible Environments: Use Docker to ensure consistent debugging environment across team (ASPICE SUP.8).

Further Reading

Books:

• The Art of Debugging with GDB, DDD, and Eclipse, Norman Matloff (No Starch Press, 2008)
• Embedded Systems Architecture, Daniele Lacamera (Packt, 2023)

Tools Documentation:

• GDB Manual: sourceware.org/gdb/documentation
• Valgrind User Manual: valgrind.org/docs/manual
• ARM CoreSight ETM: developer.arm.com/architectures/cpu-architecture/debug-visibility-and-trace

Standards:

• ISO 26262-6:2018 (Product development: Software level)
• DO-178C (Software Considerations in Airborne Systems and Equipment Certification)