5.5: SUP.9 Problem Resolution Management

Process Definition

Purpose

SUP.9 Purpose: To ensure that problems are identified, analyzed, managed, and controlled to resolution. The process establishes a disciplined approach for recording every problem detected across all lifecycle phases, performing systematic root cause analysis, implementing verified corrective actions, and feeding lessons learned back into the development process to prevent recurrence.

AI Value Proposition: AI transforms problem resolution from a reactive, labor-intensive activity into a proactive, pattern-driven discipline. Machine learning models trained on historical problem repositories can suggest root causes within minutes rather than days, detect duplicates before investigation effort is wasted, and predict emerging problem clusters before they escalate into release blockers.

Outcomes

Base Practices with AI Integration

AI Level Definitions: L1 = AI assists with templates and suggestions; L2 = AI performs analysis, human validates; L3 = AI executes autonomously with human oversight on exceptions.

AI-Assisted Problem Resolution

The following diagram illustrates the AI-assisted problem resolution workflow, from initial detection and classification through root cause analysis, corrective action, and closure verification.

Problem Report Template

Note: This automotive example demonstrates temperature-related timing investigation; adapt for project-specific domains.

# Problem Report (illustrative automotive example)
problem:
  id: PR-(year)-(number)
  title: "Door lock timing exceeds requirement at cold temperature"
  status: in_progress
  created: (creation date)
  reporter: Test Engineer

  classification:
    category: timing
    component: DoorLockControl
    severity: high
    priority: P1
    ai_confidence: 0.85

  description: |
    During HIL testing at -40C, door lock timing measured at 11.2ms,
    exceeding the 10ms requirement specified in SWE-BCM-103.

  reproduction:
    steps:
      - Set climate chamber to -40C
      - Wait for ECU temperature stabilization (30 min)
      - Send lock command via CAN
      - Measure actuator output timing
    rate: "100% reproducible at -40C"
    environment: "HIL system with climate chamber"

  ai_analysis:
    root_cause_suggestion: |
      Pattern match with 3 similar historical issues indicates
      motor driver slew rate degradation at cold temperature.

      Likely cause: GPIO driver configuration does not account
      for temperature-dependent propagation delay.

      Related issues: PR-2023-156, PR-2024-089
    confidence: medium
    suggested_investigation:
      - "Check motor driver IC datasheet for cold temp specs"
      - "Review GPIO driver strength configuration"
      - "Analyze timing margin at component level"

  investigation:
    actual_cause: |
      Motor driver transistor turn-on time increases from 50ns
      to 120ns at -40C due to reduced carrier mobility.
      Combined with 4 sequential actuator commands, adds ~0.3ms.
    root_cause_type: design
    verified: true
    verified_by: HW Engineer

  resolution:
    approach: "Increase GPIO drive strength for cold temperature"
    fix_description: |
      Added temperature-dependent GPIO configuration:
      - T > -20C: Standard drive strength (8mA)
      - T <= -20C: High drive strength (16mA)
    changed_files:
      - src/driver/gpio_driver.c
      - config/gpio_config.c
    change_request: CR-2025-015

  verification:
    test_cases:
      - id: SWE-QT-BCM-005
        result: pass
        measured: 9.8ms
    regression:
      status: pass
      scope: "Full DoorLock test suite"

  closure:
    closed_date: 2025-01-22
    closed_by: SW Lead
    lessons_learned: |
      Temperature compensation should be considered for all
      timing-critical functions during design phase.

AI-Powered Root Cause Analysis

ML Techniques for Root Cause Identification

AI-powered root cause analysis draws on multiple machine learning techniques applied to the corpus of historical problem reports. The objective is to reduce mean time to root cause identification by surfacing the most probable causes before an engineer begins manual investigation.

Data Quality Warning: ML model accuracy depends directly on the quality and consistency of historical problem data. Organizations beginning AI-assisted RCA should invest in structured problem report templates and retrospective labeling of legacy data before expecting reliable predictions.

AI-Assisted RCA

AI Root Cause Analysis Report:
------------------------------

Problem: PR-2025-042 (Door lock timing at cold temp)

Historical Pattern Analysis:
----------------------------
Searching historical issues with similar characteristics...

Match 1: PR-2023-156 (87% similarity)
- Project: Window Controller
- Symptom: Timing violation at cold temperature
- Root Cause: Transistor switching time degradation
- Resolution: Increased driver current

Match 2: PR-2024-089 (72% similarity)
- Project: Seat Controller
- Symptom: Motor response delay at -30C
- Root Cause: Power transistor thermal characteristics
- Resolution: Temperature compensation

Match 3: PR-2022-234 (65% similarity)
- Project: Mirror Controller
- Symptom: Actuator timing inconsistent
- Root Cause: PWM timing affected by temperature
- Resolution: Temperature-adjusted timing

Common Pattern Identified:
--------------------------
Category: Temperature-dependent timing degradation
Component: Power driver / Output stage
Physics: Semiconductor carrier mobility reduction at cold

Suggested Investigation Order:
------------------------------
1. [HIGH PROBABILITY] Check output driver temperature specs
2. [MEDIUM PROBABILITY] Review timing margin analysis
3. [LOW PROBABILITY] Verify power supply stability at cold

Confidence: 82%

Human Action Required:
[ ] Validate AI analysis against actual investigation
[ ] Confirm root cause
[ ] Approve resolution approach

Problem Classification

AI-Assisted Severity, Type, and Component Classification

When a new problem report is submitted, the AI classification engine analyzes the description text, attached logs, and structured metadata to automatically assign severity, problem type, and affected component. The engineer reviews the AI assignments and overrides where necessary; every override is fed back into the model as a training signal.

Override Feedback Loop: Every human override of an AI classification is stored as a labeled training sample. Models are retrained on a quarterly cadence or when override rate exceeds 25% for any classification dimension.

# AI Classification Output (illustrative)
classification_result:
  problem_id: PR-2025-042
  model_version: "cls-v3.2.1"
  timestamp: "2025-01-10T08:32:00Z"

  severity:
    predicted: high
    confidence: 0.87
    reasoning: "Timing violation on safety-relevant function at boundary temp"
    human_override: null

  problem_type:
    predicted: timing
    confidence: 0.91
    reasoning: "Keywords 'timing', 'exceeds', 'ms requirement' with HIL context"
    human_override: null

  component:
    predicted: DoorLockControl
    confidence: 0.85
    reasoning: "Test case SWE-QT-BCM-005 mapped to DoorLockControl module"
    alternatives:
      - component: GPIODriver
        confidence: 0.62
      - component: ActuatorControl
        confidence: 0.41
    human_override: null

  priority:
    assigned: P1
    rule_trace: "severity=high AND release_phase=integration AND customer_facing=true -> P1"

Duplicate Detection

NLP-Based Duplicate Problem Report Detection

Duplicate problem reports waste investigation effort and obscure true defect counts. The duplicate detection engine compares each incoming report against all open and recently closed reports using a two-stage pipeline.

Stage 1 -- Candidate Retrieval: A fast retrieval model (TF-IDF or BM25) identifies the top-50 most textually similar existing reports.

Stage 2 -- Semantic Re-ranking: A cross-encoder transformer model re-ranks candidates by semantic similarity, accounting for paraphrasing and domain synonyms (e.g., "timing violation" vs. "deadline exceeded").

Duplicate Detection Report:
----------------------------
Incoming: PR-2025-042 "Door lock timing exceeds requirement at cold temperature"

Candidate 1: PR-2025-038 (similarity: 0.68) -- NOT DUPLICATE
  Title: "Window motor timing exceeds spec at -40C"
  Verdict: Different component (WindowControl vs DoorLockControl),
           similar symptom pattern. Linked as RELATED, not duplicate.

Candidate 2: PR-2025-029 (similarity: 0.54) -- NOT DUPLICATE
  Title: "Door lock fails to engage intermittently"
  Verdict: Same component but different failure mode
           (functional vs timing). No link.

Candidate 3: PR-2024-089 (similarity: 0.71) -- RELATED
  Title: "Seat motor response delay at cold temperature"
  Verdict: Different project, similar root cause pattern.
           Linked as RELATED for cross-project learning.

Result: No duplicate found. PR-2025-042 confirmed as NEW.
Related issues linked: PR-2025-038, PR-2024-089

Human Confirmation Required: Even when the AI declares a definite duplicate (similarity >= 0.92), a human must confirm the linkage before the incoming report is merged. Auto-closure of duplicates without human review is prohibited in safety-relevant projects.

Resolution Recommendation

AI-Suggested Fixes Based on Historical Resolutions

Once a root cause category is established, the resolution recommendation engine searches the historical resolution database for verified fixes applied to similar problems. Recommendations are ranked by relevance, success rate, and applicability to the current project context.

# Resolution Recommendation Output (illustrative)
resolution_recommendations:
  problem_id: PR-2025-042
  root_cause_category: "temperature-dependent-timing-degradation"

  recommendations:
    - rank: 1
      approach: "Temperature-dependent drive strength adjustment"
      confidence: 0.88
      source_problems: [PR-2023-156, PR-2024-089]
      success_rate: "2/2 (100%) -- no recurrence in 12 months"
      estimated_effort: "16-24 hours"
      risk: low
      description: |
        Implement conditional GPIO drive strength configuration
        based on temperature sensor reading. Switch to high drive
        strength below a calibrated threshold (typically -20C).
      files_likely_affected:
        - "src/driver/gpio_driver.c"
        - "config/gpio_config.c"

    - rank: 2
      approach: "Timing margin increase via requirement relaxation"
      confidence: 0.52
      source_problems: [PR-2022-234]
      success_rate: "1/1 (100%) -- but different context"
      estimated_effort: "8-12 hours"
      risk: medium
      description: |
        Negotiate requirement relaxation from 10ms to 12ms at
        extreme cold temperatures. Requires customer agreement
        and safety impact analysis.
      files_likely_affected:
        - "docs/requirements/SWE-BCM-103.md"

    - rank: 3
      approach: "Hardware modification -- driver IC substitution"
      confidence: 0.31
      source_problems: []
      success_rate: "N/A -- inferred from datasheet analysis"
      estimated_effort: "80-120 hours (HW change)"
      risk: high
      description: |
        Replace current motor driver IC with automotive-grade
        part rated for extended cold temperature operation.
        Requires HW redesign and requalification.

  ai_recommendation: |
    Rank 1 approach (temperature-dependent drive strength) is
    strongly recommended based on 100% historical success rate,
    low implementation risk, and direct applicability to the
    current DoorLockControl architecture.

  human_action_required:
    - "Review recommended approach for technical feasibility"
    - "Approve resolution strategy before implementation"
    - "Create change request CR linked to this problem report"

Trend Analysis

Predicting Future Problems from Patterns

Trend analysis moves problem resolution from reactive firefighting to proactive prevention. The AI trend engine operates on three time horizons.

Trend Indicators and Thresholds

Trend Analysis Report (Q4 2024):
---------------------------------
Analysis period: October - December 2024
Total problems: 147 (vs 112 in Q3 -- +31%)

Category Breakdown:
  Timing:        42 (29%) -- UP from 18% in Q3 [ALERT]
  Functional:    38 (26%) -- stable
  Interface:     29 (20%) -- stable
  Resource:      21 (14%) -- DOWN from 19%
  Safety:        17 (12%) -- stable

Component Hot Spots:
  DoorLockControl:   34 problems (23%) [ALERT -- concentration]
  WindowControl:     22 problems (15%)
  SeatController:    19 problems (13%)

Prediction for Q1 2025:
  Expected problem count: 155-175 (continued upward trend)
  Highest risk category: Timing (predicted 35-40% share)
  Highest risk component: DoorLockControl

Recommended Preventive Actions:
  1. Conduct timing margin review across all DoorLockControl functions
  2. Add temperature boundary tests to CI pipeline
  3. Schedule architectural review of timing-critical paths
  4. Increase code review depth for timing-related changes

Confidence: 76%

Knowledge Base Building

Automatic Knowledge Extraction from Resolved Problems

Every resolved and closed problem report contains valuable engineering knowledge. The knowledge extraction engine automatically distills lessons learned, reusable fix patterns, and design guidelines from closed reports and organizes them into a searchable knowledge base.

# Auto-Generated Knowledge Base Article (illustrative)
knowledge_article:
  id: KB-2025-017
  title: "Temperature-Dependent Timing Degradation in Motor Drivers"
  generated_from: [PR-2025-042, PR-2024-089, PR-2023-156, PR-2022-234]
  auto_generated: true
  human_reviewed: true
  reviewed_by: "SW Architect"
  review_date: "2025-02-01"

  symptom_pattern: |
    Actuator or motor timing measurements exceed requirements at
    temperatures below -20C. Degradation is proportional to
    temperature drop and affects all output stages using standard
    MOSFET drivers without temperature compensation.

  root_cause_mechanism: |
    Semiconductor carrier mobility decreases at low temperatures,
    increasing transistor turn-on and turn-off times. For MOSFET
    drivers, Rds(on) temperature coefficient is positive but
    switching time coefficient is negative, leading to slower
    edge rates at cold extremes.

  recommended_design_practice: |
    For all timing-critical output stages:
    1. Include temperature sensor reading in driver configuration
    2. Implement at least two drive strength levels (standard / high)
    3. Define temperature threshold for switching (typically -20C)
    4. Add timing margin analysis at -40C during design review
    5. Include cold temperature boundary tests in verification plan

  applicability:
    domains: [body-control, seat-control, window-control, mirror-control]
    mcu_families: [all]
    safety_levels: [QM, ASIL-A, ASIL-B]

  related_articles: [KB-2024-008, KB-2023-042]
  tags: [timing, temperature, motor-driver, GPIO, cold-start]

Knowledge Base Maintenance: Articles are automatically flagged for review when new problem reports match the article pattern but the documented fix did not prevent recurrence. This feedback loop ensures the knowledge base remains current and trustworthy.

Root Cause Analysis

AI-Assisted RCA

AI Root Cause Analysis Report:
------------------------------

Problem: PR-2025-042 (Door lock timing at cold temp)

Historical Pattern Analysis:
----------------------------
Searching historical issues with similar characteristics...

Match 1: PR-2023-156 (87% similarity)
- Project: Window Controller
- Symptom: Timing violation at cold temperature
- Root Cause: Transistor switching time degradation
- Resolution: Increased driver current

Match 2: PR-2024-089 (72% similarity)
- Project: Seat Controller
- Symptom: Motor response delay at -30C
- Root Cause: Power transistor thermal characteristics
- Resolution: Temperature compensation

Match 3: PR-2022-234 (65% similarity)
- Project: Mirror Controller
- Symptom: Actuator timing inconsistent
- Root Cause: PWM timing affected by temperature
- Resolution: Temperature-adjusted timing

Common Pattern Identified:
--------------------------
Category: Temperature-dependent timing degradation
Component: Power driver / Output stage
Physics: Semiconductor carrier mobility reduction at cold

Suggested Investigation Order:
------------------------------
1. [HIGH PROBABILITY] Check output driver temperature specs
2. [MEDIUM PROBABILITY] Review timing margin analysis
3. [LOW PROBABILITY] Verify power supply stability at cold

Confidence: 82%

Human Action Required:
[ ] Validate AI analysis against actual investigation
[ ] Confirm root cause
[ ] Approve resolution approach

HITL Protocol for Problem Resolution Decisions

Human-in-the-Loop Decision Points

Every AI action in the problem resolution process has a defined HITL gate. The table below specifies which decisions AI can make autonomously, which require human confirmation, and which are exclusively human.

Guiding Principle: Humans own decisions; AI accelerates analysis. No problem report may be closed, reclassified to a lower severity, or marked as duplicate without explicit human approval. AI outputs are advisory inputs to human decision-making, not autonomous actions.

Override and Feedback Protocol

HITL Override Workflow:
-----------------------

1. AI presents classification / RCA / resolution recommendation
2. Engineer reviews AI output:
   a. ACCEPT  --> AI assignment stands; logged as confirmed
   b. MODIFY  --> Engineer adjusts; delta logged as training feedback
   c. REJECT  --> Engineer provides correct value; logged as override

3. All overrides stored in feedback database:
   - Problem ID
   - AI prediction (field, value, confidence)
   - Human decision (field, value, rationale)
   - Timestamp, engineer ID

4. Quarterly model retraining incorporates override data
5. Override rate monitored per classification dimension:
   - Target: < 15% override rate
   - Action trigger: > 25% override rate initiates model review

Problem Metrics Dashboard

The diagram below shows the problem resolution dashboard, presenting open/closed problem counts, resolution time trends, severity distribution, and aging analysis.

Metrics and KPIs

Core Problem Resolution Metrics

AI-Specific Performance Metrics

Calibration Period: During the first 6 months of AI deployment, accuracy targets are relaxed by 10 percentage points to allow model calibration. Monthly accuracy reviews determine when full targets apply.

Tool Integration

Issue Tracking and AI Plugin Architecture

The problem resolution process relies on tight integration between the issue tracking system and AI analysis services. The following table maps supported tools to their AI integration points.

Integration Architecture Pattern

# AI Problem Analysis Service Configuration (illustrative)
problem_analysis_service:
  name: "ai-problem-analyzer"
  version: "2.1.0"

  endpoints:
    classify:
      path: "/api/v1/classify"
      method: POST
      input: problem_report_json
      output: classification_result_json
      timeout: 10s

    detect_duplicates:
      path: "/api/v1/duplicates"
      method: POST
      input: problem_report_json
      output: duplicate_candidates_json
      timeout: 15s

    suggest_root_cause:
      path: "/api/v1/rca"
      method: POST
      input: problem_report_json
      output: rca_suggestions_json
      timeout: 30s

    recommend_resolution:
      path: "/api/v1/resolution"
      method: POST
      input: problem_with_rca_json
      output: resolution_recommendations_json
      timeout: 20s

    analyze_trends:
      path: "/api/v1/trends"
      method: GET
      params: [project, date_range, categories]
      output: trend_report_json
      timeout: 60s

  issue_tracker_webhooks:
    jira:
      event: "jira:issue_created"
      actions: [classify, detect_duplicates, suggest_root_cause]
    bugzilla:
      event: "bug.create"
      actions: [classify, detect_duplicates, suggest_root_cause]
    polarion:
      event: "workitem.created"
      actions: [classify, detect_duplicates, suggest_root_cause]

  authentication:
    method: "OAuth2 client credentials"
    token_endpoint: "https://auth.example.com/oauth/token"

  data_privacy:
    pii_scrubbing: enabled
    data_retention: "24 months for analysis; 7 years for audit trail"
    gdpr_compliance: true

Work Products

Implementation Checklist

Phase 1: Foundation (Months 1-3)

Phase 2: AI Deployment (Months 4-6)

Phase 3: Optimization (Months 7-12)

Continuous Improvement: After Phase 3, the implementation enters a continuous improvement cycle. Quarterly reviews assess model accuracy, override rates, and KPI trends. Annual process audits verify that AI integration continues to satisfy ASPICE SUP.9 outcomes.

Summary

SUP.9 Problem Resolution:

• AI Level: L2 (AI analysis, human validation)
• Primary AI Value: Root cause suggestion, pattern matching, duplicate detection, resolution recommendation
• Human Essential: Cause validation, fix implementation, closure approval
• Key Outputs: Problem reports, RCA records, resolution records, trend reports, knowledge base articles
• AI Accuracy: ~78% root cause suggestion accuracy (illustrative target; calibrate based on historical data quality)
• Critical HITL Gates: Classification override, RCA validation, duplicate confirmation, closure sign-off