1.4: Limitation Acknowledgment

AI Limitations in Safety-Critical Development

Fundamental Constraints

1. No Legal Accountability [WARN]

Limitation: AI cannot be held legally liable for product failures

Example:

  • Scenario: AI-generated brake control code causes accident (vehicle fails to stop)
  • Legal Question: Who is liable?
    • [FAIL] AI vendor (OpenAI, Anthropic): Disclaims liability in Terms of Service
    • [FAIL] AI agent: Cannot be sued (non-legal entity)
    • [PASS] Human engineer who approved code: Legally responsible

Implication: Human must review and approve all safety-critical code (non-negotiable)

2. No Domain Expertise [WARN]

Limitation: AI trained on general code corpus, not domain-specific safety standards

Example:

  • Prompt: "Design fail-safe behavior for ACC if radar sensor fails"
  • AI Output (incorrect): 
    if (radar_failed) {
    ACC_speed = 0;  // [FAIL] WRONG: Abrupt stop dangerous (rear-end collision)
}
  • Correct Output (human safety engineer): 
    if (radar_failed) {
    ACC_SetState(ACC_DEGRADED_MODE);   // Gradual deceleration
    ACC_AlertDriver("Radar fault");     // Driver takeover
    ACC_LogEvent(FAULT_RADAR_SENSOR);   // Diagnostics
}

Implication: Human safety engineer designs fail-safe logic, AI generates implementation from human spec

ISO 26262 Part 8 Reference: Section 11.4.6 requires that safety-related software development be performed by qualified personnel. AI tools are classified as support tools (TCL1-3) per Part 8, requiring tool qualification for safety-critical use. See Chapter 21 for ASPICE/ISO 26262 tool qualification guidance.

3. No Contextual Business Knowledge [WARN]

Limitation: AI doesn't know project constraints (budget, schedule, OEM preferences)

Example:

  • Question: "Should we use AUTOSAR Classic or Adaptive?"
  • AI Response: Lists pros/cons (accurate), but cannot decide
    • Classic: Proven, lower cost, mature tools
    • Adaptive: Flexible, OTA updates, higher cost
  • Decision Requires:
    • OEM requirement (does customer mandate Adaptive?)
    • Budget (Adaptive tools +€200k)
    • Schedule (Adaptive training +2 months)
  • Human Decision: Architect considers context, chooses Classic

Implication: AI advises (pros/cons), human decides (final call)

4. No Creative Problem-Solving [WARN]

Limitation: AI pattern-matches from training data, doesn't invent new solutions

Example:

  • Novel Bug: CAN message corruption only at 200 km/h (timing issue, electromagnetic interference)
  • AI Debugging: Suggests standard fixes (check message ID, verify CRC)
  • Actual Root Cause: EMI from nearby motor at high speed (requires oscilloscope, domain expertise)
  • Human Investigation: Identifies EMI, adds shielding, resolves bug

Implication: AI handles standard bugs, human handles novel problems

Workaround Strategies per Limitation: (1) Legal accountability: Maintain human review gate; (2) Domain expertise: Provide domain-specific context in prompts, use fine-tuned models; (3) Business context: Include constraints in system prompt; (4) Creative problem-solving: Use AI for hypothesis generation, human for validation. Each limitation has partial mitigations that improve AI utility without removing human oversight.

Escalation Criteria

When AI Must Escalate to Human

Rule: The AI should proactively escalate rather than attempt tasks beyond its capability

Escalation Triggers:

1. Safety-Critical Decision Required [ESCALATION]

  • Trigger: Task involves ISO 26262 ASIL-B/C safety function
  • Example: "Determine if sensor redundancy 1oo2 or 2oo3 needed for this ASIL-B function"
  • Action: Escalate to human safety engineer, provide options + analysis

2. Ambiguous Requirement [ESCALATION]

  • Trigger: Requirement lacks quantification, units, or specificity
  • Example: "System shall respond quickly" (no time specified)
  • Action: Generate clarification question, send to requirements engineer

3. Conflicting Constraints [ESCALATION]

  • Trigger: Cannot satisfy all requirements simultaneously
  • Example: Requirement A: "Latency ≤10ms", Requirement B: "Use complex Kalman filter" (needs 50ms)
  • Action: Alert human architect, suggest trade-offs

4. Tool/Library Selection [ESCALATION]

  • Trigger: Need to choose between multiple libraries (licensing, performance, safety)
  • Example: "Use FreeRTOS (MIT license) or SafeRTOS (certified, expensive)?"
  • Action: Escalate to architect, provide comparison table

5. Test Failure (No Obvious Fix) [ESCALATION]

  • Trigger: Unit test fails, AI cannot debug after 3 attempts
  • Example: CRC-32 test fails (expected 0x12345678, actual 0x12345679, off by 1)
  • Action: Provide debugging log, request human investigation

6. Out-of-Scope Complexity [ESCALATION]

  • Trigger: Task exceeds AI training (e.g., device driver for custom hardware)
  • Example: "Write SPI driver for proprietary ASIC (no datasheet available)"
  • Action: Notify human that task requires domain-specific knowledge

Escalation Template

Standard Escalation Message

## [ESCALATION] ESCALATION REQUEST

**Agent**: [Implementation Agent / Requirements Agent / etc.]
**Task ID**: ASPICE-1234
**Issue Category**: [Safety Decision / Ambiguity / Conflict / Tool Selection / Test Failure / Out of Scope]

### Context
[Describe the task, what AI attempted, why it cannot proceed]

### Specific Question
[Clear, answerable question for human expert]

### AI Analysis (if applicable)
[Options, pros/cons, preliminary research]

### Recommended Action
[AI's suggestion, if any - or "Requires human expertise"]

### Urgency
- [TIME] **Blocker**: Blocks pull request, needs immediate attention
- [TIMER] **Medium**: Can wait 24-48 hours
- [DATE] **Low**: Can wait until next sprint

### Assignee
[Tag appropriate human expert: @safety_engineer, @architect, @requirements_lead]

Example Escalation:

## [ESCALATION] ESCALATION REQUEST

**Agent**: Implementation Agent
**Task ID**: ASPICE-1234
**Issue Category**: Safety Decision

### Context
Implementing function `ACC_EmergencyBrake()` (ASIL-B safety function).
Requirement [SWE-089]: "If lead vehicle detected <5m away, apply emergency brake."

AI generated code but uncertain about fail-safe behavior if brake actuator fails.

### Specific Question
If brake actuator fault detected (CAN timeout), should we:
  A) Retry brake command 3 times (wait 100ms each)
  B) Alert driver + disable ACC immediately (driver takes over)
  C) Attempt redundant brake system (if available in vehicle)

### AI Analysis
- **Option A**: May delay response, increases collision risk (rejected by AI)
- **Option B**: Simple, deterministic, aligns with ISO 26262 safe state transition
  (RECOMMENDED by AI based on general safety principles)
- **Option C**: Requires knowledge of vehicle architecture (redundancy available?)

### Recommended Action
Implement Option B (alert + disable), pending human safety engineer approval.

Draft code:
```c
if (brake_actuator_fault) {
    ACC_SetState(ACC_FAILSAFE);
    HMI_DisplayWarning("ACC DISABLED - BRAKE FAULT");
    EventLogger_Log(FAULT_BRAKE_ACTUATOR);
}

Urgency

[TIME] Blocker: Pull request #142 awaiting approval

Assignee

@safety_engineer (Bob) - Requires ISO 26262 expert approval


---

## Limitation Communication to Users

### Transparency with Engineers

**Principle**: AI should **openly acknowledge** its limitations — doing so builds trust

**Example Interaction**:

**Engineer**: "Can you optimize this Kalman filter for 1ms latency?"

**AI Response**:

I can assist with Kalman filter optimization, but achieving 1ms latency on TriCore 300 MHz requires domain-specific embedded optimization techniques beyond my training.

What I can do: [PASS] Suggest algorithmic optimizations (reduce matrix size, simplified EKF) [PASS] Profile code (identify hot paths via profiler) [PASS] Apply standard compiler optimizations (O3, loop unrolling)

What I cannot do: [FAIL] Hand-optimize assembly code (requires TriCore ISA expertise) [FAIL] Guarantee 1ms latency (needs hardware profiling, WCET analysis) [FAIL] Tune Q/R matrices (requires system identification, control theory)

Recommendation: I'll generate an optimized C implementation (estimated 5-10ms latency). If you need <1ms, please consult an embedded systems expert or consider:

  • Hardware acceleration (DSP, FPGA)
  • Algorithm simplification (reduce state vector size)
  • Assembly optimization (TriCore specialist)

Proceed with C optimization? (Yes/No)


**Benefit**: Engineer knows AI limitations upfront, avoids wasted iteration

---

## Known Failure Modes

### Common AI Mistakes in ASPICE Context

**1. Over-Generalization** [WARN]
- **Issue**: AI applies pattern from one domain to another (incorrectly)
- **Example**:
  - AI trained on web dev (JavaScript, Python)
  - Generates C code with dynamic memory (malloc) for ASIL-B (not allowed, MISRA Rule 21.3)
- **Mitigation**: Human review checks MISRA compliance, rejects dynamic memory

**2. Missing Edge Cases** [WARN]
- **Issue**: AI generates unit tests for typical values, misses boundary cases
- **Example**:
  - Function: `int Divide(int a, int b)`
  - AI Tests: (10, 2), (100, 5), (7, 3) [PASS]
  - **Missing**: (10, 0) → Divide by zero [FAIL]
- **Mitigation**: Human adds edge case tests (zero, max/min int, negative)

**3. Shallow Doxygen Comments** [WARN]
- **Issue**: AI generates correct syntax, but shallow content
- **Example**:
  - **AI Comment**: "@brief Calculate CRC" (not helpful)
  - **Human Edit**: "@brief Calculate CRC-32 checksum using IEEE 802.3 polynomial (0xEDB88320). Used for CAN message integrity verification (ASIL-B)."
- **Mitigation**: Human enriches comments with context, safety notes

**4. Ignoring System Constraints** [WARN]
- **Issue**: AI doesn't know CPU, memory, latency constraints
- **Example**:
  - **Constraint**: ECU has 256 KB RAM
  - **AI Code**: Allocates 1 MB lookup table (exceeds RAM)
- **Mitigation**: Human specifies constraints in prompt, reviews generated code

---

## Self-Assessment Checklist (for AI Agents)

### Before Generating Output

**AI Agent Self-Check**:

```yaml
Pre-Generation Checklist (AI Internal):
─────────────────────────────────────────────────────────

[ ] Is this task safety-critical (ASIL-B/C, SIL 3, Class C)?
   → If YES: Flag for human review (100% coverage required)

[ ] Do I have sufficient context?
   → Requirement specification? (Yes/No)
   → Design constraints? (CPU, memory, latency)
   → Coding standards? (MISRA C:2012, CERT C)
   → If NO: Request additional context from human

[ ] Is this task within my training data?
   → Standard algorithm (CRC, sort)? → [PASS] Proceed
   → Novel domain (custom hardware driver)? → [FAIL] Escalate

[ ] Can I verify correctness?
   → Unit tests available? (Yes/No)
   → Reference implementation? (IEEE spec, RFC)
   → If NO: Lower confidence, flag for extra human review

[ ] Is there ambiguity in requirements?
   → "Quickly", "safe", "user-friendly" (vague terms)? → [FAIL] Escalate
   → Missing units (temperature threshold, no value)? → [FAIL] Escalate

Decision:
  [PASS] Proceed: Generate output, submit for human review
  [WARN] Partial: Generate draft, flag limitations, request completion
  [FAIL] Escalate: Cannot proceed, escalate to human expert

Summary

AI Limitation Acknowledgment Principles:

  1. No Legal Accountability: Human must approve safety-critical code (AI cannot be sued)
  2. No Domain Expertise: Human designs fail-safe logic (AI implements from spec)
  3. No Business Context: Human makes architectural decisions (AI advises)
  4. No Creative Problem-Solving: Human handles novel bugs (AI assists)
  5. Proactive Escalation: AI escalates safety decisions, ambiguities, conflicts, failures
  6. Transparency: AI openly acknowledges limitations (builds trust with engineers)
  7. Self-Assessment: AI checks task suitability before generating output

Golden Rule for AI Agents: When in doubt, escalate to human

Chapter 29 Complete: Agent Framework establishes AI roles, HITL protocol, capability mapping, and limitation acknowledgment.

Next: Chapter 30 - Process Execution Instructions (detailed task guidance per ASPICE process)

Previous
Capability Mapping
Next
Agent Teams