5.1: SUP.1 Quality Assurance

Process Definition

Purpose

SUP.1 Purpose: To provide independent and objective assurance that work products and processes comply with defined criteria and that non-conformances are resolved and further prevented.

Why QA Matters in AI-Assisted Development: When AI tools generate or modify work products, the traditional assumption that a human author has verified compliance no longer holds by default. Quality assurance must explicitly verify that AI-generated outputs meet the same standards as human-authored ones, and that the AI tools themselves are used within their qualified boundaries.

Outcomes

Base Practices with AI Integration

AI Integration Levels: L1 = AI-assisted (templates, suggestions); L2 = AI-augmented (automated checking, human confirms); L3 = AI-automated (autonomous execution with human oversight).

AI-Powered Quality Assurance

Principle: AI transforms QA from periodic, sample-based auditing to continuous, comprehensive monitoring. However, the QA engineer retains authority over all compliance decisions and finding classifications.

Automated Compliance Checking Against Standards

AI-powered compliance checking continuously verifies work products and processes against applicable standards, project plans, and regulatory requirements.

# Automated compliance check configuration (illustrative example)
compliance_check:
  name: "ASPICE SWE.1 Requirements Compliance"
  schedule: "on_commit"  # Trigger on every commit to requirements repository

  checks:
    - id: CHK-REQ-001
      name: "Requirement completeness"
      rule: "Every requirement must have: ID, description, priority, verification criterion"
      severity: HIGH
      tool: "ai_nlp_analyzer"

    - id: CHK-REQ-002
      name: "Requirement testability"
      rule: "Requirements must contain measurable acceptance criteria"
      severity: MEDIUM
      tool: "ai_nlp_analyzer"

    - id: CHK-REQ-003
      name: "Traceability completeness"
      rule: "Every SW requirement must trace to at least one system requirement"
      severity: HIGH
      tool: "polarion_api"

    - id: CHK-REQ-004
      name: "No ambiguous language"
      rule: "Flag use of 'should', 'may', 'might', 'could', 'appropriate', 'as needed'"
      severity: LOW
      tool: "ai_nlp_analyzer"

  reporting:
    format: "json"
    destination: "qa_dashboard"
    notify_on_failure: ["qa_lead@project.com"]

Process Adherence Monitoring

Continuous vs. Periodic QA: Traditional QA relies on scheduled audits that sample a subset of activities. AI-powered process monitoring checks every workflow execution, every artifact state transition, and every approval event in real time.

Work Product Quality Scoring

AI assigns quality scores to work products based on multiple dimensions, providing QA engineers with a prioritized view of where human review effort is most needed.

AI Work Product Quality Report:
Document: SRS_BCM_v1.2.docx
Analysis Date: 2025-01-15

Dimension         Score   Status    Details
─────────────────────────────────────────────────────────────
Completeness      87%     WARNING   3 requirements missing verification criteria
Consistency       94%     PASS      2 minor terminology inconsistencies
Correctness       91%     PASS      All structural rules satisfied
Clarity           83%     PASS      2 requirements flagged for ambiguity
Traceability      90%     WARNING   5 of 48 requirements missing upward traces
─────────────────────────────────────────────────────────────
Composite Score   89%     WARNING   Below 90% release threshold

Priority Review Areas:
1. [HIGH] Complete traceability for SWE-BCM-110..114
2. [MEDIUM] Add verification criteria to SWE-BCM-103, 107, 119
3. [LOW] Resolve terminology: "actuator" vs "motor driver" usage

Human Review Required:
- Address all HIGH priority items before release gate
- Confirm LOW items are acceptable terminology variations

Deviation Detection and Prediction

Predictive QA: Beyond detecting existing deviations, AI analyzes historical patterns to predict where quality problems are likely to emerge, enabling preventive action rather than corrective action.

# Deviation prediction configuration (illustrative example)
deviation_prediction:
  model: "qa_deviation_predictor_v2"
  training_data: "historical_ncr_database"

  risk_indicators:
    - name: "Requirements volatility"
      threshold: "> 15% change rate after SWE.1 baseline"
      prediction: "HIGH risk of downstream rework and traceability gaps"

    - name: "Review finding density drop"
      threshold: "< 0.5 findings per review session"
      prediction: "MEDIUM risk of superficial reviews"

    - name: "Test case growth stagnation"
      threshold: "Test cases not growing proportionally to requirements"
      prediction: "HIGH risk of insufficient test coverage"

    - name: "Late integration defects"
      threshold: "> 30% of defects found in integration testing"
      prediction: "MEDIUM risk of inadequate unit testing"

  actions:
    - trigger: "HIGH risk prediction"
      action: "Schedule focused QA audit within 5 business days"
      notify: ["qa_lead", "project_manager"]

    - trigger: "MEDIUM risk prediction"
      action: "Add to next scheduled QA review agenda"
      notify: ["qa_lead"]

AI-Assisted QA Framework

The diagram below presents the AI-assisted QA framework, showing how automated compliance checks, evidence collection, and trend analysis feed into the QA audit workflow while maintaining human decision authority.

QA Audit Automation

Note: AI-assisted audits do not replace human auditor judgment. AI handles evidence gathering, checklist pre-population, and compliance pre-screening. The auditor interprets findings, assesses context, and makes the final compliance determination.

Audit Preparation

AI accelerates audit preparation by automatically gathering evidence, pre-populating checklists, and identifying areas of concern before the auditor begins.

Audit Execution Support

# AI-assisted audit execution workflow (illustrative example)
audit_execution:
  phase: "preparation"
  steps:
    - step: "Auto-gather evidence"
      ai_actions:
        - "Query Polarion for all SWE.1 work products"
        - "Retrieve review records from GitLab merge requests"
        - "Extract test results from CI/CD pipeline artifacts"
        - "Collect approval records from workflow engine"
      output: "evidence_package.zip"

    - step: "Pre-screen compliance"
      ai_actions:
        - "Check each checklist item against gathered evidence"
        - "Mark items as PASS (evidence found and valid), FAIL (evidence missing or invalid), or REVIEW (ambiguous)"
        - "Calculate preliminary compliance score"
      output: "prescreened_checklist.yaml"

    - step: "Risk-based focus areas"
      ai_actions:
        - "Identify checklist items marked REVIEW or FAIL"
        - "Cross-reference with historical audit findings"
        - "Rank focus areas by risk and impact"
      output: "audit_focus_report.md"

  phase: "execution"
  steps:
    - step: "Auditor reviews pre-screened results"
      human_actions:
        - "Validate AI PASS determinations (sample-based)"
        - "Investigate all FAIL and REVIEW items"
        - "Interview process participants as needed"
        - "Document findings with objective evidence"

    - step: "Finding classification"
      ai_actions:
        - "Suggest finding severity based on ASPICE rating impact"
        - "Link findings to specific base practice gaps"
        - "Propose corrective action categories"
      human_actions:
        - "Confirm or override severity classification"
        - "Finalize finding descriptions"
        - "Agree corrective actions with process owner"

Process Audit Example

# QA Process Audit Checklist (illustrative example)
audit:
  id: QA-AUDIT-001
  process: SWE.1
  date: (audit date)
  auditor: AI-Assisted + QA Lead

  checklist:
    work_products:
      - id: WP-17-08
        name: "SW Requirements Specification"
        check: "Document exists and is under version control"
        status: PASS
        evidence: "SRS_BCM_v1.2.docx in Polarion"

      - id: WP-17-11
        name: "Traceability Record"
        check: "All SW requirements traced to system requirements"
        status: PARTIAL
        finding: "5 of 48 requirements missing traces"
        severity: MEDIUM

      - id: WP-17-12
        name: "Verification Criteria"
        check: "All requirements have verification criteria"
        status: PASS
        evidence: "Verification matrix complete"

    reviews:
      - type: "Requirements Review"
        check: "Review conducted with documented results"
        status: PASS
        evidence: "Review record RR-SWE1-001"

      - type: "Stakeholder Agreement"
        check: "Requirements approved by stakeholders"
        status: FAIL
        finding: "No formal approval record found"
        severity: HIGH

    tools:
      - tool: "Polarion ALM"
        check: "Tool configured per project standards"
        status: PASS

  findings_summary:
    high: 1
    medium: 1
    low: 0

  overall_status: "CONDITIONAL PASS - Address findings"

AI Quality Analysis

L2: Automated Quality Checking

AI Quality Analysis Report:
Document: SRS_BCM_v1.2.docx
Analysis Date: 2025-01-15

The diagram below visualizes the AI completeness analysis results, showing coverage percentages across requirement categories and flagging areas with insufficient specification.

Human Review Required:
- Address ambiguity in flagged requirements
- Complete missing interface requirements
- Add rationale to requirements missing justification
- Fill traceability gaps

Non-Conformance Tracking

Finding Management Workflow

# Non-conformance finding (illustrative example)
finding:
  id: NCR-(year)-(number)
  source: QA-AUDIT-001
  date: (finding date)
  severity: HIGH
  category: process

  description: |
    No formal stakeholder approval record exists for software
    requirements specification SRS_BCM_v1.2.

  evidence: |
    - Review record RR-SWE1-001 shows technical review completed
    - No customer/stakeholder sign-off document found
    - Requirements marked as "Draft" in Polarion

  root_cause_analysis:
    ai_suggestion: |
      Similar pattern in 3 previous projects suggests process gap:
      Review process does not include explicit stakeholder approval step.
    confidence: medium
    human_validation: pending

  corrective_action:
    immediate: "Obtain stakeholder approval for SRS_BCM_v1.2"
    systemic: "Add stakeholder approval gate to requirements workflow"
    owner: Project Manager
    due_date: 2025-01-22

  status: open
  tracking:
    opened: 2025-01-15
    target_close: 2025-01-22
    escalation_date: 2025-01-25

Quality Metrics Dashboard

Note: Metrics should be tailored to project size and ASPICE target capability level. The following represents a comprehensive set; select metrics appropriate to your context.

Core QA Metrics

QA Key Performance Indicators (KPIs)

AI-Driven Insights

AI augments the metrics dashboard by providing contextual analysis that goes beyond raw numbers.

The dashboard below consolidates the AI-driven QA insights described above into a single monitoring view, tracking audit compliance, finding trends, and predictive alerts.

HITL Protocol for QA Decisions

Fundamental Principle: AI augments QA efficiency but must never replace human judgment for compliance determinations. ASPICE requires that quality assurance is performed by competent personnel; AI is a tool, not a qualified auditor.

Decision Authority Matrix

Mandatory Human Checkpoints

1. Audit finding confirmation: Every finding flagged by AI must be reviewed and confirmed or dismissed by a qualified QA auditor before it is formally recorded
2. Severity classification: AI severity suggestions are advisory; the auditor determines the final severity based on project context, safety impact, and customer requirements
3. Corrective action approval: All corrective actions, whether AI-suggested or human-defined, require approval by the responsible process owner and QA lead
4. Audit report sign-off: The audit report is a controlled work product; only the designated QA auditor may sign off on the final version
5. Quality gate decisions: Gate pass/fail determinations remain exclusively human decisions; AI provides the data, humans make the call
6. Escalation authorization: While AI triggers escalation alerts, the QA lead determines whether formal escalation to management is warranted

Override and Feedback Protocol

# HITL override tracking (illustrative example)
hitl_override:
  id: HITL-QA-001
  date: (override date)
  ai_recommendation: "FAIL - Missing stakeholder approval record"
  human_decision: "CONDITIONAL PASS"

  justification: |
    Verbal approval obtained in steering committee meeting SC-2025-003.
    Formal sign-off scheduled for 2025-01-20. Risk accepted for 5 business
    days to avoid blocking downstream activities.

  approved_by: "QA Lead"
  conditions:
    - "Formal approval must be obtained by 2025-01-20"
    - "If not obtained, finding reverts to FAIL status"

  feedback_to_ai:
    category: "verbal_approval_acceptance"
    learning: "Verbal approval with documented meeting minutes may be
              accepted as interim evidence with time-bounded conditions"

Tool Integration

Note: Tool selection depends on project infrastructure, budget, and existing toolchain. The following integrations are representative; adapt to your specific environment.

SonarQube Integration

SonarQube provides continuous code quality inspection that feeds directly into the QA compliance framework.

Axivion Suite Integration

Axivion provides architecture verification and code quality analysis particularly suited to embedded and safety-critical software.

Custom AI QA Agents

Purpose-built AI agents extend QA automation beyond what commercial tools provide.

# Custom AI QA agent configuration (illustrative example)
qa_agents:
  traceability_auditor:
    schedule: "daily at 02:00 UTC"
    data_sources:
      - type: "polarion"
        query: "project:BCM AND type:requirement"
      - type: "polarion"
        query: "project:BCM AND type:test_case"
    checks:
      - "Every requirement has at least one test case linked"
      - "Every test case traces to at least one requirement"
      - "No orphaned requirements (no parent system requirement)"
      - "No suspect links (requirement changed after test case baselined)"
    output:
      format: "json"
      destination: "qa_dashboard"
      alert_threshold: "any gap found"

  review_quality_checker:
    trigger: "merge_request_merged"
    checks:
      - name: "Minimum review time"
        rule: "Review duration >= 30 minutes for changes > 100 lines"
      - name: "Reviewer coverage"
        rule: "At least 2 reviewers including domain expert"
      - name: "Finding resolution"
        rule: "All review findings marked resolved or deferred with justification"
    output:
      format: "markdown"
      destination: "merge_request_comment"

Work Products

Common Quality Issues in AI-Assisted Development

Critical Awareness: AI-assisted development introduces new categories of quality issues that traditional QA processes may not detect. QA criteria and checklists must be explicitly updated to address these risks.

AI-Specific QA Checklist Extension

# Additional QA checklist items for AI-assisted development
ai_qa_checks:
  - id: AI-QA-001
    question: "Are AI-generated work products clearly identified and labeled?"
    evidence: "Provenance metadata in document properties or commit messages"

  - id: AI-QA-002
    question: "Has every AI-generated requirement been reviewed by a domain expert?"
    evidence: "Review record with explicit AI-content review confirmation"

  - id: AI-QA-003
    question: "Are AI tools used within their qualified scope and version?"
    evidence: "Tool qualification record; tool version log"

  - id: AI-QA-004
    question: "Is AI confidence level documented for AI-generated analyses?"
    evidence: "Confidence scores in analysis reports; thresholds defined in QA plan"

  - id: AI-QA-005
    question: "Are HITL override decisions documented with justification?"
    evidence: "HITL override log with rationale and approver"

  - id: AI-QA-006
    question: "Is the AI model/tool version recorded for reproducibility?"
    evidence: "Tool version in CI/CD logs; model version in analysis reports"

  - id: AI-QA-007
    question: "Are AI-generated test cases validated for relevance and coverage?"
    evidence: "Test case review record; coverage analysis report"

Implementation Checklist

Usage: This checklist supports incremental adoption of AI-powered QA. Organizations should target Level 1 items first, then progress to Level 2 and Level 3 as capability matures.

Level 1: Foundation (AI-Assisted)

Level 2: Integration (AI-Augmented)

Level 3: Optimization (AI-Automated with Oversight)

Summary

SUP.1 Quality Assurance:

• AI Level: L1-L2 (AI detects, human confirms)
• Primary AI Value: Automated compliance checking, continuous process monitoring, predictive deviation detection
• Human Essential: Finding validation, corrective action approval, quality gate decisions, audit sign-off
• Key Outputs: QA reports, non-conformance records, corrective action logs, quality metrics
• Focus: Process adherence, work product quality, AI-specific quality risks
• Tool Ecosystem: SonarQube, Axivion, custom AI QA agents, ALM integration
• HITL Principle: AI augments QA efficiency; humans retain all compliance decision authority