Skip to main content

Harness Engineering

22 min read
AI documentation guide
Extension concept (engineering-playbook's own content)

Ontology and harness engineering are not part of the official AWS Labs AIDLC methodology; they are engineering-playbook's own extension content. They reflect DDD and 2026 agentic-AI best practices to strengthen enterprise AI reliability. When applying official AIDLC, this axis can be adopted optionally.

"Agents aren't hard, harnesses are" — NxCode, 2026

Overview​

Harness engineering builds the checks and execution limits around agent work. It turns domain constraints from the ontology into validation rules and adds controls such as retry budgets, timeouts, output gates, and circuit breakers. The 2026 quote below illustrates why harness design matters and comes from one reported project:

When OpenAI Codex generated 1 million lines of code, human engineers wrote 0 lines (an illustrative figure from one reported case, not a measurement representative of every setting). The engineer's role has shifted from writing code to designing harnesses.

Expand the diagram to read labels and follow connections.
Diagram

Role of Harness:

  • Architecturally validate constraints defined by ontology
  • Design retry budgets, timeouts, output gates, circuit breakers
  • Ensure independent validation (code generation agent ≠ validation agent)
  • Feed validation results back to ontology evolution

Harness vs Guardrails​

Many teams use "guardrails" and "harness" interchangeably, but they are fundamentally different in scope and timing.

AspectGuardrailsHarness
ScopeRuntime input/output filtering plus dialog, retrieval, and tool-execution constraintsEntire architecture design
RoleBeyond input/output filtering (PII masking, prompt injection defense): constraints at the dialog, retrieval, and tool-execution stagesRetry budgets, timeouts, output gates, circuit breakers
TimingDuring executionFrom design time
Failure ModeBlock individual requestsProtect entire system
ExamplesBedrock Guardrails, NeMo GuardrailsAIDLC Quality Gates, Independent validation agents

Key Difference:

  • Guardrails are "controls that filter risk not only at input/output but also at the dialog, retrieval, and tool-execution stages" — e.g., prompt injection detection, PII masking, tool-call restriction
  • Harness is "architecture that constrains AI to operate safely" — e.g., preventing 847 retries, blocking cost runaway

Fintech Runaway Case: AI Failure Without Harness​

Hypothetical Case: $2,200 Loss Scenario

The figures in this box are illustrative figures (a hypothetical scenario, not measured data). No source postmortem could be found (as of 2026-10-29), so do not cite it as a real incident.

Suppose a fintech startup's AI agent executed 847 API retries in a single loop; the result would be:

  • $2,200 in LLM API costs
  • 14 incomplete emails sent to customers
  • 3 hours of service outage (manual intervention required)

Root Cause Analysis:

  • ❌ Not a model problem (used GPT-4)
  • ❌ Not a prompt problem (prompts were clear)
  • ✅ Architectural failure — Absence of harness

Missing Harness Patterns:

  1. No retry budget (unlimited retries)
  2. No timeout (loop could run infinitely)
  3. No output gate (failed to block incomplete emails)
  4. No circuit breaker (continued attempting after 847 failures)
  5. No cost limit (no alerts until $2,200 charge)

Lesson: The example illustrates why retries, execution time, outputs, and cost need explicit limits. These controls bound the consequences of a failed agent action while the underlying model, prompt, or integration problem is investigated.

Harness Patterns Across AIDLC 3 Phases​

PhaseHarness TypeValidation TargetImplementation Method
InceptionSpec validation harnessRequirements completeness, conflicts, NFR complianceOntology-based automatic Spec consistency validation
ConstructionBuild/test harnessCode accuracy, security, architecture complianceIndependent agent review + ontology violation detection
OperationsRuntime harnessAI Agent behavior constraints, cost limitsCircuit breakers, retry budgets, output gates
Expand the diagram to read labels and follow connections.
Diagram

Harness Pattern Catalog​

1. Circuit Breaker​

Purpose: Block additional attempts on repeated failures to protect entire system

Pattern:

circuit_breaker:
  failure_threshold: 5        # Open after 5 failures
  timeout: 60s               # Half-Open after 60 seconds
  success_threshold: 2       # Closed after 2 successes

Application Cases:

  • LLM API calls (Bedrock, OpenAI)
  • External service integration (payment, email)
  • Agent-to-agent communication (multi-agent systems)

2. Retry Budget​

Purpose: Prevent unlimited retries, control overall costs

Pattern:

retry_budget:
  max_attempts: 3            # Maximum 3 retries per single operation
  backoff: exponential       # Exponential backoff
  max_backoff: 30s          # Maximum 30 second wait
  budget_limit: 10          # Hourly cap on total retries

max_attempts caps how many times one failed operation is retried, while budget_limit caps the total number of retries allowed per hour across operations. The two values bound different scopes, so they are used together. A cost ceiling is enforced not by retry counts but by the amounts in the Cost Limit pattern (section 7 below).

Fintech Runaway Prevention (with the configuration above):

  • ✅ 847 retries → Limited to 3 per operation
  • ✅ Immediate retries → Exponential backoff (1s, 2s, 4s)
  • ✅ $2,200 cost → Blocked by the Cost Limit pattern's $10/hour and $100/day ceilings

3. Timeout​

Purpose: Prevent infinite loops, ensure response time

Pattern:

timeout:
  request: 30s              # Single request timeout
  total: 300s               # Total operation timeout
  idle: 60s                 # Idle timeout on no response

Application Cases:

  • LLM inference (stop waiting after 30 seconds)
  • Code generation (5 minute total limit)
  • Test execution (stop waiting after 1 minute idle)

A timeout only bounds how long the caller waits; by itself it does not stop in-flight work or reverse side effects that have already occurred. To actually halt the work you need an explicit cancellation signal, and to undo data already written you need separate compensation handling, both designed alongside the timeout.

4. Output Gate​

Purpose: Block incomplete or harmful outputs

Pattern:

output_gate:
  validators:
    - syntax_check          # Code syntax validation
    - schema_validation     # JSON schema validation
    - pii_detection        # PII detection and masking
    - toxicity_filter      # Harmful content filtering
    - completeness_check   # Required-field and body completeness (needs separate implementation)
  action_on_failure: reject # Reject output on failure

The syntax_check, schema_validation, pii_detection, and toxicity_filter validators above check only syntax, schema, sensitive data, and toxicity, so they cannot judge whether an email body is incomplete in business terms. To block incomplete emails you must implement a separate validator such as completeness_check that explicitly checks required fields and body-completion conditions, and add it to the output gate.

Fintech Runaway Prevention:

  • ✅ 14 incomplete emails → Blocked at the output gate once a completeness validator is added
  • ✅ Prevent PII exposure
  • ✅ Filter harmful content

5. PII Masking​

Purpose: Protect sensitive information (both input and output)

Pattern:

pii_masking:
  patterns:
    - email: "***@***.***"
    - ssn: "***-**-****"
    - credit_card: "****-****-****-****"
  redact_in_logs: true      # Mask in logs as well

6. Prompt Injection Defense​

Purpose: Block malicious prompts

Pattern:

prompt_injection_defense:
  techniques:
    - instruction_hierarchy  # System prompt priority
    - delimiter_isolation    # Isolate user input with delimiters
    - output_validation     # Output schema validation

7. Cost Limit​

Purpose: Prevent LLM API cost runaway

Pattern:

cost_limit:
  per_request: 0.50         # $0.50 limit per request
  per_hour: 10.00          # $10 limit per hour
  per_day: 100.00          # $100 daily limit
  alert_threshold: 0.80    # Alert at 80% threshold

The YAML for patterns 5–7 (PII masking, prompt injection defense, cost limit) only declares configuration keys and thresholds; it does not act on its own. These settings take effect only when backed by an enforcing implementation: a masking engine that actually redacts, a detector that identifies injection attempts, and metering that aggregates usage to block or alert at the thresholds.

Quality Gates — End-to-End Quality Assurance​

Human validation in AI-DLC is a Loss Function — it catches errors early at each stage to prevent downstream propagation. Quality Gates systematize this Loss Function.

Inception          Construction          Operations
    │                   │                    │
    ▼                   ▼                    ▼
[Mob Elaboration    [DDD Model         [Pre-deployment
 Output Validation]  Validation]        Validation]
    │                   │                    │
    ▼                   ▼                    ▼
[Spec Consistency]  [Code + Security   [SLO-based
                     Scanning]          Monitoring]
    │                   │                    │
    ▼                   ▼                    ▼
[NFR Compliance]    [Test Coverage]    [AI Agent Response
                                        Validation]

🛡️ Quality Gates

AI-Driven Multi-Layer Validation

AI Code Review Validation Items

Code Quality
Q Developer, Copilot
Coding standards, Complexity, Duplication
Security Scan
Q Developer Security
OWASP Top 10, Secret Detection
IaC Validation
tflint, OPA
Terraform Best Practices, Policy Compliance
K8s Validation
Kube-linter, Datree
Security Context, Resource Limits
Test Coverage
Go test, pytest
Minimum 80% Coverage
Performance Regression
k6, Artillery
Latency, Throughput Benchmarks

Auto-Approval Criteria

Condition
Auto Approve
Manual Review Required
Security Scan Results
✓ 0 Critical/High
⚠ ≥1 Critical/High
Test Coverage
✓ ≥ 80%
⚠ < 80%
K8s Validation
✓ 0 Warnings
⚠ ≥1 Warning
Performance Regression
✓ P99 < SLO Target
⚠ P99 ≥ SLO Target
Change Scope
✓ < 500 lines
⚠ ≥ 500 lines
Missing Scan Data
✓ N/A
⚠ Default when a scan did not run or returned no data
Note: AI code review excels at detecting pattern-based issues, but business logic accuracy and architectural suitability require human judgment. We recommend a hybrid approach: use AI review as a first filter, with humans performing final review of critical changes.

AI-Based PR Review Automation​

Traditional code review relies on lint rules and static analysis, but AI-based review validates architecture patterns, security best practices, and business logic consistency.

Validation Items​

1. DDD Pattern Compliance

  • Detect Aggregate encapsulation violations
  • Prevent direct Entity modifications
  • Validate Value Object immutability

Example:

// ❌ Violation: Direct Entity modification outside Aggregate
func UpdateUserEmail(userID string, email string) error {
    user, err := userRepo.FindByID(userID)
    if err != nil {
        return err
    }
    user.Email = email  // ❌ Direct Entity modification
    return userRepo.Save(user)
}

// ✅ Recommended: Modify through Aggregate method, then persist
func UpdateUserEmail(userID string, email string) error {
    user, err := userRepo.FindByID(userID)
    if err != nil {
        return err
    }
    if err := user.ChangeEmail(email); err != nil {  // ✅ Use Aggregate method
        return err
    }
    return userRepo.Save(user)  // ✅ Persist the change to the repository
}

2. Microservice Communication

  • Synchronous calls: Use gRPC
  • Asynchronous events: Use SQS/SNS
  • External APIs: HTTP REST (OpenAPI spec required)

3. Observability

  • OpenTelemetry instrumentation in all handlers
  • Expose business metrics as Prometheus custom metrics
  • Structured logging (JSON format, include contextual fields)

4. Security

  • Authentication: JWT (this project's policy requires RS256 and prohibits HS256. HS256 is also a valid algorithm; choosing asymmetric signing is an organizational policy to reduce the risk of a leaked shared secret.)
  • Sensitive information: Retrieve from AWS Secrets Manager
  • SQL queries: Use Prepared Statements (prohibit string concatenation)

Independent Validation Principle​

The Same Agent's Test Trap

"Tests written by the same AI agent cannot catch that agent's errors" — This is like AI grading its own homework.

Symptoms:

  • Tests pass ✅
  • CI is green ✅
  • PR merge complete ✅
  • 3 days later, feature half-working ❌

Cause:

  • Tests optimized for 'completion' not 'accuracy'
  • Code generation agent and test generation agent are identical
  • Share the same bias

Solution: Independent Validation Harness

Expand the diagram to read labels and follow connections.
Diagram

Implementation Principles:

  1. Use different agents — Code generation ≠ Test generation
  2. Use different models — GPT-4 generation → Claude validation
  3. Human validation — Humans give final approval for core logic

Switching the validator to a different agent or model reduces the bias it shared with the generator. A different validator is not by itself a guarantee of correctness, however. Independent validation holds only when there is an independent test oracle (a basis for the expected result) or specification separate from the generation process; swapping the model alone can still miss errors when both models share the same misunderstanding.

Harness Design Checklist​

Harness pattern checklist for AIDLC 3 phases:

Inception Harness​

  • Automatic Spec consistency validation
  • NFR compliance check
  • Ontology violation detection
  • Requirements conflict inspection

Construction Harness​

  • Build automation (CI/CD)
  • Test coverage 80%+
  • Security scanning (SAST, SCA)
  • Architecture pattern validation
  • Independent agent review
  • Quality Gates pass

Operations Harness​

  • Circuit breaker implementation
  • Retry budget configuration
  • Timeout definition
  • Output gate activation
  • PII masking
  • Prompt injection defense
  • Cost limit configuration
  • SLO-based alerting

References​

External References​