Deployment Strategies — Shadow·Canary·A/B·Blue-Green

Published 2026-04-18·Updated 2026-10-29·15 min read

AI documentation guide

Model Replacement Strategies

Shadow Testing

Concept: The new model receives actual production traffic, but responses are not delivered to users. Only the old model's responses are returned, and new model outputs are collected only for logging/evaluation.

Figure 1. Request flow of Shadow Testing

Expand the diagram to read labels and follow connections.

Diagram

When to Use:

• To compare the new model's latency, error rate, and output quality on production requests without showing its responses to users
• When the additional cost is acceptable (the comparison here assumes 2x cost per request)

Implementation Example (Python, LiteLLM): The LiteLLM Router and Proxy provide traffic mirroring, enabled by setting silent_model in a deployment's litellm_params (as of 2026-10-29). It runs the shadow call in the background and tags it with is_silent_experiment metadata. The example below implements the same pattern when you call acompletion directly without the Router:

import asyncio
from litellm import acompletion

_background_tasks = set()

async def shadow_call(user_request):
    # Old model (production)
    old_task = asyncio.create_task(acompletion(model="gpt-4", messages=user_request))
    # New model (shadow)
    new_task = asyncio.create_task(acompletion(model="claude-3-5-sonnet-20241022", messages=user_request))

    # Wait only for the old model; its errors propagate to the caller unchanged
    try:
        old_resp = await old_task
    except Exception:
        new_task.cancel()
        raise

    # Logging: collect the shadow result in the background and compare both responses (a shadow failure does not affect the user response)
    async def log_shadow():
        try:
            new_resp = await new_task
        except Exception as exc:
            new_resp = exc
        log_to_langfuse(user_request, old_resp, new_resp, shadow=True)

    task = asyncio.create_task(log_shadow())
    _background_tasks.add(task)
    task.add_done_callback(_background_tasks.discard)

    # Return only old model response to user
    return old_resp

Advantages:

• Users receive only the old model's response. The example above waits only for the old model and records the shadow result or error in the background, so the shadow call neither holds the response nor makes it fail. If the old model call fails, its error reaches the caller unchanged.
• Testing with actual traffic patterns

Disadvantages:

• 2x cost
• Cannot collect user feedback (users don't see shadow responses)

---

Canary Rollout

Concept: Start with a small percentage of traffic (5%) and gradually increase the ratio.

5% → Observe (24h) → If no issues 25% → 50% → 100%

When to Use:

• When the new model is sufficiently validated, but full production replacement is high risk
• When fast rollback is needed upon regression detection

Implementation Example (LaunchDarkly): Control model selection with Feature Flag

import ldclient
from ldclient import Context
from ldclient.config import Config

ldclient.set_config(Config("your-key"))
ld_client = ldclient.get()

def get_model_for_user(user_id: str):
    context = Context.builder(user_id).kind("user").build()
    model = ld_client.variation("llm-model-selection", context, default="gpt-4")
    return model

# Set "llm-model-selection" flag in LaunchDarkly console to 5% claude-3-5-sonnet, 95% gpt-4

Monitoring Criteria:

• Canary group vs Control group success rate (200 response ratio)
• Latency P50/P99 difference
• User feedback (thumbs up/down) ratio
• Cost (token usage)

Regression Alert (detection only):

# Example: Prometheus AlertManager rule
- alert: CanaryRegressionDetected
  expr: |
    (rate(llm_success_total{model="claude-3-5-sonnet"}[5m]) 
     / rate(llm_requests_total{model="claude-3-5-sonnet"}[5m]))
    < 0.95
  for: 10m
  annotations:
    summary: "Canary success rate below 95%, rollback needed"

This rule only detects the regression and fires an alert; it does not perform the rollback. The actual rollback is a separate action (for example a webhook or runbook) wired to this alert.

Advantages:

• Progressive risk distribution
• Can collect real user feedback

Disadvantages:

• Longer deployment period (days to weeks)
• Monitoring infrastructure required

Canary Implementation for EKS Model Serving

This document covers the general principles and selection criteria of deployment strategies. For kgateway HTTPRoute weight-based Canary implementation in EKS model serving, Eval Gate integration, Argo Workflows automation, and MLflow Registry rollback, see Eval Gate · Registry · KPI.

---

A/B Testing

Concept: Randomly split traffic into two groups (A: old model, B: new model) and statistically compare business metrics (conversion rate, user satisfaction, etc.).

When to Use:

• When you need to prove "Is the new model really better?" with statistical significance
• Marketing, UX optimization (prompt tone changes, etc.)

Experiment Design:

1. Null Hypothesis: "No performance difference between new and old models"
2. Alternative Hypothesis: "New model improves conversion rate by 5%p (percentage points) or more"
3. Sample Size Calculation: AB Test Calculator
   Example: Baseline 10%, detect 5%p improvement, 80% power → 2,348 per group needed
4. Experiment Duration: Until sufficient samples are collected (typically 1-4 weeks)

Implementation Example (Unleash):

import { Unleash } from 'unleash-client';

const unleash = new Unleash({
  url: 'https://unleash.example.com/api',
  appName: 'agent-service',
  customHeaders: { Authorization: 'your-token' }
});

function selectModel(userId: string): string {
  const context = { userId };
  // 'ab-test-claude-vs-gpt' variant: 50% 'A', 50% 'B'
  const variant = unleash.getVariant('ab-test-claude-vs-gpt', context);
  return variant.name === 'B' ? 'claude-3-5-sonnet-20241022' : 'gpt-4';
}

Analysis: Validate significance with chi-square test after experiment completion

from scipy.stats import chi2_contingency

# A: gpt-4, B: claude-3-5-sonnet
# Success/failure contingency table
obs = [[2100, 300],   # A: 2100 success, 300 failure
       [2200, 200]]   # B: 2200 success, 200 failure

chi2, p, dof, ex = chi2_contingency(obs)
print(f"p-value: {p}")  # p < 0.05 → B is statistically significantly superior

Advantages:

• Prove business impact with numbers
• Favorable for marketing, executive persuasion

Disadvantages:

• Long experiment period
• Statistical expertise required
• Sufficient traffic needed to ensure significance

---

Blue-Green Deployment

Concept: Operate old environment (Blue) and new environment (Green) simultaneously, then switch traffic to Green all at once. Immediately revert to Blue if issues occur.

When to Use:

• When replacing the model serving infrastructure itself (vLLM 0.5 → 0.6)
• Runtime changes rather than prompt changes

Implementation Example (Kubernetes Service + Ingress):

Prerequisites: each vLLM pod needs a GPU node with the NVIDIA device plugin and matching drivers installed, and the model weights must be available (downloaded at startup or mounted). The manifests below request one GPU per pod and use a readiness probe so traffic is sent only after the model has loaded.

# blue-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: llm
      version: blue
  template:
    metadata:
      labels:
        app: llm
        version: blue
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.5.4
        args: ["--model", "meta-llama/Llama-3.1-8B-Instruct"]
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            nvidia.com/gpu: 1
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 10
---
# green-deployment.yaml (new version)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: llm
      version: green
  template:
    metadata:
      labels:
        app: llm
        version: green
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.6.3
        args: ["--model", "meta-llama/Llama-3.1-8B-Instruct"]
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            nvidia.com/gpu: 1
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 10
---
# service.yaml (initially points to blue)
apiVersion: v1
kind: Service
metadata:
  name: llm-service
spec:
  selector:
    app: llm
    version: blue  # ← Change this to 'green' to switch
  ports:
  - port: 8000

Switch Procedure:

1. Complete Green deployment → Check health check
2. kubectl patch svc llm-service -p '{"spec":{"selector":{"version":"green"}}}'
3. Keep Blue running during the monitoring window so it remains available as the rollback target
4. If issues occur, immediately rollback with version: blue
5. Delete Blue only after the monitoring/soak window closes with no issues

Advantages:

• Fastest rollback speed (seconds)
• Simple switch process

Disadvantages:

• 2x infrastructure cost (during switch period)
• No progressive validation (all-or-nothing)

---

Feature Flag-Based Prompt Rollout

LaunchDarkly

LaunchDarkly is an enterprise-grade Feature Flag platform.

Prompt Rollout Example:

import ldclient
from ldclient import Context
from ldclient.config import Config

ldclient.set_config(Config("sdk-key"))
ld_client = ldclient.get()

def get_prompt_version(user_id: str, org_id: str) -> int:
    context = Context.builder(user_id) \
        .kind("user") \
        .set("org_id", org_id) \
        .build()

    # flag 'prompt-version-financial': organization-level targeting
    # Example: org_id='acme-corp' → version=5, others → version=4
    version = ld_client.variation("prompt-version-financial", context, default=4)
    return version

Kill Switch: Revert all users to safe version in emergency situations

# Force 'prompt-version-financial' flag to 4 in LaunchDarkly console
# Applied to all users within the SDK's flag-propagation interval (near-real-time), without code changes

Targeting Rule Examples:

• Beta users: user.beta == true → new version
• Specific region: user.region == "us-east-1" → canary version
• Organization tier: user.tier == "enterprise" → latest version priority

---

Unleash

Unleash is an open-source Feature Flag platform.

Advantages:

• Self-hosted capable
• Postgres backend, RBAC, audit log provided by default

Prompt Rollout:

import { Unleash } from 'unleash-client';

const unleash = new Unleash({
  url: 'https://unleash.internal.corp/api',
  appName: 'agent-gateway',
  customHeaders: { Authorization: 'token' }
});

const defaultPrompt = 'v4';  // Safe fallback prompt version

function getPromptVariant(userId: string): string {
  const context = { userId, properties: { region: 'us-west-2' } };
  const variant = unleash.getVariant('prompt-experiment-2026-04', context);
  // variant.name: 'control', 'treatment-A', 'treatment-B'
  // A disabled flag or a variant with no payload must fall back to the default prompt
  if (!variant.enabled || !variant.payload) return defaultPrompt;
  return variant.payload.value;  // Actual prompt text or version number
}

---

AWS AppConfig

AWS AppConfig supports Feature Flags and dynamic configuration.

Advantages:

• AWS native, integrates with Lambda/ECS/EKS
• Deployment strategy: Linear, Canary, All-at-once
• CloudWatch alarm-based automatic rollback

Example:

import boto3
import json
import time

appconfig = boto3.client('appconfigdata')

session = appconfig.start_configuration_session(
    ApplicationIdentifier='agent-app',
    EnvironmentIdentifier='production',
    ConfigurationProfileIdentifier='prompt-config'
)
session_token = session['InitialConfigurationToken']

prompt_config = None
# Each GetLatestConfiguration returns a new token that must be used for the next poll.
# An empty 'Configuration' means "no change"; keep the last config in that case.
while True:
    response = appconfig.get_latest_configuration(ConfigurationToken=session_token)
    session_token = response['NextPollConfigurationToken']
    data = response['Configuration'].read()
    if data:
        prompt_config = json.loads(data)
        print(prompt_config['version'])  # Example: 5
        print(prompt_config['text'])
    # else: configuration unchanged; keep prompt_config
    time.sleep(response.get('NextPollIntervalInSeconds', 60))

Deployment Strategy:

{
  "DeploymentStrategyId": "AppConfig.Canary10Percent20Minutes",
  "Description": "Exponential growth with a 10% step over a 20-minute deployment window, then a final bake time"
}

Automatic rollback when CloudWatch alarm (LLMErrorRate > threshold) fires.

---

Deployment Strategy Comparison

These ratings are qualitative and typical; the actual values vary by implementation (infrastructure, traffic volume, automation).

Table 1. Comparison of deployment strategies by risk, validation speed, cost, and rollback speed

Strategy	Risk	Validation Speed	Cost	User Feedback	Rollback Speed
Shadow	Minimal	Fast	2x	Not possible	N/A
Canary	Low	Medium	1x	Possible	Fast (minutes)
A/B	Medium	Slow	1x	Possible	Medium (hours)
Blue-Green	High	Fast	2x (during switch)	Possible	Very fast (seconds)

Selection Guide:

• Initial validation: Shadow → Canary 5%
• Business impact measurement: A/B Testing
• Infrastructure replacement: Blue-Green
• Emergency rollback needed: Blue-Green + Canary combination

---

References

Feature Flag Platforms

• LaunchDarkly: launchdarkly.com
  • AI/ML related posts
• Unleash: getunleash.io
• AWS AppConfig: AWS Documentation

Deployment Strategies

• Canary Deployment Pattern: martinfowler.com/bliki/CanaryRelease.html
• Blue-Green Deployment: martinfowler.com/bliki/BlueGreenDeployment.html
• Shadow Testing: Google SRE Workbook - Canarying Releases

Statistical Testing

• A/B Test Calculator: evanmiller.org/ab-testing
• scipy.stats: docs.scipy.org/doc/scipy/reference/stats.html

Related Documents (Internal)

• Eval Gate · Registry · KPI — kgateway HTTPRoute-based Canary implementation, Eval Gate, and MLflow rollback (canonical for model evaluation gates)
• Governance & Automation — Automatic regression detection and rollback system

---

Next Steps

Once you've selected a deployment strategy:

1. Governance & Automation — Build automatic regression detection and rollback system
2. Prompt & Model Registry — Build version control system
3. Agent Monitoring — Build real-time observability