Continuous Training Pipeline

Published 2026-04-18·Updated 2026-10-29·13 min read

AI documentation guide

Overview

The Continuous Training Pipeline implements a Self-Improving Agent Loop that turns production inference records into candidate training data, evaluates the trained model, and determines whether to update it. It collects Langfuse OTel traces in an S3 Data Lake and uses a Reward Labeler to evaluate their quality. The prepared data is used for GRPO/DPO preference tuning.

Training and deployment require approval from the responsible team. A model that passes evaluation is first applied to a portion of traffic through a canary rollout. Check the results before increasing its traffic share.

Why Continuous Training

Production feedback can reveal tasks, terminology, and failure patterns that a static training dataset does not cover. Selected feedback can supply new training examples; evaluate the resulting model before deciding whether the update improves performance on those tasks.

The figures in the table below are not measurements but an illustrative scenario contrasting the two approaches. "Feedback Integration" refers to the time to promote a reviewed and approved model, not the data-capture latency. The Continuous Training "Cost" cell means only that manual labeling is removed; the automated Judge (LLM scoring) and data curation themselves still incur inference and operating cost.

Challenge	Traditional Approach (example)	Continuous Training (example)
Data Collection	Manual labeling (monthly)	Automatic trace collection (real-time)
Feedback Integration	3-6 months	~1 week after review and approval
Quality Improvement	Wait for new dataset	Integrate selected user feedback
Cost	Manual labeling cost	Automated Judge + curation cost (replaces manual labeling)

Design Document Link

This document covers how to implement the 5-stage architecture of Self-Improving Agent Loop on EKS. Refer to the design document for background and strategic decisions.

ADR Agreement Required Before Production

To apply this pipeline to production traffic, the scope, automation boundaries, data gates, and rollback criteria defined in ADR — Self-Improving Agent Loop Decision must be agreed upon at the organizational level. Operate the Train/Deploy stages with manual approval gates.

5-Stage Pipeline Flow

Expand the diagram to read labels and follow connections.

Diagram

Key Concepts:

1. Trace → Dataset: Convert Langfuse production inference logs into training data
2. Reward Labeling: Score trace quality 0-1 with Ragas + LLM Judge
3. Preference Tuning: Choose between two paths by situation. DPO pairs preferred/non-preferred responses to the same prompt and raises the probability of the preferred one. GRPO generates several responses (a group) to the same prompt and trains the policy on advantages normalized by the relative reward within the group. For the detailed contracts and code, see GRPO/DPO Training Job.
4. Eval Gate: Verify quality threshold after training
5. Canary → 100%: Increase traffic in stages; when a regression is detected, the routing controller (kgateway/HTTPRoute) shifts traffic back to the previously approved stable model version. Define a measured recovery objective (detection-to-switch time); this traffic switch is separate from recording the version in the MLflow Registry. See Eval Gate · Registry · KPI for the concrete rollback contract.

Sub-Documents

Trace to Dataset

Export approved, redacted Langfuse generations as Parquet candidates and score them using explicit evaluator contracts. Conversion to GRPO/DPO inputs is a separate stage.

GRPO/DPO Training

Production configuration for running NeMo-RL (GRPO) and TRL (DPO) training jobs with labeled preference datasets on Karpenter Spot node pools and Volcano Gang Scheduling.

Evaluation & Rollout

Checkpoint evaluation, approval-gated canary promotion, Registry versioning, verified routing recovery and cost/quality KPI contracts.

• Trace → Dataset Materializer — Langfuse OTel collection, S3 Iceberg tables, Reward Labeler Fleet
• GRPO/DPO Training Job — NeMo-RL/TRL-based preference tuning with Karpenter Spot node pools
• Eval Gate · Registry · KPI — Threshold verification, Canary deployment, MLflow Registry, cost KPIs

Summary

The Continuous Training Pipeline automatically incorporates production feedback into model improvement through a 5-stage workflow:

1. Trace → Dataset: Langfuse OTel → S3 Iceberg (partitioned by date/model/consent)
2. Reward Labeling: Ragas + Qwen3-4B Judge Fleet (KServe + KEDA)
3. GRPO/DPO Training: NeMo-RL or TRL (Karpenter Spot p5en.48xlarge × 3 nodes)
4. Eval Gate: Threshold verification + Canary 5% → 25% → 100% (kgateway)
5. Registry & Rollback: MLflow + Agent Versioning + automatic rollback

Key Points:

• Cost Efficiency: Spot instances with bi-weekly iterations target about $4K/month. This is an author's estimate, not a measured result. The target workload, region, pricing date and cost boundary (training GPUs only, or also evaluation, storage and operations) are not defined, so recalculate it for your environment.
• Quality Improvement: The target is a 1% monthly faithfulness increase. Whether this 1% is a relative increase or percentage points, and which evaluation set measures it, are not defined. Define the fixed evaluation set used by the Eval Gate and the unit first so that monthly values can be compared.
• Safety: Eval Gate + gradual Canary + automatic rollback
• ROI: In an illustrative scenario, the goal is to increase revenue relative to training cost (varies by workload and quality improvement).

Next Steps

• Self-Improving Agent Loop — Design architecture and strategy
• Custom Model Pipeline — SFT training prerequisites
• Cascade Routing Tuning — Post-deployment routing optimization
• Agent Versioning — Model/code/prompt synchronization

References

Official Documentation

• NVIDIA NeMo Framework — Large-scale model training and RLHF
• HuggingFace TRL — DPO/PPO reference implementation
• MLflow — Model registry and version management
• Gateway API — Canary traffic splitting

Papers & Technical Blogs

• GRPO Paper (arxiv 2402.03300) — Group Relative Policy Optimization
• DPO Paper (arxiv 2305.18290) — Direct Preference Optimization

Related Documents

• Self-Improving Agent Loop
• ADR — Self-Improving Loop
• Ragas Evaluation

Production Checklist

• Enable Langfuse OTel trace collection (add user_consent field)
• Configure S3 Data Lake + Glue Iceberg tables
• Deploy Reward Labeler Fleet (Qwen3-4B KServe + KEDA)
• Set up NeMo-RL or TRL training environment (Karpenter Spot node pool)
• Define Eval Gate thresholds (faithfulness >= 0.85)
• Configure Canary Deployment HTTPRoute + monitoring alerts
• Integrate MLflow Registry + Agent Versioning
• Automate rollback (Argo Rollouts)
• Build cost KPI dashboard (Grafana)
• Establish bi-weekly/monthly iteration schedule