Reference Architecture
Implementation guides for platform engineers deploying and operating the Agentic AI Platform. Choose the relevant area, then check its environment requirements and validation steps. Design decisions are covered in the platform architecture.
Table 1. Focus, audience, and deliverables of Documentation vs Reference Architecture
| Aspect | Documentation | Reference Architecture |
|---|---|---|
| Focus | Architecture concepts, design principles, technology comparison | Production deployment procedures, manifests, verification |
| Audience | Decision makers, architects | Platform engineers, DevOps |
| Deliverables | Architecture documents, decision records | Deployable YAML, scripts, checklists |
| Update Cadence | On design changes | As deployment/operations experience accumulates |
Documents
Inference Gateway Deployment
4 itemsModel Lifecycle
5 itemsIntegrations & Cost
4 itemsPrerequisites
Prerequisites for deploying the Reference Architecture.
AWS Account and Permissions
- EKS cluster creation permissions (IAM, VPC, EC2, EKS)
- GPU instance Spot quotas (p5en.48xlarge: 192+ vCPUs)
- S3 bucket creation permissions
- AMP/AMG creation permissions (for monitoring setup)
- ECR registry creation permissions (for custom image builds)
Tools
Check the tool versions and cluster compatibility specified by the deployment chapter. Record installed versions with the validation results.
Table 2. Tools used for deployment and their purpose
| Tool | Purpose |
|---|---|
eksctl, aws CLI | Cluster and AWS resource management |
kubectl, helm | Kubernetes resource and chart management |
| Container build tool | Custom image builds |
s5cmd | S3 model-file transfer used by the examples |
Networking
- Public subnets: For NLB deployment (when coding tools need external access)
- Private subnets: For GPU nodes, vLLM, Bifrost deployment
- NAT Gateway: For S3, ECR, HuggingFace Hub access
- VPC Endpoints (recommended): S3, ECR, AMP
Platform Architecture
The diagram shows six runtime layers, three cross-cutting planes, the Knowledge Feature Store, and serving/fine-tuning paths. The six deployment areas below describe implementation stages, not matching runtime-layer numbers. The Qwen3-32B label on the large-model serving box in the diagram is an example of the large-model serving role (the largest dense model in the official Qwen3 list, which runs from 0.6B to 32B dense plus the 30B-A3B and 235B-A22B MoE models), not a recommendation. Choose the actual model and revision for your deployment environment.
Open in draw.io — Opens without login for viewing and editing. Use File → Save As to save a copy.
Architecture Overview
The diagram below shows the 6 areas of the Reference Architecture and the deployment sequence.
Figure 1. The six areas of the Reference Architecture and the deployment sequence
Deployment Sequence
The Reference Architecture is configured in the following order. Each phase depends on the outputs of the previous phase, so the order must be followed.
Phase 1: GPU Infrastructure Setup
Configure the EKS cluster and GPU node groups. Covers differences between Auto Mode and Standard Mode, and considerations when installing GPU Operator.
Table 3. Phase 1 GPU infrastructure setup items and details
| Item | Details |
|---|---|
| EKS Version | A version supported by the selected deployment guide and add-ons |
| Node Group | MNG or NodePool selected for model memory, SLOs, and available capacity |
| GPU Operator | Check pre-installed AMI components and avoid duplicate ownership |
| Monitoring Agents | DCGM Exporter, GFD, Node Status Exporter |
Phase 2: Model Deployment
Serve large open-source models with vLLM. Covers custom image building, S3 model caching, and considerations for multi-node deployment.
Table 4. Phase 2 model deployment items and details
| Item | Details |
|---|---|
| Serving Engine | vLLM (custom image) |
| Model Cache | S3 → s5cmd → node ephemeral storage (emptyDir). To back it with instance-store NVMe, configure the node to bootstrap/mount the NVMe and validate capacity (see note below) |
| Parallelism | Tensor Parallelism (single node recommended) |
| Validation | OpenAI-compatible API endpoint |
emptyDir uses the node's default ephemeral storage (the root disk) and does not automatically select instance-store NVMe. To back the cache with NVMe, format and mount the instance-store NVMe during node bootstrap, configure kubelet's --root-dir (or the emptyDir path) to use that mount, and then validate the capacity against the model size. Without this, the model cache is bound by the root disk capacity.
Phase 3: Inference Gateway
Configure the 2-Tier inference gateway based on kgateway + Bifrost/LiteLLM. Includes Complexity-based Cascade Routing, Semantic Caching, and Guardrails.
Table 5. Phase 3 inference gateway items and details
| Item | Details |
|---|---|
| L1 Gateway | kgateway (Gateway API, mTLS, rate limiting) |
| L2-A Gateway | Bifrost (CEL Rules conditional routing, failover) or LiteLLM (native complexity-based routing) |
| Load Balancer | NLB (TCP/TLS) |
| Routing Strategy | Complexity-based Cascade (SLM → LLM), Hybrid Routing, Fallback |
Phase 4: Monitoring and Observability
Configure the monitoring stack based on Prometheus + AMP + AMG + Langfuse.
Table 6. Phase 4 monitoring and observability items and details
| Item | Details |
|---|---|
| Metrics Collection | Prometheus → AMP (Pod Identity authentication) |
| Dashboards | AMG Grafana (SigV4 ec2_iam_role) |
| LLM Observability | Langfuse (OTel traces, cost tracking) |
| GPU Metrics | DCGM Exporter (GPU utilization, VRAM, temperature) |
Phase 5: Pipelines
Configure LoRA Fine-tuning and Cascade Routing pipelines.
Table 7. Phase 5 pipeline items and details
| Item | Details |
|---|---|
| Fine-tuning | LoRA adapter training → S3 storage → vLLM hot-reload |
| Cascade Routing | SLM (8B) → LLM (744B) cost optimization |
| Evaluation | Ragas + custom benchmarks |
Phase 6: Coding Tool Integration
Connect AI coding tools such as Aider and Cline to self-hosted models.
Table 8. Phase 6 coding tool integration items and details
| Item | Details |
|---|---|
| Coding Tools | Aider, Cline, Continue.dev |
| Protocol | OpenAI-compatible API |
| Connection Path | Coding tool → NLB → kgateway → Bifrost/LiteLLM → vLLM |
| Monitoring | Bifrost/LiteLLM OTel → Langfuse (per-request tracing) |
Core Design Principles
The Reference Architecture follows these principles.
1. Single-Node First
Multi-node distribution significantly increases complexity and failure potential. Prioritize selecting instances with sufficient VRAM (p5en, p6) to serve with Tensor Parallelism only on a single node.
2. Spot Instance Utilization
Choose Spot according to interruption tolerance as well as price. Inference servers still need image and weight loading, engine initialization, and handling for in-flight requests; recovery is not instantaneous. Validate replacement capacity, ready replicas, drain/retry policy, and actual cold-start time. Review the real-time workload considerations in EKS Compute and Autoscaling.
3. Standard Toolchain
Use standard tools from the CNCF and Kubernetes ecosystem wherever possible.
Table 9. Standard tools and alternatives by area
| Area | Standard Tool | Alternative |
|---|---|---|
| Node provisioning | Karpenter / MNG | Auto Mode NodePool |
| Model Serving | vLLM | SGLang, llm-d |
| AI Gateway | Bifrost / LiteLLM | OpenClaw, Helicone |
| Metrics | Prometheus + AMP | CloudWatch |
| LLM Observability | Langfuse | Helicone, LangSmith |
| Multi-node serving deployment | LeaderWorkerSet (LWS) | KubeRay |
The roles in the table are distinct. Karpenter and MNG provision nodes, which differs from the GPU Pod scheduling that places GPU Pods onto nodes. LeaderWorkerSet (LWS) is an API for inference serving deployment across multiple nodes (multi-host serving), not a distributed-training orchestration tool. The AI gateway alternative OpenClaw is an agent-assistant gateway that assumes a trusted operator/team boundary, so confirm its scope before treating it as a drop-in replacement for a general multi-provider inference-governance proxy.
4. Layered Cost Optimization
Effects depend on traffic, cache-hit rate, quality criteria, and operating hours. Do not add percentage savings across techniques. Compare total cost per successful request and SLOs under the same conditions.
Figure 2. Cost optimization review sequence
Next Steps
For concepts and architecture design, refer to the following documents:
- Agentic AI Platform Architecture — Overall design principles and component structure
- GPU Resource Management — Karpenter, KEDA, DRA-based GPU autoscaling
- vLLM Model Serving — vLLM architecture and optimization techniques
- Inference Gateway Routing Strategy — 2-Tier architecture and Cascade/Semantic/Hybrid Routing design
The verification environment of the examples and your actual production environment can differ. Record the reproduction conditions together with the verification results, and please open an issue for anything that needs correction.