Open-Weight Model Deployment Guide
Overview
This document is a decision guide for customers evaluating whether to self-host open-weight large language models in place of commercial APIs. It centers on two value drivers. The first is Token Economics — the break-even between per-token pricing of commercial APIs and the fixed-cost structure of self-hosting. The second is Data Sovereignty — regulatory requirements that data and inference must not leave controlled regions and VPC boundaries. The intended audience is architects and decision makers in regulated industries such as finance and the public sector.
This document focuses on decisions and trade-offs. For the higher-level platform-selection framework, see AI Platform Decision Framework; for the detailed derivation of cost figures, see Coding Tools Cost Analysis; and for the deployment implementation procedure, see Custom Model Deployment Guide.
Background: Why Open-Weight Now
Open-weight models are models whose weights are publicly released, allowing inference on customer-controlled infrastructure. Entering 2026, both licensing and performance have changed the premises for the self-hosting decision.
- Spread of permissive licenses: Many flagship open-weight models are released under MIT or Apache 2.0. For example, Z.ai's GLM-5.2 was released under the MIT license (Z.ai blog). This is distinct from the era when license constraints were an obstacle to self-hosting and commercial use.
- Approaching frontier-grade performance: Open-weight models' coding and reasoning benchmarks have closed the gap with commercial models. However, benchmark numbers vary by source and evaluation conditions, so they must be validated against customer workloads before adoption.
- Limits of commercial-API dependence: Per-token pricing scales costs linearly as volume grows. In addition, prompts and outputs traverse external services, making data sovereignty requirements difficult to satisfy.
Model specifications and versions change quickly. The model examples in this document are based on publicly available information as of the time of writing (2026-06); model cards and inference-engine compatibility must be reconfirmed before adoption.
Pillar 1 — Token Economics
Differences in Pricing Structure
Commercial APIs and self-hosting incur cost in different ways.
Table 1. Pricing structure: commercial API vs self-hosting
| Aspect | Commercial API | Self-hosting (EKS + vLLM) |
|---|---|---|
| Cost model | Per-token (metered) | Per-GPU-hour fixed cost + per-request variable cost |
| Volume sensitivity | Scales linearly with usage | Fixed cost dominates within capacity; recalculate at each node-addition range |
| Initial cost | Service setup and integration cost | GPU and operations setup |
| Marginal cost | Incurred per request | Low within capacity, but not zero (power, operations, variable costs) |
Commercial APIs are often advantageous at low volume, while self-hosting can yield a lower effective per-token cost once a certain volume is exceeded. When a crossover exists, that point is the break-even; depending on unit price, quality, capacity, and variable costs, there may be no crossover at all.
Break-Even Analysis
Detailed break-even simulations (cost tables by monthly request volume; Cascade and Spot savings) are covered in Coding Tools Cost Analysis. Those break-even figures are a hypothetical example assuming specific unit prices and fixed costs, not a universal crossover that applies to every situation. Only the core baselines are summarized here.
- Under the linked example's assumptions, the break-even between commercial APIs and 24/7 self-hosting forms at the scale of several million requests per month. The actual crossover depends on unit price, quality, capacity, and variable costs, and may not exist at all.
- Cascade Routing (simple requests to SLM, complex requests only to large models) can lower the break-even volume when quality and latency are held equal. However, in a fixed-cost structure where both model pools run continuously, spend can instead increase, so verify against actual node hours.
- Operating only during business hours (8 hours/day) or using Spot instances can lower the fixed cost itself, but factor in model loading time, reserve capacity, and off-hours service requirements.
When applying this logic to open-weight models in general, the variables are model throughput and GPU requirements. Higher throughput on the same GPU budget yields a lower per-token cost.
Hidden Costs
The total cost of ownership (TCO) of self-hosting is not just the GPU bill. The decision must include:
- Engineering and operations staff: FTEs for deployment, upgrades, and incident response. Model swaps and inference-engine version management are ongoing.
- Cold start: Large models have weights of hundreds of GB. The first launch's download and load time affects availability.
- GPU idle: GPU cost when there is no traffic. EKS scale-to-zero can cut idle cost, but it conflicts with cold start, so a balance is needed.
For an AIDLC-perspective TCO framework, see Enterprise Cost Estimation.
Performance Is Effectively Unit Cost
In self-hosting, inference performance optimization directly lowers per-token unit cost. Higher throughput on the same GPU generates more tokens at the same cost. The main levers are:
- Quantization: FP8 and INT4 weights reduce required GPU memory and GPU count. However, quality loss is possible, so workload-specific validation is required. The credibility of official checkpoints versus community-quantized versions must be distinguished (see warning below).
- vLLM engine tuning: PagedAttention, FP8 KV cache, prefix caching, chunked prefill, and Multi-Token Prediction (MTP)-based speculative decoding raise throughput.
--gpu-memory-utilization,--max-num-batched-tokens,--max-num-seqs, and--max-model-lenare the key flags that tune concurrency and memory usage. For details on these concepts, see vLLM Model Serving. - Local NVMe storage: Using instance store (NVMe SSD) as a model cache speeds up cold start and KV offload. Availability depends on the node provisioning method (see the infrastructure section below).
- EFA (Elastic Fabric Adapter): In tensor/pipeline-parallel (TP/PP) distributed inference, EFA carries all inter-node communication, including collective, activation, and KV transfer. Physical bandwidth alone does not determine end-to-end KV transfer performance; the engine, topology, and scheduling also matter. For single-node inference, the intra-node interconnect governs performance, and whether NVLink/NVSwitch is present depends on the chosen instance, so do not assume every single node has it.
HuggingFace hosts many third-party quantized checkpoints. Using weights without verifying source, calibration, and licensing in production carries quality and security risks. Prefer the model provider's official quantized version when possible, and adopt community-quantized versions only after internal evaluation.
Pillar 2 — Data Sovereignty
Four Levels of Sovereignty Requirements
Data sovereignty requirements are a spectrum rather than a single threshold. The AI Platform Decision Framework breaks them into four levels.
Table 2. Inference location and recommended approach by sovereignty level
| Sovereignty Level | Inference Location | Recommended Approach |
|---|---|---|
| Public | No region constraint | Commercial API / AWS Native |
| In-country | Pinned to in-country region | Region enforcement + in-region managed service (e.g. Bedrock with in-region processing) or in-region self-hosting |
| Hybrid | On-premises + in-country | EKS Hybrid Nodes + self-hosting |
| Air-gapped | Fully isolated (disconnected) | EKS Anywhere or another disconnected-capable Kubernetes distribution + mirrored dependencies (managed EKS/Hybrid Nodes require AWS control-plane connectivity and are not air-gapped) |
The stronger the requirement, the lower the managed-service dependence and the larger the share of self-hosting and on-premises. However, a region-residency requirement such as "pinned to in-country region" must be distinguished from a requirement that the customer operate the infrastructure itself. Apply the concrete legal/contractual control requirements first; if an in-region managed service satisfies them (for example, Bedrock with in-region processing), the condition can be met without self-hosting. For SCP region enforcement and EKS Hybrid Nodes implementation, see Sovereign & Hybrid Deployment.
What Self-Hosting Lets You Control
Self-hosting open-weight models does not automatically "guarantee" sovereignty requirements. Deployment location alone does not remove external paths; the architecture can still include external clients, observability systems, embeddings/judges/fallbacks, and configured model-access paths. Treat the following as controls to implement and verify.
- In-boundary processing of data and inference (control): Map the end-to-end data path of prompts, outputs, and embeddings, and confirm there is no point where it leaves the controlled VPC/region.
- Blocking external paths (control): Set egress policy, authentication, and network boundaries so inference traffic does not leave, and explicitly review observability, fallback, and external-model-access paths.
- Auditability (control): Keep inference traces and access logs in in-house systems, and confirm with evidence that the boundaries hold in the actual deployment.
Application of Korean Regulations
Detailed controls mapping the e-Finance Supervisory Regulation, ISMS-P, SOC2, and ISO 27001 to AI operations are covered in Enterprise Compliance Framework. From a deployment-form perspective, the key points are:
- In-region self-hosting forms the infrastructure-level foundation that satisfies data-location and access-control requirements (e-Finance Supervisory Regulation Articles 15 and 17, ISMS-P 2.6).
- Self-hosting alone does not complete compliance. Implementation of control items such as access control, encryption, audit logs, and PII controls is also required.
Sovereignty Premium
Meeting sovereignty requirements incurs additional cost. Decisions must quantify these.
- Region premium: GPU instance unit prices differ by region. For example, using the illustrative unit prices in the infrastructure section below, the p5en.48xlarge On-Demand price in Tokyo (ap-northeast-1) is about 25% higher than in us-east-1. This ratio is calculated from those illustrative prices and is not a basis for current pricing. Region pinning may entail unit-price increases.
- Regulatory operations cost: Audit response, evidence production, and the staffing and time imposed by deployment constraints.
- Opportunity cost: The self-implementation burden incurred by giving up managed features such as Bedrock AgentCore and SageMaker.
Intersection of the Two Drivers — Decision Matrix
Placing token economics (monthly inference volume) and data sovereignty requirements on two axes organizes deployment forms into four quadrants.
Table 3. Deployment-form matrix by volume and sovereignty requirement
| Low Sovereignty Requirement | High Sovereignty Requirement | |
|---|---|---|
| Low Volume | Commercial API (fastest to start) | In-region self-hosting (small scale) or hybrid |
| High Volume | Hybrid (Cascade) or cost-driven self-hosting | In-region / air-gapped self-hosting |
Figure 1. Deployment-form decision across volume and sovereignty axes
Two principles guide interpretation:
- If sovereignty is a hard constraint, apply it first. When sovereignty requires in-country or stronger, self-hosting or hybrid is enforced ahead of cost and volume conclusions.
- If sovereignty is soft, volume decides. Without sovereignty constraints, the break-even volume is the criterion for choosing between commercial APIs and self-hosting.
For customer-meeting discovery questions, use the checklist in AI Platform Decision Framework.
Reference Architecture Summary
The standard form that satisfies both sovereignty and cost is in-region EKS-hosted vLLM self-hosting.
Figure 2. Reference architecture for in-region EKS-hosted vLLM self-hosting
Details of the deployment implementation (image build, S3 caching, multi-node LeaderWorkerSet, incident cases) are covered in Custom Model Deployment Guide, and GPU-node provisioning choice is covered in EKS GPU Node Strategy.
Impact of Infrastructure Choice on Cost and Sovereignty
The GPU-node provisioning method determines the achievable range of performance optimization. The following are constraints verified against primary sources.
- Local NVMe: EKS Auto Mode does not support a CSI driver that exposes instance store as a Pod volume, and the NodeClass
ephemeralStorageconfigures an EBS volume. The standard KarpenterEC2NodeClasscan useinstanceStorePolicy: RAID0to RAID0 local NVMe (EKS Auto Mode NodeClass). - EFA: EKS Auto Mode supports statically configuring
interfaceType: efa-onlyinterfaces via the NodeClassadvancedNetworking.networkInterfaces, in addition to the EFA device plugin. The standard KarpenterEC2NodeClassnetworkInterfaces(Karpenter v1.11+) and launch-template paths for managed/self-managed nodes also support EFA configuration, so the premise that maximum-bandwidth configuration is possible only via standard Karpenter does not hold as of this writing. The number of network cards differs by instance: p5.48xlarge supports 32 and p5en.48xlarge supports 16 network cards, and both instances have an aggregate network bandwidth of 3,200 Gbps (Static Network Interface Configuration, Manage EFA devices on Amazon EKS, Maximize network bandwidth). - Conclusion: Single-node inference can be served with EKS Auto Mode. EFA-based multi-node distributed inference can be configured on Auto Mode via
advancedNetworking.networkInterfaces(efa-only). The one remaining gap is exposing local NVMe instance store as a Pod volume, which Auto Mode does not support; use the standard Karpenter node pool when that requirement exists.
p5en.48xlarge (8×H200, 1,128 GiB GPU memory, EFAv3 3,200 Gbps) is available in the Tokyo (ap-northeast-1) region (AWS News). On-Demand pricing has been reported around $63/hr in us-east-1 and around $79/hr in Tokyo, but a primary price source stating the region, operating system, tenancy, purchase option and date has not been verified as of 2026-10-29. These two values are used only as illustrative unit prices to explain the regional difference. Prices change frequently; for actual figures, see the AWS official pricing page.
Summary
Self-hosting open-weight models is decided by two drivers: token economics and data sovereignty. If sovereignty requires in-country or stronger, in-region self-hosting or hybrid is selected ahead of cost and volume. Without sovereignty constraints, the criterion is whether monthly inference volume crosses break-even. In self-hosting, performance optimization — quantization, vLLM tuning, NVMe, EFA — directly lowers per-token unit cost. Node provisioning choice (Auto Mode vs standard Karpenter) governs the achievable range of optimization and must be decided together with deployment form.
References
Official Documentation
- Manage EFA devices on Amazon EKS — EKS EFA device-plugin and DRA support scope
- Create a Node Class for Amazon EKS — EKS Auto Mode NodeClass specification
- Amazon EC2 P5 Instances — p5 and p5en GPU instance specs and pricing
- Z.ai GLM-5.2 blog — Example of an open-weight model's license and architecture
Related Documents (Internal)
- AI Platform Decision Framework — High-level decision: managed vs open-source vs hybrid
- Coding Tools Cost Analysis — Detailed break-even, Cascade, and Spot cost simulations
- Enterprise Compliance Framework — Mapping to e-Finance Supervisory Regulation, ISMS-P, and SOC2
- Custom Model Deployment Guide — Detailed EKS deployment for large open-weight models
- EKS GPU Node Strategy — Auto Mode vs Karpenter node provisioning