Ralph Loop Part 3: Controlled EKS Inference Optimization
Series: Index · Part 1: Bedrock · Part 2: EKS baseline · Part 3: optimization.
The baseline has not been measured. No speedup, cost saving, cache hit ratio or correctness result is claimed. Freeze the same model, checkpoint, precision, GPUs and client protocol as Part 2 before comparing variants.
Controlled variants
Use the registered matrix: B0 baseline, P0 exact-prefix reuse with cold reusable state, P1 the same configuration with fixed warm-up, and S1 separate read-only semantic-cache fixtures. B0/P0/P1 each have five predeclared attempts.
Clear reusable prefix state before each P0 attempt while keeping model weights resident. Verify the warm state before P1 and record warm-up usage, time and cost separately. Hold all other settings fixed; preserve failed attempts and partial usage.
S1 does not reuse stateful or tool-enabled Ralph responses. Its held-out fixtures cover paraphrases, similar questions with different answers, stale versions, namespace isolation, misses and errors. Record incorrect hits and embedding/lookup overhead as well as hit ratio. S1 fixtures are not Ralph completions.
Required evidence
| Variant | Raw attempts | Cost reconciliation | State |
|---|---|---|---|
| B0 | None | Pending | Planned |
| P0 | None | Pending | Planned |
| P1 | None | Pending | Planned |
| S1 | No held-out result | Pending | Planned |
Publish per-attempt correctness/outcomes, completed-run medians/ranges, failure-inclusive cost and full operating cost. Improvement ratios require an explicitly named matched baseline and a nonzero denominator; undefined values remain undefined.
KV dtype changes, offload, cache-aware routing, multi-replica load and combined variants are outside this core matrix until a measured bottleneck and a bounded revision justify them. Keep the approved budget, allocation limits, idle deadline and cleanup reserve in force.
Return to the series evidence gates. Actual execution, raw evidence and verified cleanup remain tracked in Issue #216.