Gateway API Implementation Performance Benchmark Plan
A systematic benchmark plan for objectively comparing 5 Gateway API implementations in the same Amazon EKS environment. The goal is to quantitatively identify each solution's strengths and weaknesses to enable data-driven architecture decisions.
This benchmark plan targets the 5 solutions compared in the Gateway API Adoption Guide.
1. Benchmark Objectiveโ
This benchmark aims to objectively compare 5 Gateway API implementations in the same EKS environment, quantitatively identifying each solution's strengths and weaknesses.
Key Questions:
- Which solution is the fastest? (throughput, latency)
- Which solution has the best resource efficiency? (performance relative to CPU/Memory)
- Which solution scales best in large-scale environments?
- What are the trade-offs for each solution?
2. Test Environment Designโ
3. Test Scenariosโ
1. Basic Throughput (Throughput Test)โ
Purpose: Measure maximum RPS (Requests Per Second)
Measures maximum throughput for each solution by increasing concurrent connections from 100, 500, 1000, to 5000.
2. Latency Profileโ
Purpose: Measure P50/P90/P99/P99.9 latency
Measures response time distribution under steady load to compare tail latencies.
3. TLS Performanceโ
Purpose: Measure TLS termination throughput and handshake time
Measures TLS termination performance and handshake overhead for HTTPS traffic.
4. L7 Routing Complexityโ
Purpose: Measure performance impact of header-based routing and URL rewrite
Measures the impact of complex routing rules on performance.
5. Scaling Testโ
Purpose: Measure performance changes as route count increases (10, 50, 100, 500 routes)
Measures routing performance and memory usage with many HTTPRoutes.
6. Resource Efficiencyโ
Purpose: Throughput relative to CPU/Memory usage
Compares efficiency of each solution under the same resource constraints.
7. Failure Recoveryโ
Purpose: Traffic impact during controller restart
Measures downtime and recovery time when a Gateway controller restarts.
8. gRPC Performanceโ
Purpose: gRPC streaming throughput
Measures gRPC protocol support and performance.
4. Measured Metricsโ
| Metric | Unit | Measurement Method |
|---|---|---|
| RPS (Requests Per Second) | req/s | k6 summary or Prometheus rate() |
| Latency (P50/P90/P99) | ms | k6 histogram_quantile or Grafana |
| Error Rate | % | (failed requests / total requests) x 100 |
| CPU Usage | % | Prometheus container_cpu_usage_seconds_total |
| Memory Usage | MB | Prometheus container_memory_working_set_bytes |
| Connection Setup Time | ms | k6 http_req_connecting |
| TLS Handshake Time | ms | k6 http_req_tls_handshaking |
| Network Throughput | Mbps | Prometheus rate(container_network_transmit_bytes_total) |
5. Expected Results (Theoretical Analysis)โ
Expected strengths/weaknesses per solution:
AWS Native (ALB + NLB)
- Strengths: Fully managed, auto-scaling, AWS integration
- Weaknesses: Latency increase from ALB hop, cost
- Expected Performance: Medium (throughput 10K RPS, P99 50ms)
Cilium Gateway API (ENI mode)
- Strengths: Best eBPF performance, native routing, Hubble visibility
- Weaknesses: Configuration complexity, learning curve
- Expected Performance: Highest (throughput 30K RPS, P99 15ms)
NGINX Gateway Fabric
- Strengths: Proven NGINX engine, stability, rich features
- Weaknesses: Higher memory usage
- Expected Performance: High (throughput 20K RPS, P99 25ms)
Envoy Gateway
- Strengths: Rich L7 features, extensibility, observability
- Weaknesses: Resource overhead
- Expected Performance: Medium-high (throughput 15K RPS, P99 30ms)
kGateway (Solo.io)
- Strengths: AI routing, enterprise features
- Weaknesses: Enterprise license required
- Expected Performance: Medium-high (throughput 18K RPS, P99 28ms)
6. Benchmark Execution Planโ
| Phase | Description | Tools | Duration |
|---|---|---|---|
| 1. Environment Setup | Deploy EKS cluster and 5 solutions separately | eksctl, Helm | 2 days |
| 2. Basic Tests | Measure throughput, latency | k6, Prometheus | 1 day |
| 3. TLS Tests | Measure HTTPS performance | k6 (TLS) | 0.5 day |
| 4. L7 Tests | Test complex routing rules | k6 (custom) | 0.5 day |
| 5. Scale Tests | Test route count increase | kubectl, k6 | 1 day |
| 6. Resource Measurement | CPU/Memory profiling | Prometheus, Grafana | 1 day |
| 7. Result Analysis | Data analysis and report writing | Jupyter, Matplotlib | 2 days |
Benchmark execution results will be updated in this document. For related network benchmarks, see CNI Performance Comparison.
Related Documentsโ
- Gateway API Adoption Guide โ Detailed comparison of 5 solutions
- CNI Performance Comparison Benchmark โ VPC CNI vs Cilium network performance
- Infrastructure Performance Benchmark โ Comprehensive infrastructure performance testing