EKS Pod Health Checks & Lifecycle Management
13 min read
📌 Reference Environment: EKS 1.33+, Kubernetes 1.30+, AWS Load Balancer Controller v2.7+
1. Overview
Pod health checks and lifecycle management address different parts of service availability:
- Rolling updates: Readiness checks and shutdown coordination help reduce request loss; the outcome depends on application behavior and traffic timing.
- Failure detection: Readiness checks determine whether a Pod should receive traffic, while liveness checks can trigger container restarts.
- Startup protection: Startup probes give slow applications time to initialize before liveness checks begin.
- In-flight requests: Shutdown handlers give existing requests time to finish within the configured termination period.
This guide covers the entire Pod lifecycle, from how Kubernetes Probes work to language-specific Graceful Shutdown implementations, Init Container patterns, and container image optimization.
Related Documentation
- Probe debugging: See the "Probe Debugging and Best Practices" section of the EKS Troubleshooting Guide.
- High availability design: See the "Graceful Shutdown," "PDB," and "Pod Readiness Gates" sections of the EKS High Availability Architecture Guide.
Reading Paths by Objective
- Configure health checks: Probe Basics → Workload-Specific Patterns → Antipatterns
- Investigate request loss during deployments: Termination Sequence → SIGTERM Handling → Connection Draining
- Handle node replacement: Karpenter and Node Drain → Fargate or Auto Mode
- Improve startup time: Init Containers → Lifecycle Hooks → Container Images
Lifecycle Flow
Review initialization, health checks, request processing, and termination separately. Each chapter provides the detailed conditions and configuration examples.
Figure 1. Pod lifecycle flow
Expand the diagram to read labels and follow connections.
Diagram
All Topics
Probes
- Probe Types and Configuration Basics: Review Probe types, mechanisms, and timing together.
- Workload-Specific Probe Patterns: Review examples for REST, gRPC, batch, JVM, and AI workloads.
- Probe Antipatterns: Identify unnecessary restarts and incorrect health check configurations.
- ALB/NLB and Probe Integration: Connect load balancer health checks with Pod Readiness Gates.
- EKS Features and Probe Integration: Review Probe integration examples for individual EKS features.
Termination and Node Lifecycle
- Pod Termination Sequence: Follow the sequence from a termination request to container termination.
- SIGTERM Handling by Language: Review application shutdown handling examples.
- Connection Draining: Review patterns for completing in-flight requests and closing connections.
- Karpenter and Node Drain: Check Pod termination during node replacement and scale-down.
- Node Readiness Controller: Review configurations that manage infrastructure readiness on nodes.
- Fargate Pod Lifecycle: Review startup, health check, and termination configurations for Fargate.
Startup, Deployment, and Optimization
- Init Container Patterns: Compare initialization tasks and Sidecar configurations.
- Pod Lifecycle Hooks: Review execution flows and examples for PostStart and PreStop.
- Container Images and Startup Time: Review image builds, pre-pulling, and startup time comparisons.
- Deployment Checklist and References: Review pre-deployment checks and the complete reference list.
- EKS Auto Mode Checklist: Check Probe and shutdown configurations in Auto Mode environments.
- AI-Assisted Probe Optimization: Review examples that use observability data and AI to optimize Probes.
Find a previous section
Previously shared section links open the chapter that contains the corresponding content.
- Kubernetes Probe Deep Dive
- Three Probe Types and How They Work
- Startup Probe: Protecting Applications with Slow Startup
- Liveness Probe: Detecting Deadlocks
- Readiness Probe: Controlling Incoming Traffic
- Probe Mechanisms
- httpGet Example
- tcpSocket Example
- exec Example
- grpc Example (Kubernetes 1.27+)
- Probe Timing Design
- Timing Design Formulas
- Recommended Timing by Workload
- Probe Patterns by Workload
- Pattern 1: Web Service (REST API)
- Pattern 2: gRPC Service
- Pattern 3: Worker/Batch Processing
- Pattern 4: Applications with Slow Startup (Spring Boot, JVM)
- Pattern 5: Sidecar Pattern (Istio Proxy + Application)
- Native Sidecar Containers (K8s 1.33 Stable)
- Probe Considerations for Windows Containers
- Differences in Probe Behavior Between Windows and Linux
- Probe Configuration Example for Windows Workloads
- Probe Timeout Considerations for Windows Workloads
- CloudWatch Container Insights for Windows (2025-08)
- Unified Monitoring Strategy for Mixed Clusters (Linux + Windows)
- Probe Antipatterns and Pitfalls
- ❌ Antipattern 1: Including External Dependencies in a Liveness Probe
- ❌ Antipattern 2: A High initialDelaySeconds Without a Startup Probe
- ❌ Antipattern 3: Using the Same Endpoint for Liveness and Readiness
- ❌ Antipattern 4: An Overly Aggressive failureThreshold
- ❌ Antipattern 5: An Excessively Long timeoutSeconds
- ALB/NLB Health Check and Probe Integration
- ALB Target Group Health Checks vs. Readiness Probes
- Health Check Timing Synchronization Strategy
- Pod Readiness Gates (Ensuring Zero-Downtime Deployments)
- Gateway API Health Check Integration (ALB Controller v2.14+)
- Gateway API vs. Ingress Health Check Comparison
- Gateway API Architecture and Health Checks
- L7 Health Checks: HTTPRoute/GRPCRoute with ALB
- L4 Health Checks: TCPRoute/UDPRoute with NLB
- Gateway API Pod Readiness Gates
- Health Check Transition Checklist for Migrating from Ingress to Gateway API
- Integrating Probes with New EKS Features in 2025-2026
- Validating Probe Connectivity with Container Network Observability
- CloudWatch Observability Operator + Control Plane Metrics
- Ensuring Probe Performance with Provisioned Control Plane
- Integration with GuardDuty Extended Threat Detection
- Complete guide to graceful shutdown
- Pod termination sequence in detail
- SIGTERM handling patterns by language
- Node.js (Express)
- Java/Spring Boot
- Go
- Python (Flask)
- Connection draining patterns
- Handling HTTP Keep-Alive connections
- Cleaning up WebSocket connections
- gRPC Graceful Shutdown
- Cleaning up database connection pools
- Interaction with Karpenter and node drain
- Karpenter disruption and graceful shutdown
- AZ evacuation pattern with ARC + Karpenter integration
- Handling the 2-minute Spot instance warning
- Node Readiness Controller — managing node-level readiness
- Overview
- Core features
- Continuous mode - ongoing monitoring
- Bootstrap-only mode - initialization only
- Dry-run mode - safe validation
- nodeSelector - selecting target nodes
- YAML examples
- CNI bootstrap - bootstrap-only mode
- Continuous monitoring of GPU nodes
- Checking EBS CSI driver readiness
- Dry-run mode - test rule
- EKS usage scenarios
- Waiting for VPC CNI initialization
- NVIDIA driver readiness on GPU nodes
- Node Problem Detector integration
- Workflow diagram
- Relationship to Pod readiness
- Installation and configuration
- Install Node Readiness Controller
- Verify installation
- Check node status
- Debugging and troubleshooting
- When a taint is not removed
- Test rules in dry-run mode
- References
- Special considerations for the Fargate Pod lifecycle
- Architecture comparison: Fargate vs EC2 vs Auto Mode
- Automatic Fargate Pod eviction for OS patches
- Fargate Pod startup time characteristics
- Sidecar patterns for Fargate without DaemonSet support
- Recommended graceful shutdown timing for Fargate
- Comparison of Fargate, EC2, and Auto Mode from a probe perspective
- Init Container Best Practices
- How Init Containers Work
- Init Container Use Cases
- Use Case 1: Database Migration
- Use Case 2: Configuration File Generation (ConfigMap Transformation)
- Use Case 3: Waiting for Dependent Services
- Use Case 4: Setting Volume Permissions
- Init Container vs Sidecar Container (Kubernetes 1.29+)
- Pod Lifecycle Hooks
- PostStart Hook
- PreStop Hook
- Hook Execution Mechanisms
- exec Hook Example
- httpGet Hook Example
- Container Image Optimization and Startup Time
- Multi-Stage Builds
- Go Application
- Node.js Application
- Java/Spring Boot Application
- Image Pre-Pulling Strategies
- Image Pre-Pulling with Karpenter
- Image Pre-Pulling with a DaemonSet
- distroless and scratch Images
- distroless Example
- Startup Time Benchmarks
- Comprehensive Checklist & References
- Pre-Production Deployment Checklist
- Pod Health Checks
- Graceful Shutdown
- Resources and Images
- High Availability
- Related Documents
- External References
- Official Kubernetes Documentation
- Official AWS Documentation
- Red Hat OpenShift Documentation
- Additional References
- EKS Auto Mode Environment Checklist
- What Is EKS Auto Mode?
- How Auto Mode Characteristics Affect Probes
- Probe Checklist for Auto Mode Environments
- Probe Configuration Differences Between Auto Mode and Manual Management
- Handling Automatic Eviction for OS Patching in Auto Mode
- Verifying That Auto Mode Is Enabled
- AI/Agentic Probe Optimization
- CNS421 Session Highlights - Agentic AI for EKS Operations
- Automatic Probe Optimization with Kiro + EKS MCP
- Debugging Probe Issues with Amazon Q Developer
- Learning Probe Patterns from Tribal Knowledge
- Integrated Probe Optimization Dashboard