Probe patterns by workload
Pod lifecycle overview · Checklist and references
Probe Patterns by Workload
Image names and health endpoints are examples. Replace them with images and configurations validated by the organization, and measure probe and shutdown timing on actual workloads.
Pattern 1: Web Service (REST API)
apiVersion: apps/v1
kind: Deployment
metadata:
name: rest-api
spec:
replicas: 3
selector:
matchLabels:
app: rest-api
template:
metadata:
labels:
app: rest-api
spec:
containers:
- name: api
image: myapp/rest-api:v1.2.3
ports:
- containerPort: 8080
protocol: TCP
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
# Startup Probe: Verify startup completes within 30 seconds
startupProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 6
periodSeconds: 5
# Liveness Probe: Internal health checks only (exclude external dependencies)
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 0 # Set to 0 when using a Startup Probe
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
# Readiness Probe: Can include external dependencies
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 2
successThreshold: 1
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- sleep 5
terminationGracePeriodSeconds: 60
Kubernetes automatically sends SIGTERM to PID 1 in the container after the preStop hook completes. Running kill -TERM 1 separately in preStop sends a duplicate SIGTERM and can produce unexpected behavior when PID 1 is an init process (tini or dumb-init). Leave SIGTERM delivery to kubelet. The sleep 5 above is an example that assumes traffic propagation delay; measure actual EndpointSlice and load balancer propagation times to determine whether a delay is needed and how long it should be. The preStop execution time is also included in the total termination grace period.
Health check endpoint implementation (Node.js/Express):
This partial example is intended to be added to an existing Express application. The application must configure initialization of app, db, and redis, along with timeouts for dependency calls. Before including the database and Redis in readiness checks, determine whether they are essential for request processing and whether a shared failure would remove all replicas from service simultaneously.
// /healthz - Liveness: Check only the application's own health
app.get('/healthz', (req, res) => {
// This minimal example checks whether the event loop responds.
// Do not use momentary memory utilization or external service failures as restart conditions.
res.status(200).json({ status: 'ok' });
});
// /ready - Readiness: Include external dependency checks
app.get('/ready', async (req, res) => {
try {
// Check the database connection
await db.ping();
// Check the Redis connection
await redis.ping();
res.status(200).json({ status: 'ready' });
} catch (err) {
res.status(503).json({ status: 'not_ready', reason: err.message });
}
});
Pattern 2: gRPC Service
apiVersion: apps/v1
kind: Deployment
metadata:
name: grpc-service
spec:
replicas: 3
selector:
matchLabels:
app: grpc-service
template:
metadata:
labels:
app: grpc-service
spec:
containers:
- name: grpc-server
image: myapp/grpc-service:v2.1.0
ports:
- containerPort: 9090
name: grpc
resources:
requests:
cpu: 300m
memory: 512Mi
limits:
cpu: 1
memory: 1Gi
# gRPC native probe (K8s 1.27+)
startupProbe:
grpc:
port: 9090
service: myapp.HealthService # all three probes must query the same service
failureThreshold: 30
periodSeconds: 10
livenessProbe:
grpc:
port: 9090
service: myapp.HealthService
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
grpc:
port: 9090
service: myapp.HealthService
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 2
terminationGracePeriodSeconds: 45
gRPC health check implementation (Go):
package main
import (
"log"
"net"
"google.golang.org/grpc"
"google.golang.org/grpc/health"
"google.golang.org/grpc/health/grpc_health_v1"
)
func main() {
server := grpc.NewServer()
// Register the health service
healthServer := health.NewServer()
grpc_health_v1.RegisterHealthServer(server, healthServer)
// Set the service status to SERVING
healthServer.SetServingStatus("myapp.HealthService", grpc_health_v1.HealthCheckResponse_SERVING)
// The status can be changed to NOT_SERVING after checking dependencies
// healthServer.SetServingStatus("myapp.HealthService", grpc_health_v1.HealthCheckResponse_NOT_SERVING)
// Start the gRPC server
lis, err := net.Listen("tcp", ":9090")
if err != nil {
log.Fatal(err)
}
if err := server.Serve(lis); err != nil {
log.Fatal(err)
}
}
Pattern 3: Worker/Batch Processing
Batch workers use an exec probe because they do not have an HTTP server.
apiVersion: apps/v1
kind: Deployment
metadata:
name: batch-worker
spec:
replicas: 2
selector:
matchLabels:
app: batch-worker
template:
metadata:
labels:
app: batch-worker
spec:
containers:
- name: worker
image: myapp/batch-worker:v3.0.1
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 2
memory: 4Gi
# Startup Probe: Verify worker initialization
startupProbe:
exec:
command:
- /bin/sh
- -c
- test -f /tmp/worker-ready
failureThreshold: 12
periodSeconds: 5
# Liveness Probe: Check the heartbeat file
livenessProbe:
exec:
command:
- /bin/sh
- -c
- find /tmp/heartbeat -mmin -2 | grep -q heartbeat
initialDelaySeconds: 10
periodSeconds: 30
failureThreshold: 3
# Readiness Probe: Check the job queue connection
readinessProbe:
exec:
command:
- /app/check-queue-connection.sh
periodSeconds: 10
failureThreshold: 3
terminationGracePeriodSeconds: 120
A readinessProbe only adds or removes the Pod from Service endpoints, so a worker going unready does not pause its queue polling loop or job intake. Because a queue worker typically does not receive traffic through a Service, a readiness failure does not stop job intake either. To pause intake, the application itself must stop consuming from the queue; do not rely on the readinessProbe to do it.
Worker application (Python):
import os
import time
from pathlib import Path
HEARTBEAT_FILE = Path("/tmp/heartbeat")
READY_FILE = Path("/tmp/worker-ready")
def worker_loop():
# Signal that initialization is complete
READY_FILE.touch()
while True:
# Update the heartbeat periodically
HEARTBEAT_FILE.touch()
# Process jobs
process_jobs()
time.sleep(5)
def process_jobs():
# Actual job logic
pass
if __name__ == "__main__":
worker_loop()
Pattern 4: Applications with Slow Startup (Spring Boot, JVM)
JVM applications can take 30 seconds or longer to start. Protect startup with a Startup Probe.
apiVersion: apps/v1
kind: Deployment
metadata:
name: spring-boot-app
spec:
replicas: 4
selector:
matchLabels:
app: spring-boot
template:
metadata:
labels:
app: spring-boot
spec:
containers:
- name: app
image: myapp/spring-boot:v2.7.0
ports:
- containerPort: 8080
resources:
requests:
cpu: 1
memory: 2Gi
limits:
cpu: 2
memory: 4Gi
env:
- name: JAVA_OPTS
value: "-Xms1g -Xmx3g"
# Startup Probe: Wait up to 5 minutes (30 x 10s)
startupProbe:
httpGet:
path: /actuator/health/liveness
port: 8080
failureThreshold: 30
periodSeconds: 10
# Liveness Probe: Activate after Startup succeeds
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8080
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
# Readiness Probe: checks the readinessState group (which by default does not include external dependencies)
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 2
terminationGracePeriodSeconds: 60
Spring Boot Actuator configuration:
# application.yml
management:
endpoints:
web:
exposure:
include: health
health:
livenessState:
enabled: true
readinessState:
enabled: true
endpoint:
health:
probes:
enabled: true
show-details: when-authorized
# To include external dependencies in readiness, add them to the group explicitly.
# (the default readiness group only includes readinessState)
group:
readiness:
include: readinessState,db,redis
Pattern 5: Sidecar Pattern (Istio Proxy + Application)
In the sidecar pattern, configure probes for both the main container and the sidecar.
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-with-sidecar
spec:
replicas: 3
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
# Main application container
- name: app
image: myapp/app:v1.0.0
ports:
- containerPort: 8080
startupProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 10
periodSeconds: 5
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 5
# Do not add the Istio sidecar as a manual container.
# Listing only the proxyv2 image and probes does not work: it also needs the
# proxy startup arguments, discovery/identity environment variables, token volumes,
# and iptables/init setup that automatic injection generates. Use automatic injection
# (istio-injection=enabled), or the Native Sidecar example above if you must run a
# supporting container yourself.
terminationGracePeriodSeconds: 90
When Istio uses automatic injection (the istio-injection=enabled label), it automatically adds appropriate probes to the sidecar. Manual configuration is unnecessary.
Native Sidecar Containers (K8s 1.33 Stable)
Native Sidecar Containers were first introduced in Kubernetes 1.28, enabled by default in 1.29, and became Stable in 1.33. This official feature allows an Init Container to operate as a sidecar by setting restartPolicy: Always. It resolves the termination ordering issue in the conventional sidecar pattern.
Conventional issue: A regular sidecar receives SIGTERM at the same time as the main container. If the Istio proxy terminates first, the main application loses network connectivity.
Native Sidecar solution: A sidecar defined as an Init Container terminates after all regular containers have terminated.
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-with-native-sidecar
spec:
selector:
matchLabels:
app: "app-with-native-sidecar"
template:
metadata:
labels:
app: "app-with-native-sidecar"
spec:
initContainers:
# Native Sidecar: Start before the main container and terminate after it
- name: log-collector
image: ghcr.io/your-org/log-collector:replace-with-tested-tag
restartPolicy: Always # This setting enables Native Sidecar behavior
ports:
- containerPort: 2020
resources:
requests:
cpu: 50m
memory: 64Mi
containers:
- name: app
image: myapp:v1
ports:
- containerPort: 8080
Guaranteed termination order:
- Send SIGTERM to the regular container (app).
- Wait for the regular container to finish terminating.
- Send SIGTERM to the Native Sidecar (log-collector).
- Terminate the Native Sidecar.
This ordering is still carried out within the single Pod terminationGracePeriodSeconds budget. Once it elapses, the kubelet force-kills any still-running containers, including the Native Sidecar, with SIGKILL, so set the grace period long enough for the whole sequence to complete within it.
This pattern is suitable for supporting containers that must outlive the main application, such as Istio sidecars, log collectors, and monitoring agents.
Probe Considerations for Windows Containers
EKS supports Windows nodes based on Windows Server 2019/2022. Probes behave differently in Windows containers than in Linux containers.
Differences in Probe Behavior Between Windows and Linux
Table 1. Probe behavior differences and their impact between Windows and Linux containers
| Item | Linux containers | Windows containers | Impact |
|---|---|---|---|
| Container runtime | containerd | containerd (1.6+) | Same runtime, different OS layer |
| exec probe execution | /bin/sh -c | cmd.exe /c or powershell.exe | Different script syntax |
| httpGet probe | Same | Same | No difference |
| tcpSocket probe | Same | Same | No difference |
| Cold start time | Fast (a few seconds) | Slow (10-30 seconds) | Requires a higher Startup Probe failureThreshold |
| Memory overhead | Low (50-100MB) | High (200-500MB) | Requires increased resource requests |
| Probe timeout | Typically 1-5 seconds | 3-10 seconds recommended | Account for Windows I/O latency |
The cold-start time, memory-overhead, and probe-timeout figures in Table 1 are experiential values, not official specifications; measure them in your own environment.
Probe Configuration Example for Windows Workloads
IIS/.NET Framework application:
The stock mcr.microsoft.com/windows/servercore/iis image serves the default IIS site at / but does not expose /healthz or /ready; those endpoints belong to the ASP.NET Core app shown below, which must first be built into the image. The probes below therefore point at /. If you build an image that serves dedicated health endpoints, point the liveness and readiness probes at them instead.
apiVersion: apps/v1
kind: Deployment
metadata:
name: iis-app
namespace: windows-workloads
spec:
replicas: 2
selector:
matchLabels:
app: iis-app
template:
metadata:
labels:
app: iis-app
spec:
nodeSelector:
kubernetes.io/os: windows
kubernetes.io/arch: amd64
containers:
- name: iis
image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2022
ports:
- containerPort: 80
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 2000m
memory: 2Gi
# Startup Probe: Account for Windows cold starts
startupProbe:
httpGet:
path: /
port: 80
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 5
failureThreshold: 12 # 2 times the Linux value (up to 60 seconds)
successThreshold: 1
# Liveness Probe: IIS process health (the stock image serves "/", not "/healthz")
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
# Readiness Probe: IIS process readiness (the stock image serves "/", not "/ready")
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 15
periodSeconds: 5
timeoutSeconds: 5
failureThreshold: 3
successThreshold: 1
terminationGracePeriodSeconds: 60
ASP.NET Core health check endpoint implementation:
// Program.cs (ASP.NET Core 6+)
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Diagnostics.HealthChecks;
var builder = WebApplication.CreateBuilder(args);
// Add health checks
builder.Services.AddHealthChecks()
.AddCheck("self", () => HealthCheckResult.Healthy())
// AddSqlServer is provided by the AspNetCore.HealthChecks.SqlServer NuGet package, not by the built-in ASP.NET Core registration.
.AddSqlServer(
connectionString: builder.Configuration.GetConnectionString("DefaultConnection"),
name: "sqlserver",
tags: new[] { "ready" }
);
var app = builder.Build();
// /healthz - Liveness: Application only
app.MapHealthChecks("/healthz", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("self") || check.Tags.Count == 0
});
// /ready - Readiness: Include external dependencies
app.MapHealthChecks("/ready", new HealthCheckOptions
{
Predicate = _ => true // All health checks
});
app.Run();
Probe Timeout Considerations for Windows Workloads
Windows containers can require longer probe timeouts for the following reasons:
- Windows kernel overhead: System call latency caused by the heavy Windows OS layer
- Disk I/O performance: Metadata overhead in the NTFS filesystem
- .NET Framework warmup: CLR JIT compilation and assembly loading time
- Windows Defender: Process startup delays caused by real-time scanning
Probe timing example (Windows):
The following partial example is intended to be merged into a container configuration. Adjust the numbers after measurement; range strings such as 5-10 cannot be used in Kubernetes integer fields. Also update the health endpoints to match the application.
startupProbe:
httpGet:
path: /healthz
port: 8080
timeoutSeconds: 5
periodSeconds: 5
failureThreshold: 12
livenessProbe:
httpGet:
path: /healthz
port: 8080
timeoutSeconds: 5
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: 8080
timeoutSeconds: 5
periodSeconds: 5
failureThreshold: 3
CloudWatch Container Insights for Windows (GA 2024-04)
AWS launched CloudWatch Container Insights support for Windows workloads in April 2024.
Installing Container Insights on Windows nodes:
# CloudWatch Agent ConfigMap (Windows)
kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
name: cwagentconfig
namespace: amazon-cloudwatch
data:
cwagentconfig.json: |
{
"logs": {
"metrics_collected": {
"kubernetes": {
"cluster_name": "my-eks-cluster",
"metrics_collection_interval": 60
}
}
},
"metrics": {
"namespace": "ContainerInsights",
"metrics_collected": {
"statsd": {
"service_address": ":8125"
}
}
}
}
EOF
# Deploy the Windows DaemonSet
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset-windows.yaml
Checking Container Insights metrics:
# Windows node metrics
aws cloudwatch get-metric-statistics \
--namespace ContainerInsights \
--metric-name node_memory_utilization \
--dimensions Name=ClusterName,Value=my-eks-cluster Name=NodeName,Value=windows-node-1 Name=InstanceId,Value=i-0123456789abcdef0 \
--start-time $(date -u -d '3 hours ago' +%Y-%m-%dT%H:%M:%SZ) \
--end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
--period 300 \
--statistics Average
# Windows Pod metrics
aws cloudwatch get-metric-statistics \
--namespace ContainerInsights \
--metric-name pod_cpu_utilization \
--dimensions Name=ClusterName,Value=my-eks-cluster Name=Namespace,Value=windows-workloads \
--start-time $(date -u -d '3 hours ago' +%Y-%m-%dT%H:%M:%SZ) \
--end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
--period 60 \
--statistics Average
The examples above use a relative window to query a recent range. GetMetricStatistics retention depends on the period you request (data at a 60s period is retained for 15 days, and at a 300s period for 63 days), so a fixed past date can fall outside the retention for that period and return no datapoints.
Unified Monitoring Strategy for Mixed Clusters (Linux + Windows)
1. Separation by node selector:
apiVersion: v1
kind: Service
metadata:
name: unified-app
spec:
selector:
app: unified-app # OS-independent
ports:
- port: 80
targetPort: 8080
---
# Linux Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: unified-app-linux
spec:
replicas: 3
selector:
matchLabels:
app: unified-app
os: linux
template:
metadata:
labels:
app: unified-app
os: linux
spec:
nodeSelector:
kubernetes.io/os: linux
containers:
- name: app
image: myapp:linux-v1
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 5
timeoutSeconds: 3
---
# Windows Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: unified-app-windows
spec:
replicas: 2
selector:
matchLabels:
app: unified-app
os: windows
template:
metadata:
labels:
app: unified-app
os: windows
spec:
nodeSelector:
kubernetes.io/os: windows
containers:
- name: app
image: myapp:windows-v1
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 10 # Windows: Longer interval
timeoutSeconds: 10 # Windows: Longer timeout
2. Unified CloudWatch Logs Insights query:
# Search Linux and Windows Pod logs together
fields @timestamp, kubernetes.namespace_name, kubernetes.pod_name, kubernetes.host, @message
| filter kubernetes.labels.app = "unified-app"
| sort @timestamp desc
| limit 100
3. Grafana dashboard integration:
# Prometheus Query (mixed cluster)
# Linux + Windows Pod CPU cores used (rate() of this counter yields CPU cores consumed, not a utilization percentage)
sum(rate(container_cpu_usage_seconds_total{namespace="default", pod=~"unified-app-.*"}[5m])) by (pod, node, os)
# Aggregate by OS
sum(rate(container_cpu_usage_seconds_total{namespace="default", pod=~"unified-app-.*"}[5m])) by (os)
- Image size: Windows images are several GB (Linux images are tens of MB).
- Licensing costs: Windows Server licensing costs apply (included in EC2 instance costs).
- Node boot time: Windows nodes boot slowly (5-10 minutes; an experiential figure, so measure it in your own environment).
- Privileged containers: Windows does not support Linux's
privilegedmode. - HostProcess containers: Available from Kubernetes 1.22 (GA in 1.26) on Windows Server 2019 and later.