Skip to main content

Kubernetes Event Retention and AI Agent Query Architecture

20 min read
AI documentation guide

Overview​

An AI agent investigating an incident may need Events older than the cluster retains. Kubernetes Events have a default retention period of 1 hour, so historical analysis requires an export pipeline to external storage. This guide separates collection, storage, and querying into a 3-layer architecture and explains how the EKS MCP server and CloudWatch MCP server expose recent and retained event data to agents.

Background: Structural Constraints of Kubernetes Events​

1-Hour TTL​

Kubernetes Event objects are stored in etcd, and the default value of the kube-apiserver --event-ttl flag is 1h0m0s. EKS has maintained the upstream default of 60 minutes, and because events filling up etcd degrades API server performance, EKS did not allow changing this setting for a long time (containers-roadmap #785). After 60 minutes, etcd deletes the events, so kubectl get events can only retrieve the most recent hour of events.

EKS Control Plane Customization (New)

EKS has recently started exposing some scheduler, controller manager, and API server settings through the Kubernetes control plane customization feature, which includes the event time-to-live setting (EKS console help). However, increasing the TTL increases the number of objects stored in etcd, which can affect control plane performance, and the best-effort characteristics below still apply. TTL extension should therefore be treated as a complement to an export pipeline, not a replacement. Supported versions and API details should be verified in the latest documentation before adoption.

Best-Effort Data​

The Kubernetes Event v1 API documentation defines events as follows: "Events have a limited retention time... should be treated as informative, best-effort, supplemental data." Events are supplemental signals with no guarantee of retention or delivery, and cannot serve as the sole basis for incident analysis.

Audit Logs Are Not a Substitute​

Enabling EKS control plane audit logs does not capture Event objects, because the EKS audit policy explicitly specifies Do not log events resources (level: None) (EKS Best Practices: Auditing and logging). Audit logs record "who called which API" and are useful for root cause tracing, but they cannot serve as a retention mechanism for Event resources themselves.

A Common Misconception About EventBridge

Amazon EventBridge aws.eks source events only deliver EKS service events (add-on creation/deletion, health, etc.). Kubernetes cluster events such as Pod scheduling failures and OOMKilled are not delivered to EventBridge.

Architecture: Collect → Store → Query in 3 Layers​

To make events queryable by an AI Agent, design collection (export), storage (durable store), and query (query interface) as separate layers.

Expand the diagram to read labels and follow connections.
Diagram

Collection Layer: Export Options Compared​

MethodCollection TargetCharacteristicsBest Fit
CloudWatch Observability add-on (Container Insights)Container/host/dataplane logsManaged add-on, Fluent Bit basedCloudWatch-centric stack
kubernetes-event-exporterAll Event objectsAttribute-based filtering/routing, leader election HA, 20+ sinksDedicated event pipeline
ADOT / OTel Collector (k8sobjects receiver)K8s objects including EventsIntegrates into existing OTel pipelinesOTel-standardized environments
Custom watcher (aws-samples/eks-event-watcher)Selected eventsCustom, based on Kubernetes API watchSpecial filtering requirements
Default Collection Scope of Container Insights

What the standard Container Insights Fluent Bit DaemonSet collects by default is /aws/containerinsights/{cluster}/application (container logs), /host (host logs), and /dataplane (dataplane logs such as kubelet). Event objects (kubectl get events) are not collected by default. If the claim is "events are being stored via Container Insights," first verify that a separate event collection setup actually exists and which log group it writes to.

kubernetes-event-exporter is the de facto community standard for event pipelines. The EKS Workshop also uses this tool to export events to OpenSearch. Apply the following for production configurations:

# kubernetes-event-exporter configuration example (Loki sink)
logLevel: info
kubeQPS: 100          # prevents event loss in large clusters
kubeBurst: 500
maxEventAgeSeconds: 60
leaderElection:
  enabled: true        # prevents duplicate delivery in HA deployments
receivers:
  - name: "loki"
    loki:
      url: http://loki.monitoring:3100/loki/api/v1/push
      streamLabels:          # static labels only (templates work only in layout/headers)
        app: kube-events
route:
  routes:
    - match:
        - receiver: "loki"
      drop:
        - type: "Normal"   # store only Warnings — Normal is the majority, so this greatly reduces ingestion volume

Storage Layer: Choose by Query Pattern​

Choose the store based on "how the events will be queried."

StoreQuery PatternRetention CostMCP Integration
CloudWatch LogsLogs Insights queries, pattern analysisMedium (reduce via S3 export)Directly supported by CloudWatch MCP and EKS MCP
OpenSearchPrecise field-based search, dashboardsMedium–highCustom tool required
LokiLogQL, low-cost long-term retentionLowCustom tool required
S3 + AthenaSQL post-hoc analysis, archiveVery lowCustom tool required
Kinesis / FirehoseReal-time stream processingVolume-basedCustom tool required

If events are already accumulating in CloudWatch, re-shipping them to Loki (CloudWatch → Loki) wastes a hop. To use Loki as the destination, sending directly from kubernetes-event-exporter to Loki also reduces CloudWatch ingestion costs.

Query Layer: AI Agent Integration via MCP Servers​

Division of Roles Between the EKS MCP Server and the CloudWatch MCP Server​

AWS provides a fully managed EKS MCP server (announced November 2025, preview) and a CloudWatch MCP server. The two servers query different targets, so connecting both is the standard configuration for an incident analysis Agent.

AspectEKS MCP ServerCloudWatch MCP Server
PerspectiveCurrent state of cluster and K8s resourcesHistory of accumulated logs, metrics, alarms
Key toolsget_k8s_events, get_pod_logs, list_k8s_resources, get_cloudwatch_logs, get_eks_insightsAlarm Based Troubleshooting, Log Analyzer (anomaly/error patterns), Metric Definition Analyzer, Alarm Recommendations
AuthenticationAWS IAM (SigV4), CloudTrail auditingAWS IAM (SigV4)
Access controlIAM permission separation (eks-mcp:CallReadOnlyTool / CallPrivilegedTool), AmazonEKSMCPReadOnlyAccess managed policyRead-oriented tool composition
get_k8s_events on the EKS MCP Server Is Not a Retention Solution

get_k8s_events queries the live Kubernetes API and returns events for a specific resource (kind/name/namespace). Because it queries the API server without a separate storage layer, the etcd TTL constraint applies as-is. Events older than 1 hour require querying the log group populated by the export pipeline via get_cloudwatch_logs or the CloudWatch MCP server. The EKS MCP server is currently in preview, so GA status and tool changes should be re-verified before production adoption.

Integrating Stores Without an MCP Server​

For stores without a provided MCP server, such as OpenSearch and Loki, implement custom tools that wrap the query API (LogQL, OpenSearch DSL) using the Strands Agents SDK or similar, and expose them to the Agent. Whatever the store, standardizing on an MCP or tool interface minimizes Agent code changes when the store is replaced.

Pattern A: Minimal Change (CloudWatch-Centric)​

A configuration for starting without adding a pipeline in environments already using Container Insights.

Event collector → CloudWatch Logs → CloudWatch MCP + EKS MCP → AI Agent
                                    (+ enable control plane api/audit logs)
  1. Verify the log group where events are actually stored (see the verification section below).
  2. Enable control plane logs (api, audit). Durable API call history compensates for the gaps in (best-effort) events.
aws eks update-cluster-config \
  --name my-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit"],"enabled":true}]}'
  1. Connect both the CloudWatch MCP server (history analysis) and the EKS MCP server (current state queries) to the AI Agent.

Pattern B: Enhanced Search (Dedicated Event Pipeline)​

A configuration for cases where precise search on event fields (reason, involvedObject, message) and long-term retention matter.

kubernetes-event-exporter ─┬→ OpenSearch/Loki (search & dashboards → custom tools)
                           └→ S3 (long-term archive → Athena SQL post-hoc analysis)

Operating Principles​

  1. Never treat events as the sole evidence — they are best-effort data; cross-verify with audit logs, metrics, and application logs.
  2. Operate the collector explicitly — manage "who watches events and where they are sent" as a pipeline, and monitor loss indicators (watch lag).
  3. Choose the store by query pattern — combine real-time (Kinesis), search (OpenSearch/Loki), and archive (S3+Athena) to match requirements.
  4. Expose to AI Agents through standard interfaces — wrap with MCP servers or SDK custom tools to prepare for store replacement and expansion. Hedge MCP preview-stage risk with your own implementation.

Verification: Confirming Events Are Being Stored​

CloudWatch Logs Insights queries to confirm events are actually accumulating in CloudWatch. Run them against the log group your event collector writes to.

fields @timestamp, @message
| filter @message like /(?i)(FailedScheduling|OOMKilling|BackOff|Unhealthy|FailedMount)/
| sort @timestamp desc
| limit 50

Aggregating occurrence counts by Warning type:

fields @timestamp
| parse @message /"reason":\s*"(?<reason>[^"]+)"/
| filter ispresent(reason)
| stats count(*) as cnt by reason
| sort cnt desc

If events with timestamps older than 1 hour are returned, the export pipeline is working correctly. If only the last hour of data exists, the result is either a live API query or the pipeline started recently, so inspect the collector configuration.

Summary​

Kubernetes Events can be queried while they remain in the cluster, but their default 1-hour TTL and best-effort delivery limit their use as an incident history. Choose a durable store according to real-time query, search, and archive requirements; CloudWatch is one option. In the architecture described here, the EKS MCP server provides current state and the CloudWatch MCP server provides accumulated data. The get_k8s_events tool cannot recover Events that have expired.

References​

Official Documentation​

Blogs / Workshops​