GenAI on EKS Starter Kit¶

A starter kit for deploying and managing GenAI components and examples on Amazon EKS (Elastic Kubernetes Service). This project provides a collection of tools, configurations, components, and examples to help you quickly set up a GenAI project on Kubernetes.

What's Included¶

Architecture Overview¶

graph TB
    subgraph EKS["Amazon EKS Cluster"]
        subgraph Gateway["AI Gateway"]
            LiteLLM
            Kong
        end
        subgraph Models["LLM / Embedding Models"]
            vLLM
            SGLang
            TGI
            Ollama
            TEI
        end
        subgraph Apps["Applications"]
            OpenWebUI["Open WebUI"]
            n8n
            OpenClaw
        end
        subgraph Observability
            Langfuse
            Phoenix
            MLflow
        end
        subgraph Storage["Vector Databases"]
            Qdrant
            Chroma
            Milvus
        end
        subgraph NVIDIA["NVIDIA Platform"]
            Dynamo["Dynamo Platform"]
            GPUOp["GPU Operator"]
        end
    end
    OpenWebUI --> Gateway
    Gateway --> Models
    Gateway --> Langfuse
    Models --> NVIDIA
    Apps --> Gateway

Disclaimer¶