GPU Inference Platform on AWS EKS
Context
An LLM inference lab on AWS EKS that I am building to measure what self-hosted serving costs and how it behaves on spot GPUs. The first GPU session is still to come. It runs in short burst sessions on my own AWS account and is torn down the same day. It started as a multi-cloud MLOps platform; I cut it back to AWS and LLM inference when the multi-cloud surface turned out to be unverified (ADR-017).
Problem
Teams adopting self-hosted LLMs rarely know what serving actually costs per token on the hardware they can get, or how it behaves when spot capacity disappears. I wanted measured answers, not vendor numbers.
Approach
A small, reproducible platform: Karpenter-provisioned spot A10G nodes, vLLM on KServe, GPU and request telemetry, and a signed, policy-enforced supply chain, rebuilt from empty state for each session.
Why: OpenAI-compatible API and continuous batching out of the box; the bf16 vs AWQ comparison is the measurement, not an assumption
Why: Simpler operations; I proved live that canaryTrafficPercent is ignored in RawDeployment (ADR-016), so rollback is driven by an error-rate analysis instead of a traffic split
Why: Lowest cost per GPU-hour for short sessions; accepted spot interruptions, which are part of what is being measured
Why: More moving parts in CI; every first-party image that runs is traceable to a reviewed commit
My role
Designed and built it end to end: Terraform for EKS and the GPU pool, the serving path, CI/CD with OIDC, the admission policies, the teardown tooling, and the runbooks. The July 2026 deployment and its green rebuild surfaced 38 defects (34, then 4 more), all fixed through the pipeline and written up as a retrospective.
- Serving Loop
- sklearn/MLServer loop end to end on EKS, July 2026 (38 defects fixed)
- Rollback
- Metric-gated, demonstrated live
- Burst Session
- One clean cycle, Nov 2026: deploy 33 min, teardown 12.5 min
- GPU Benchmarks
- Not published yet