Platform Infrastructure for LLM-Powered Products: Lessons from Building the Matching API at myTomorrows

The Problem: Building Infrastructure for a Product That Doesn't Exist Yet

In early 2026, myTomorrows needed infrastructure for a new product: a clinical trial matching API powered by multi-model LLM inference. The product would process de-identified medical records, run them through multiple LLM providers, and return structured clinical trial matches via a B2B API.

There was no existing infrastructure for this. The matching API was a new product line, not an extension of an existing service. This meant every infrastructure decision was greenfield — and every decision would be hard to reverse once medical data started flowing.

This post covers the architectural decisions I made, the alternatives I considered, and what I learned shipping it to production in March 2026.

Position 1: A Shared Cluster With Hard Boundaries (I Changed My Mind)

The first decision was whether to run the matching API in the existing shared EKS cluster or provision a dedicated one.

My first ADR chose a dedicated cluster: blast-radius isolation for medical data, a simple answer to "where does the data flow?", and an independent upgrade cadence. Before production, I reversed it.

What changed my mind:

• The regulation does not ask for it. GDPR Article 32 is risk-based. No regulator requires cluster-level separation for processing medical data; what matters is the controls you can show.
• The cost was not the $200-400 a month. It was about two weeks of cluster and add-on setup, and seven duplicated add-ons to patch and upgrade for the life of the product. For a small platform team, that operational load is the real risk.
• The boundaries that matter can be drawn inside a cluster. The matching API got its own namespace, a dedicated Karpenter node pool with taints (its pods share nodes only with each other), and per-service IAM scoped to its own queues, buckets and database. Partners reach it through CloudFront and WAF, never the cluster API.

My position: For regulated data, start from the controls you need to demonstrate, not from "dedicated by default". A dedicated cluster is the right answer when you need a different control plane, upgrade cadence or blast radius than the shared platform can give. Otherwise it is cost you pay forever for isolation you could have had for less.

Position 2: CloudFront VPC Origins Over API Gateway for B2B APIs

The production ingress architecture was the most consequential security decision. The options:

1. Internet-facing ALB: Simple, but the load balancer has a public IP. Attack surface includes the ALB directly.
2. API Gateway + VPC Link: AWS-native, but adds latency and has throughput limits that require capacity planning.
3. CloudFront + VPC Origins + internal ALB: The ALB has no public IP. All traffic enters through CloudFront, which provides Shield Standard DDoS protection, WAF integration, and geo-restriction at the edge.

I chose option 3. The key insight: with VPC Origins, the ALB is genuinely internal. There is no public IP to attack. Traffic can only reach it through CloudFront, which means:

• DDoS mitigation happens at the edge, not at the VPC
• WAF rules (7 managed rule groups including known bad inputs, SQL injection, and IP reputation) are evaluated before traffic ever reaches the cluster
• EU geo-restriction is enforced at CloudFront — requests from outside the EU never reach the origin
• The ALB security group only allows traffic from CloudFront's managed prefix list

The tradeoff is debugging complexity. When a request fails, you need to correlate CloudFront access logs, WAF logs, and ALB access logs. I mitigated this with a Grafana dashboard that joins these log sources by request ID.

My position: For B2B APIs processing sensitive data, CloudFront + VPC Origins is strictly better than an internet-facing ALB. The debugging complexity is a one-time investment in observability tooling. API Gateway is the right choice when you need request transformation, usage plans, or API key management — but for a simple reverse proxy with security at the edge, CloudFront VPC Origins gives you more security with less operational overhead.

Position 3: KEDA Over HPA for Async LLM Workloads

The matching API uses an SQS-driven async architecture: API requests enqueue work, worker pods process the queue, results are stored and returned via polling or webhook. This pattern is correct for LLM workloads because inference is slow (10-60 seconds per request) and batch processing is more efficient than synchronous request-response.

The scaling question: how do workers scale?

HPA with CPU metrics is the default Kubernetes answer. But CPU is the wrong signal for queue-processing workloads. A worker might be at 10% CPU while waiting for an LLM response, but it is fully occupied — it cannot accept more work. CPU-based scaling would leave messages sitting in the queue while workers appear underutilized.

KEDA with SQS triggers scales on the right signal: queue depth. When messages accumulate, KEDA adds workers. When the queue drains, KEDA scales down.

The nuance for LLM workloads is in the KEDA configuration:

• scaleOnInFlight=false: Do not count in-flight messages (messages being processed) toward the scale target. LLM inference is slow — if a worker is processing a message for 60 seconds, we do not want KEDA to think it needs another worker for that message.
• cooldownPeriod=7200s: Long-running LLM tasks mean workers should not scale down aggressively. A 2-hour cooldown prevents thrashing during sustained batch workloads.
• Scale range 1-10: Minimum 1 worker for latency-sensitive requests, maximum 10 for batch processing spikes.

In the pre-handover test, 15 messages on the search queue scaled the search workers from 1 to 3 within 30 seconds, and they scaled back down after the queue drained. The ceiling of 10 is configured headroom for batch spikes, not a measured result.

My position: For any async workload where processing time is variable and CPU is not the bottleneck (LLM inference, video processing, ETL), KEDA with queue-depth triggers is strictly better than HPA with CPU metrics. The configuration requires understanding your workload's timing characteristics, but the scaling behavior is fundamentally more correct.

The ADR Process: How Decisions Get Made

Every decision above was documented in an Architecture Decision Record before implementation began. The ADR format I use:

1. Context: What is the situation? What constraints exist?
2. Decision: What did we choose?
3. Alternatives: What else was considered?
4. Consequences: What are the tradeoffs? What do we gain and lose?

The ADR serves two purposes: it forces me to articulate the reasoning (which often reveals gaps in my thinking), and it provides a record for future engineers who will ask "why was it built this way?"

For the matching API, the ADR was a 15-page document covering cluster strategy, networking, security, observability, and scaling. It was reviewed by the engineering lead before implementation began. This investment in upfront design meant implementation was fast — I was not making architectural decisions while writing Terraform.

Production Handover: What It Looks Like

Infrastructure is not done when the Terraform applies cleanly. It is done when the development team can operate it independently. The production handover for the matching API included:

• Grafana dashboard with 18 panels: Request rates, latency percentiles, error rates, queue depth, worker scaling, pod health, node utilization, and cost tracking.
• 8 alert rules: Queue depth exceeding threshold, worker pod restarts, error rate spikes, certificate expiry, node pressure, and budget alerts.
• Runbook: How to deploy, how to scale manually, how to debug common issues, how to handle incidents.
• Infrastructure test results: Documenting how the ingress and autoscaling behaved under load, so the team knows the envelope and what was not tested.

The handover document is not a formality. It is the artifact that determines whether the development team can own the infrastructure without calling me at 2am.

Infrastructure Testing: What Was and Was Not Validated

Before handing over, I ran an infrastructure readiness test. It was deliberately not an application load test: the full path (API → SQS → worker → LLM gateway → models) needed an application API key provisioned in the database and was the product team's test to run.

• Ingress: through CloudFront and WAF to the internal ALB, about 150 requests per second at 50 concurrent connections (p50 304 ms, p99 427 ms), with zero failures. These requests hit a lightweight health endpoint, so they test the edge and ingress, not the application.
• Queue-driven scaling: 15 messages on the search queue scaled the search workers from 1 to 3 replicas within 30 seconds. The match worker correctly stayed at 1, because it only scales on visible messages (scaleOnInFlight=false).
• Stability: zero pod restarts across the whole run. Karpenter did not need new nodes; the existing nodes had headroom.

What it proved: the edge, ingress and autoscaling behave as designed. What it did not prove: how the system behaves under real LLM latency, which is where the timeout and retry tuning later mattered most.

What I Would Do Differently

Start with the observability dashboard earlier: I built the Grafana dashboard after implementation. Building it during implementation would have caught configuration issues sooner — several KEDA tuning parameters were adjusted based on dashboard observations during stress testing.

Document the CloudFront + ALB log correlation pattern from day one: The development team spent time debugging a WAF false positive because they did not know how to correlate CloudFront request IDs with ALB access logs. This should have been in the runbook from the start.

The Thread

This project reinforced my core belief about platform engineering: the infrastructure decisions that matter most are not about which tool to use, but about where to draw boundaries. Dedicated vs. shared clusters is a boundary decision. Internal vs. external ALB is a boundary decision. Queue-depth vs. CPU scaling is about choosing the right signal for the boundary between "needs more capacity" and "has enough."

Good platform engineering is boundary engineering. The tools are interchangeable; the boundaries are architectural.

This post is based on production infrastructure I built and handed over at myTomorrows. If you are building infrastructure for LLM-powered products and want to discuss cluster isolation strategies or async scaling patterns, I am happy to compare notes.