Capacity wasn’t the only assumption that broke. A vLLM pod can take minutes to start serving, and live traffic doesn’t drain off as quickly as you would see with microservices. Every instinct I built came from workloads that start fast and drain fast, so this sounded like an interesting space to explore.

I decided to forget about surging canary rollouts with weighted traffic and focus solely on using Argo Rollouts for analysis across a couple of different deployment scenarios. Argo still calls the new revision the canary even with no traffic split, so that’s the word I use for it throughout. This setup had the added benefit of controlling my GPU costs instead of wrestling with how to minimize waste as I temporarily spun up canaries. I decided to swap out a live pod for analysis, which I hypothesized would make this more of a networking and rollout configuration problem than a resource problem.

I want to give a disclaimer up front that this is a new area for me. These notes are my rough understanding of this process and my journey to learning and experimenting within this space. I’m not exploring this through a production setup angle and I’ll be explicit about the lines that I drew along my journey.

The Architecture

For this project, running a Kubernetes cluster with a GPU node was a clear starting place. To simplify the setup, I decided to keep it as a single node that I can spin up and down as needed. In my homelab, I use k3s as my Kubernetes distribution and since I didn’t need all of the features that come with a managed cluster, I decided to spin up a Google Cloud Platform VM with k3s installed as my foundation.

I had separately landed on using vLLM as my inference engine so GPU compute was the next clear requirement. It was then pretty clear that I needed to run an accelerator-optimized VM, and I landed on the G2 series. The next big constraint that drove a lot of my architecture design was minimizing costs. I didn’t want to be surprised by my cloud spend so keeping my experimentation cheap was important. That meant that I needed to use a small model that would fit on a small machine. This wasn’t a big deal for me because the model wasn’t what I was testing.

Given the nature of my experiment, I knew I needed at least 2 inference workers so that a rollout wouldn’t kill traffic entirely. I eventually realized that I landed on a machine that didn’t have native support for partitioning (MIG) so time-slicing had to be my path for sharing GPU resources. I configured the device plugin on my g2-standard-8 instance to advertise 4 slices and assumed that meant 4 replicas. That was incorrect - more on that later.

Here’s a summary of all of the components I deployed to my cluster:

Inference Workload

vLLM serving Qwen3-0.6B

Deployed as a Rollout without traffic routing features. With no trafficRouting block, Argo has no connection to my networking layer. The weight instead sets the replication ratio (how many pods run the new revision). With maxSurge=0 that ratio is satisfied by converting an existing pod rather than adding one.

The Qwen model was small enough to fit without causing OOM errors for replicas sharing resources. I had to install the NVIDIA device plugin so that the node would advertise the GPU as a schedulable resource. Nothing can request one without it.

maxSurge: 0
maxUnavailable: 1
steps:
  - setWeight: 50

Networking

Envoy Proxy Deployment with a static config sitting in front of headless vLLM services

I needed a way to talk to the inference engine. Based on my earlier hypothesis, I went in expecting to have to tune this a bit. I could have used vLLM router, llm-d router, or a standard k8s controller that I’ve used in the past (like nginx ingress or an API gateway) but I didn’t want something that would hide too much, too early.

My Envoy config uses the LEAST_REQUEST load balancing policy, which scores endpoints on in-flight request count and knows nothing about what’s happening inside vLLM. A cache-aware router would have been the better production choice, which is exactly why I didn’t use one. One of the things I wanted to see was what a naive policy does to a pod that just came up cold. In some ways I’ll describe later, this decision caused some setup headaches but provided a lot of great learnings.

Observability

Prometheus & Grafana

Easiest decision I made since vLLM and Envoy metrics could be easily scraped by Prometheus. I also deployed the dcgm-exporter so that I could see GPU device metrics.

Orchestration & Tools

I also needed a way to interact with the Kubernetes control plane and the inference workloads. Since I was deploying into GCP I considered Identity-Aware Proxy (IAP) or Tailscale (I already use it for my homelab so was familiar with that setup). I decided to take a simpler path: SSH + port-forwarding.

One of my favorite parts about scaffolding this project was how I ended up orchestrating everything. Some of my early design choices when building my Flow CLI project were influenced by how I would experiment in similar ways in the past and how I wanted a better tool to orchestrate those things. It was a no-brainer for me to call on it here.

Under the hood, I used a lot of standard tools like Terraform for the infra, Makefile and shell files for scripting, but Flow wrapped all that up in a nice package that provides some nice ergonomics for working on this project. The full cluster setup is one command (flow provision cluster) and includes preflight checks via the serial runner type. Managing the lifecycle of the VM and workloads is done with easy to remember commands (flow start cluster, flow deploy workloads, flow show status, etc.). Running and viewing the report for experiments is standardized (flow run experiment <X>, flow analyze experiments). All those things are documented and very easy to recall within the Flow CLI terminal UI and the Mochi Desktop that I’ve been building around Flow.

I also leaned on Claude a lot in this project. I didn’t want to have to spend a ton of time looking at Envoy documentation to understand how to configure outlier detection or how to turn on access logs, or NVIDIA documentation to figure out how to set up the vLLM workers so that they’re time sharing on the node. Claude also gave me clear answers on the many things that were new to me, like inference benchmarking. It was important for me to set up some clear rules, though. I still wanted to learn and experiment myself so my CLAUDE.md anchored the coding agents to take a slower pace at implementing and to share many more details than they would have without my guidance. This was also a great test of some run provenance features I have been building into Flow & Mochi. I now have a cleaner structure for following along and understanding what’s running/ran and which of my agents ran it.

Analysis Metrics

A core aspect of this project was understanding which metrics were best suited for monitoring the state of LLM inference workload deployments. Coming into this project, I honestly didn’t know too much about what would be important here. I used a variety of resources, like this post, to ground myself in the common inference metrics like time to first token (TTFT), time per output token (TPOT), goodput, and GPU utilization.

The initial analysis gating metrics that I landed on were the p95 TTFT, p95 TPOT, and the request error ratio. Those didn’t give me the full story, though. With the help of Claude, I ended up scaffolding a bunch of different SLIs that I could monitor throughout this project, spanning the whole stack: the inference engine, Envoy, networking, and NVIDIA GPU usage. It was very interesting watching how tuning my traffic within the cluster changed the shape of metrics and how one metric alone didn’t give the full story.

To generate enough data for the metrics to be meaningful I needed a traffic simulator. I initially started with vllm bench serve, which worked pretty well, but I struggled with getting a variety of output shapes over a longer period of time without more complexity. After doing some more research, I discovered the inference-perf project. What was very interesting was that switching to this from bench serve showed pretty close to the same benchmark measurements, which I read as a good signal. I have two traffic generation paths:

1. Benchmark runner: used to help me define some of the base metrics that I use within my AnalysisTemplate.
2. Sustained load runner: used to run continuous and varied load that includes short prompts and long prompts to simulate the batch and interactive user types while running my experiments.

Where I landed

I originally figured I could just point steady traffic at the cluster and not think too hard about the level, since I wasn’t optimizing for capacity or speed. That was wrong. My first baseline sat at about 12% of my TTFT threshold with nothing ever queuing, so a rollout could do almost anything and the gate would still read green. My experiments were passing for the wrong reason.

So I cranked the batch tenant up until it failed, holding interactive (short) prompts steady at 3 req/s with 128 in / 64 out:

My target sat between 6 x 512 and 6 x 1024, where TTFT climbed 2.4x while output tokens per second fell. The 9 x 1024 row confirms it by delivering less for 50% more offered load. Having a real operating point meant I could set thresholds off measurement instead of estimated numbers. I pinned my generator load at 6 req/s & 1024 input.

Experiment Harness

Before I jump into talking about my results, I want to give a quick overview of the experiment harness that I had. As I mentioned above, Flow was my orchestrator for a lot of this. The last time I ran similar experiments I ended up using kustomize to apply variations on top of the cluster resources. That worked super well and inspired this setup, but with different tooling. I am using raw Kubernetes manifests and envsubst to drop in environment variables from a resolved envfile within those templates. I decided to go with this approach because the envfile configuration works natively with Flow executables and it also works well as a drop-in to my various scripts. I can define my configurations in one file and have that single source of truth be used across the orchestration stack. I considered Helm here as well, but that also would have required a translation layer from the values file to script / Flow inputs.

When it comes to experiments, I can just override some of the configurations that I had defined within that configuration env and run a script that deploys that change to my workloads. It starts the load generator and triggers the rollouts by incrementing a nonce that I have defined on the Rollout pod specs. This forces the analysis process and pod replacement (even without real workload changes).

At the end of this process, I can either jump right into Grafana to see what metrics report or I can run a Flow executable that will gather all the run data and render a standard markdown template with the results.

You can see my entire repo setup, including my experiments, my Flow executables, my Kubernetes manifests, Terraform scripts, etc., all here: inference-cluster-ops.

Experiments

0. Cluster Baseline

I didn’t want to run into a case where I had 100% failures during a rollout so I knew that I needed at least 2 replicas. But I also didn’t want my replicas to grow. With maxSurge=0 and maxUnavailable=1, a stable pod is terminated to make room for the new revision rather than a new pod being added, so the cluster serves at N-1 for the entire pod-startup window.

This ended up turning from a policy preference to a requirement as I began tuning. I missed that time-slicing splits compute, not memory. The 4 slices I configured meant 4 workloads taking turns on the same device, but each one still needs its own full copy of everything resident in VRAM. Nothing gets shared. Time-slicing also means one bad neighbor can impact the rest, which is its own problem.

That’s when I had to get a better sense of what these workloads were actually using. I admittedly still don’t fully understand all the various components of the arithmetic behind this, but at a high level I found that I needed to account for the size of the model weights, the KV cache size (which I was able to configure upfront), and some framework-specific costs, like the CUDA context. I started to go a little bit too far into the weeds here. This was where I gave Claude more rein in terms of just running some tests within the cluster and finding the right setup. I landed on just the 2 pods after doing some calculations and after running some load against them and reviewing metrics, including the device metrics.

Based on my configurations, I came up with this math:

1137 weights + 1679 runtime + 5120 cache = 7936 MiB per pod
Budget: 23034 card − 472 driver = 22562 MiB usable.

This leaves just 6690 MiB free and a third pod needs 7936, so the 4 replicas I originally planned for were out at this size.

Note: only 2816 MiB of that per-pod number is fixed cost. The 5120 MiB of KV cache is what I picked. I wanted to start with a generous number on purpose so that cache pressure wouldn’t be the thing shaping my results. Trimming the cache to around 4700 MiB would have fit a third but making that change while establishing a baseline would have also changed my batching behavior.

Tuning Analysis & Networking

While figuring this out, I ran into even more issues. I mistakenly defined a rollout that had an analysis template before validating that the pods would start up. While this slowed me down, it did give me some more useful insights. I was reminded that an analysis run needs traffic to evaluate against. This was a flaw in my Prometheus queries. They came back empty and the analysis run transitioned to an error state instead of passing. I ended up having to tune my template so that the lack of traffic resolved to a success for non-experiment spec changes.

Then I noticed many failed requests, which pointed to my first networking problem. One of the first things I had to do was disable the request timeout since these requests would use streaming and I didn’t want to prematurely kill them before they were done. Then I saw that during a rollout, Envoy was still sending requests to the pod that had been destroyed! This, bundled with the load balancing policy that I had set for Envoy, made the whole routing situation a lot worse than I wanted to settle for at baseline.

I ended up having to reduce the DNS refresh interval, add request retries, and configure outlier detection so that requests wouldn’t be stuck going to the missing pod for too long. The retries only cover requests that get sent to a pod that is already gone; Envoy doesn’t retry once it starts streaming. That distinction turns out to matter a lot in the second experiment. This was the first sign that my hypothesis was right: the hard part here was configuration, not resources.

Snippet of the Envoy config on the cluster:

outlier_detection:
  consecutive_5xx: 2
  base_ejection_time: 10s
dns_refresh_rate: 2s

And on the route:

retry_policy:
  retry_on: "5xx,reset,connect-failure,refused-stream"
  num_retries: 2
  # without this, a retry can land on the same dead pod it failed against
  host_selection_retry_max_attempts: 3

Finally, I had a good enough base state that allowed me to inject a couple of problem scenarios and see how Argo Rollouts captured and handled them.

1. Cold Pod Problem

What I injected

I ran 2 tests:

1. First I updated the vLLM mount path so that, on rollout, the model’s weights would have to be downloaded as if this was its first rollout.
2. Then I artificially increased the startup time for the pod by adding a 180s sleep before starting the vLLM process.

What the metrics showed

• canary ready: how long the new revision took to start serving
• capacity ratio: replicas_available / replicas_desired, read from Argo’s controller metrics. With 2 replicas it’s 1.0 when both pods are serving, 0.5 when one is.
• capacity mean: the mean of that ratio across every 15s sample in the run window. 0.688 means that averaged over the whole rollout, 68.8% of desired capacity was actually available.
• below full: the amount of time where capacity ratio was under 1
• incidents: continuous stretches where the overall health gate failed. That gate is a composite of the TTFT p95, TPOT p95, error ratio and truncation ratio, each measured against its own objective. breached by records the measurement that caused the incident.

Doubling the startup time barely moved the peak numbers. The damage was the longer stretch of time that the cluster sat below full capacity.

Did the gate catch it

No, and the reason is my configuration rather than Argo itself. The important metric that I thought I needed to watch here was the time until the canary was ready. Given the small model size, I only saw a 19s difference in my first test, which is what convinced me to try artificially increasing the startup with a sleep.

I was running analysis as a canary step, and a step doesn’t start until the new pod has been transitioned to the Ready state. The entire startup window happens before the gate is ever evaluated, which is exactly the window I was injecting into. I should have used background analysis instead, analyzing across the whole rollout rather than at a single step. This would have allowed me to capture issues during that startup window.

The clear issue that I was able to draw from the data was the batch request truncation across all of the tests I ran.

2. Truncated Batch Requests

What I injected

I ended up running several more tests to understand how to fix the truncation issue:

1. Increased the batch output size so that there are always multi-second streams in flight when the pod goes away. This was an attempt to zoom into the truncation issues that I saw with the last experiment.
2. Then I ran a test with the same batch output size, but an increase in the pod’s terminationGracePeriodSeconds.
3. I further optimized #2 by setting up a naive preStop hook that waits long enough for the requests to drain before killing the pod:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep ${PRESTOP_SLEEP_SECONDS}"]
terminationGracePeriodSeconds: ${TERMINATION_GRACE_SECONDS}

What the metrics showed

The grace period includes the preStop time so both of them needed to be set in the last test. This additional configuration didn’t cost me any more time.

Did the fix work

Eventually. At first, I thought the grace period would provide enough time for the requests to drain completely. I saw in the Envoy access logs that requests weren’t being sent to the draining pod after I started the rollout process but the grace period configuration didn’t solve the truncation issue. It seemed to make it slightly worse. I ran another pair of tests and saw 15 and 22 as my truncation values, which confirmed that this was just due to run variance.

The timings provided a more complete picture. The vLLM workloads stopped producing tokens when they received SIGTERM. This left the remaining streams in a zombie state until they died at 27s with a 30s grace and 87s with a 90s grace - when the pod received SIGKILL at the end of the grace period. Adding the preStop hook was the actual fix. The hook runs before the SIGTERM is sent, so it holds off the signal that stops generation instead of extending the window after it. The grace period still has to be long enough to cover the hook but on its own it was never going to help.

3. Aborting a Bad Revision

What I injected

My focus here was to see how Argo’s automatic rollback prevented long-running incidents. I held the changes from experiment 2 and then did the following test:

1. Collapsed vLLM batching so the pod serves one sequence at a time and everything else queues. This triggered higher latency that I knew would trip the analysis gates
2. Same test as #1 but without the rollout gating

What the metrics showed

Did the gate catch it

Yes, as expected. The p95 TTFT on the canary failed on 3/4 samples and aborted the rollout after 260s. On the test without the gate, the same revision was rolled out to all pods, leaving the overall experience in a degraded state until I restored baseline. My gated test didn’t prevent the damage. Requests still hit the low performing pod once it was ready. However, it reduced the duration and impact of the bad change automatically.

Final Thoughts

One of my biggest takeaways here was that the Envoy routing is what provided the most connection resiliency against rollouts, but still left a gap at the stream level. I came in thinking that using Envoy for networking would allow me to just see rollouts in action, and it did, but the retries and outlier detection absorbed enough of the disruption that the gates had very little left to detect at my baseline. It leaves me wondering how another iteration of this experiment with Argo handling traffic routing alongside the metrics analysis would play out.

I was able to validate the experience of Argo as an instrument during these scenarios; however in some cases it didn’t give me as much of a signal as I would have expected. As I reflect on this, I think that has a lot to do with how I configured my load generation and metric thresholds. I had the advantage of knowing exactly what the inputs were and tuning things to get the state into what I hypothesize would happen. It was really cool seeing Argo Rollouts applied to this type of workload and seeing that understanding the system’s behavior and core signals is key to using Argo.

If I were to take this exploration a step further, I would bring back the canary. The highest KV usage I saw was at around 70%. That peak didn’t come at my saturation point, it came during a drain. The reduced capacity during the rollout added a lot more pressure that could ahve been mitigated if I could have trimmed the KV size to fit a 3rd temporary pod. That introduces a new challenge, though: what do you do with all that wasted space when you’re not running a rollout?