vLLM Production Monitoring: A Practical Stack Guide for AI Engineers

vLLM has become the de facto open-source inference engine for serving large language models in production. Its PagedAttention mechanism delivers 2-5x throughput improvements over naive Hugging Face Transformers by managing the KV cache memory more efficiently. But that performance gain comes with new operational complexity: GPU memory pressure, KV cache hit rates, prefill/decoding phase imbalances, and speculative decoding behavior all require specialized monitoring that traditional DevOps tooling misses.

This guide walks you through building a production monitoring stack for vLLM covering what metrics matter, which open-source tools to use, and how to wire everything together.

Why vLLM Needs Its Own Monitoring Stack

Traditional API services expose request rate, error rate, and latency percentiles. vLLM is different. The core performance gains come from memory management decisions made inside the inference loop decisions that affect throughput by 2-10x depending on your workload.

The key things you need to monitor that standard HTTP observability will not catch:

• GPU memory utilization: vLLM pre-allocates a KV cache based on gpu_memory_utilization. If this is set too high, you will get OOM kills. Too low, you are wasting expensive GPU memory.
• KV cache hit rate: vLLM cache is your primary throughput lever. A low cache hit rate means you are recomputing tokens instead of serving from memory.
• Prefill vs decode throughput imbalance: If prefill is bottlenecking, you need different optimizations than if decode is the constraint.
• Number of ongoing sequences: vLLM block manager tracks active sequences. Understanding batch composition helps you tune max_num_seqs.

If you are running vLLM through Ray Serve (via vllm.entrypoint.api or serve run), you also have the Ray dashboard and Ray metrics to contend with.

The Metrics That Actually Matter

Throughput Metrics

• vllm:num_generation_tokens_total: Total tokens generated. Monitor rate.
• vllm:num_prefill_tokens_total: Total prefill tokens processed. Ratio to decode tracks workload shape.
• vllm:scheduler_running_steps: Number of active generation steps. Shows actual parallelism.

Memory Metrics

• vllm:gpu_cache_usage: KV cache memory used over allocated. 85-95% healthy, above 98% OOM risk.
• vllm:gpu_cache_usage_utilization: Normalized 0-1 scale. Primary tuning knob.
• vllm:num_mixed_chunked_prefill: Prefill batches mixed with decode. Indicates memory pressure.

Latency Metrics

• vllm:e2e_request_latency_seconds: Wall clock time from request to last token. p99 under 10s for most apps.
• vllm:time_to_first_token_seconds: TTFT, critical for streaming UX. p99 under 1 second.
• vllm:time_per_output_token_seconds: TPOT, inter-token latency. p99 under 100ms.

Setting Up Prometheus + Grafana for vLLM

Step 1: Enable vLLM Metrics

vLLM exposes metrics via a Prometheus endpoint at /metrics when you start the server. Make sure you are running with --enable-metrics (enabled by default in recent versions):

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192 \
  --enable-metrics

The server will expose metrics at http://your-host:8000/metrics.

Step 2: Set Up Prometheus to Scrape vLLM

Add a scrape_config to your prometheus.yml:

scrape_configs:
  - job_name: vllm
    static_configs:
      - targets: [localhost:8000]
    metrics_path: /metrics
    scrape_interval: 10s

If you are running behind a reverse proxy or on Kubernetes, adjust the target accordingly.

Step 3: Grafana Dashboard

Here is a starting dashboard JSON for Grafana. Import via + Import and paste the JSON:

GPU Cache Utilization Gauge

• Metric: vllm:gpu_cache_usage_utilization
• Type: Gauge
• Thresholds: green (0-0.85), yellow (0.85-0.95), red (0.95-1.0)

Tokens Generated per Minute

• Metric: rate(vllm:num_generation_tokens_total[5m])
• Type: Time series graph
• Shows actual throughput in tokens per second

Request Latency Percentiles (p50/p95/p99)

• Metric: histogram_quantile(0.50, rate(vllm:e2e_request_latency_seconds_bucket[5m]))
• Same for p95 and p99. Overlay all three on one graph.

Time to First Token (TTFT) p99

• Metric: histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))
• Alert threshold: over 1 second for most models is noticeable UX degradation.

Scheduler Running Steps

• Metric: vllm:scheduler_running_steps
• Shows actual GPU parallelism. If this frequently drops to 1-2 while the queue is non-empty, you have a scheduling bottleneck.

Ray Dashboard: If You Are Running via Ray Serve

If you are using ray serve run or the vllm.entrypoint.api with Ray, you get the Ray dashboard for free:

ray start --head
# Access at http://localhost:8265

ray metrics --dashboard-address=localhost:8265

Ray metrics include:

• ray_num_task_executions: task throughput
• ray_resource_usage: CPU/GPU/memory per actor
• ray_get_task_latency: end-to-end task latency

Combined with vLLM native metrics, this gives you a full picture from request arrival to token delivery.

Common Issues and Detection Patterns

Issue 1: OOM Kills from Overallocated KV Cache

Symptom: vLLM container gets OOM-killed. nvidia-smi shows GPU memory at 99% right before crash.

Detection: Alert when vllm:gpu_cache_usage_utilization exceeds 0.95 for more than 5 minutes.

Fix: Reduce --gpu-memory-utilization by 5-10% increments. If you are below 0.80 and still OOMing, reduce max_model_len.

Issue 2: Thrashing Due to Too Many Concurrent Sequences

Symptom: Throughput drops suddenly. Scheduler running steps fluctuates wildly. Latency spikes.

Detection: If vllm:scheduler_running_steps frequently drops to 1-2 while the queue is non-empty, you have a scheduling bottleneck.

Fix: Lower --max-num-seqs (default 32). Fewer, larger batches often outperform many small sequences.

Issue 3: Low Cache Hit Rate

Symptom: Tokens per second is lower than expected for your hardware.

Detection: vLLM exposes KV cache metrics in recent versions including vllm:kv_cache_prefix_hit_rate if available.

Fix: Increase average prompt similarity (batch similar requests together), or serve with a larger gpu_memory_utilization to fit more sequences in cache.

Issue 4: Prefill Bottleneck

Symptom: TTFT is high but TPOT is normal. The model is slow to start generating.

Detection: Compare num_prefill_tokens_total rate against num_generation_tokens_total rate. If prefill rate is much higher than decode rate, you are prefill-bound.

Fix: Use continuous batching (vLLM default). For very long prompts, consider prompt caching or splitting across smaller chunks.

Cost Optimization: Getting More Tokens Per Dollar

vLLM efficiency directly translates to cloud spend. Here is how to optimize:

1. Tune gpu_memory_utilization with profiling

Profile your specific model to find the optimal setting. Reduce in 5% steps until OOM-free with headroom.

2. Use fp8 quantization for inference

Requires H100/A100 fp8 support. Delivers approximately 40% memory reduction with minimal accuracy loss.

3. Enable speculative decoding for lower latency workloads

Uses a smaller draft model to predict tokens, accepting speed-latency tradeoffs for batch inference scenarios.

4. Monitor tokens per GPU-second as your primary cost metric

Calculate as: tokens generated per hour divided by (number of GPUs times GPU cost per hour). This gives you cost per million tokens generated. If you are sizing a deployment across H100, A100, and L40S SKUs and want a back-of-envelope number before the benchmark runs, the LLM API cost calculator gives you a token-throughput-to-monthly-bill estimate in under a minute using current cloud pricing.

OpenTelemetry Integration

For enterprise environments, vLLM supports OpenTelemetry traces:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("vllm_generate") as span:
    outputs = llm.generate(prompts)
    span.set_attribute("num_tokens", len(outputs[0].outputs[0].token_ids))

Export to Jaeger, Datadog, or any OTLP-compatible backend. The same monitoring surface (Prometheus scrape + Grafana dashboards + OTel traces) keeps working regardless of whether the request took the native vLLM path or the transformers modeling backend path discussed below.

vLLM v0.25 Goodput Tuning (July 2026)

vLLM v0.25 makes Model Runner V2 the dense-model default and removes PagedAttention, so raw tokens per second is a weaker release KPI. Track goodput: requests per second that meet a latency contract. Sweep gpu_memory_utilization, max_num_batched_tokens, and max_num_seqs with pinned model, GPU, driver, workload, and SLO thresholds. Run chatbot, reasoning, and agentic workloads separately, publish a compliant-request counter to Prometheus, and alert on the goodput-to-throughput ratio.

Transformers vLLM Modeling Backend (July 2026)

On July 8, 2026, Hugging Face's Harry Mellor and Lysandre Jolidon published results from the new --model-impl transformers flag in vLLM. The result is that the transformers modeling backend now matches or exceeds native vLLM throughput on three reference workloads:

• Qwen3-4B dense: transformers backend equal within noise to native vLLM at single-node FP8
• Qwen3-32B TP=2: transformers backend slightly above native vLLM on token throughput at TP=2 across 2xH100
• Qwen3-235B-A22B-FP8 MoE DP+EP: transformers backend matches native vLLM at FP8 across 8xH100 with data-parallel + expert-parallel routing

This is the single biggest vLLM-side change since v0.19 (April 14, 2026). Before, anyone shipping a custom model had to port the modeling code into vLLM's internal backend to get PagedAttention, continuous batching, and speculative decoding. The --model-impl transformers flag removes that constraint: you can run unmodified Hugging Face transformers models on vLLM's serving stack and keep the production-grade scheduling and KV cache management.

When to choose which backend

• Native vLLM (the default): still the right call for legacy production stacks already tuned on it. Most production guides, dashboards, and incident playbooks written before July 2026 assume native vLLM. Switching has real cost: the metrics names, the OTel attribute names, and the error surfaces change.
• transformers backend: the right call for new models, models with custom modeling code (custom attention, custom MoE routing, custom linear-attention hybrids), and any team whose bottleneck is porting modeling code rather than serving scale. Custom attention implementations like linear attention, state-space models, or sliding-window attention can now ride vLLM without a port.

One-command migration cookbook

For teams already running vLLM, the migration is one flag plus one upgrade:

# Upgrade vLLM (the --model-impl transformers flag landed in v0.20.x)
uv pip install --upgrade vllm --torch-backend auto

# Smoke-test the same model with the transformers backend
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3-4B \
  --model-impl transformers \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192 \
  --enable-metrics

# Verify the load succeeded
curl -s http://localhost:8000/metrics | grep vllm:gpu_cache_usage_utilization

Linear-attention exception: some attention variants (linear-attention hybrids, RWKV-style state-space layers) still require the transformers backend because vLLM's native kernels target standard multi-head attention. If you are running a hybrid model, you are already on the transformers path by necessity; the new flag just makes that the supported, performant choice rather than a workaround.

Monitoring implications

The monitoring surface is identical between the two backends. The same DCGM exporter for GPU power and utilization, the same Prometheus scrape of /metrics, the same Grafana dashboards built around vllm:gpu_cache_usage_utilization and the latency histograms, and the same OpenTelemetry gen_ai.* spans (where supported) all work without modification. The only change is in the modeling code path, not in the inference loop or in what gets exposed.

What does change operationally:

• First-token latency variance increases slightly on the transformers backend for models that weren't ported (typically 5-15% higher p99 TTFT in the first 24 hours after switch). Plan a burn-in window.
• Out-of-memory patterns differ: native vLLM OOMs on KV cache exhaustion; transformers backend can OOM on activation memory for models with custom ops that aren't pre-fused. Watch vllm:gpu_cache_usage_utilization and DCGM_FI_DEV_FB_USED together.
• Speculative decoding (--speculative-model) still requires the native vLLM backend as of v0.20.x. If you rely on speculative decoding for latency, do not switch yet.

The Monitoring Stack at a Glance

The architecture that works:

• vLLM Server: exposes /metrics endpoint with native Prometheus format for GPU utilization, cache hit rate, latency histograms
• Prometheus: scrapes vLLM every 10s, stores time series, fires alerts on OOM risk and latency spikes
• Grafana: dashboards for KV cache utilization, throughput, latency percentiles, and token efficiency

If you are running vLLM in production today and not monitoring these metrics, start with two panels: GPU cache utilization (catch OOM risk before it kills your service) and request latency p99 (track user experience degradation). From there, add throughput efficiency metrics to correlate GPU spend with business value.

Next Steps

If you are running vLLM in production today and not monitoring these metrics, start with two panels:

1. GPU cache utilization: catch OOM risk before it kills your service
2. Request latency p99: track user experience degradation

From there, add throughput efficiency metrics to correlate GPU spend with business value.

CVE-2025-9141: why your vLLM host is now a trust boundary

Every section above treats vLLM as performance infrastructure — GPU utilization, cache hit rate, latency, throughput. The August 2025 disclosure of CVE-2025-9141 (CVSS 8.8) added a new dimension: vLLM is now security infrastructure, and the vLLM host is a high-value target on your network.

The vulnerability is an unsafe deserialization flaw in vLLM's Qwen3-Coder tool-call parser: the parser passed almost every tool-call argument through Python's eval() when converting parameters to typed values. The full advisory (filed as GHSA-79j6-g2m3-jgfw) lists three preconditions: --enable-auto-tool-choice is set, --tool-call-parser qwen3_coder is selected, and the tool-call parameter type is not explicitly defined. Under those conditions, a model emit that looks like a legitimate tool-call argument becomes host-level arbitrary code execution. The fix landed in vLLM PR #21396 (commit 4594fc3b) and is in the vLLM 0.10.1.1 release. NVD's vulnerable range is >= 0.10.0, < 0.10.1.1; the canonical reference essay by Boyd Kane — "LLMs could control their host machines by exploiting inference engines" — covers the architectural pattern in depth (2026-08-24).

This is qualitatively different from a typical CVE in a serving dependency. The exploit class is tool-parser RCE via LLM-emitted tokens: the model emits a string the parser eval()s, and the host is compromised without any prompt injection, jailbreak, or input-side attack. The reference essay documents why the threat is structurally novel — vLLM's documentation lists support for more than 200 model architectures, and its examples directory contains about 35 Jinja chat templates, each a small DSL that parses the model's emitted tokens into chat turns, tool calls, and structured responses. Slightly misspecified parsing logic result in an LLM's output being interpreted as code to execute.

Why the vLLM host is a high-value target

The blast radius of a successful exploit is not "context leak" — it is full host compromise. A successful CVE-2025-9141 exploit on a vLLM host gives the attacker:

• Frontier-model compute. H100 / H200 / B200 / MI300X GPUs are scarce, expensive, and high-priority targets for cryptominers and model-extraction attackers. A foothold on the vLLM host is a foothold on commodity compute at scale.
• Model weight access. Self-hosted model weights are typically the most sensitive artifact on the inference host — fine-tuned on proprietary data, often regulated, and increasingly the subject of extraction attacks. A compromised host can ship the weights to an attacker-controlled endpoint.
• Datacenter-adjacent foothold. The vLLM host usually has high-bandwidth network access to model registries, vector databases, training pipelines, and downstream services. A successful exploit pivots easily.

The mitigation is to treat the vLLM host as if it were an internet-exposed production database: keep it off the same network segment as production secrets, credentials, and weights; sandbox the parser process so a successful exploit does not reach host resources; alert on anomalous outbound traffic from the vLLM pod. See our guide to agent-sandbox vs agent-substrate runtime primitives for the seccomp + gVisor + Firecracker patterns that apply directly to the inference engine host.

The Jinja chat-template parser hardening checklist

The Jinja2 layer is where the parser logic lives. Five concrete items:

1. Pin vLLM ≥ 0.10.1.1 (or the equivalent patched release for your inference engine — SGLang, TGI, TensorRT-LLM all maintain their own advisories). Subscribe to vLLM GitHub Security Advisories for the next CVE in this class.
2. Grep the parser source for eval(), exec(), compile(), and pickle.loads() in any forked or pinned vLLM version. CVE-2025-9141 is one of a class — the same pattern in other parsers is plausible.
3. Disable tool calling if you do not use it. The three preconditions for CVE-2025-9141 all start with --enable-auto-tool-choice; if your inference workload does not need tool calling, do not enable it. The exploit class evaporates.
4. Fuzz the parsers like any user-input parser. Treat each chat template and tool-call parser as if it were parsing untrusted network traffic (because it is). OSS-Fuzz has corpus coverage for some inference engines; check whether your parser is in scope, and otherwise build a small fuzz harness with hypothesis or libFuzzer.
5. Sign and review chat-template changes. The 35 Jinja chat templates in vLLM's examples directory are written by humans and interpolate model output. Treat chat-template changes as security-relevant code: review, sign, and roll them through the same change-management pipeline as your application code.

The pattern is the same as the security primitives covered earlier in this guide — input validation, output filtering, capability scoping — applied to the inference engine's own parsing code paths. The threat model is new (output-side, not input-side; tool-parser RCE, not prompt injection), but the discipline is the same: identify the trust boundary, sandbox what crosses it, and never eval() anything you did not write yourself.