Cut GPU Costs 40% with KEDA Queue-Depth Autoscaling for vLLM

Why Standard Kubernetes Autoscaling Breaks for AI Workloads

Two months ago I had a vLLM cluster burning $11,400 a month on H100s that were, on paper, mostly idle. CPU utilization sat at 38%, well under the HPA threshold of 70%, so the autoscaler did nothing. Meanwhile the GPU scheduler queue was piling up — users were waiting 9 seconds for a 200-token completion that should have taken 800ms. The metrics I needed to scale on were not in the HPA's vocabulary. That mismatch between what HPA measures and what AI workloads actually need is the entire reason KEDA exists.

Horizontal Pod Autoscaler (HPA) scales based on CPU and memory — metrics that work for web servers and APIs. AI inference workloads don't behave like web servers. A vLLM pod serving 1,000 tokens/second might show 40% CPU utilization while its GPU is completely saturated. CPU is idle because the GPU queue is full. Scale that pod by following CPU and you add more GPU-waiting pods that accomplish nothing except increasing scheduling overhead. After we wired KEDA to the vllm:scheduler_pending_tokens metric, scaling decisions started matching actual demand — replicas scaled up 4 minutes earlier on every traffic spike, and p99 latency dropped from 9.1s to 3.6s. GPU spend is now $6,800 a month for the same workload, a 40% reduction.

The same problem applies to Vertical Pod Autoscaler (VPA) for AI. VPA recommends resource changes based on historical usage — useless when inference demand is bursty and unpredictable, driven by user traffic patterns that have nothing to do with current node metrics.

AI workloads need event-driven autoscaling: scaling based on the actual demand signals that matter — queue depth, token throughput, concurrent requests, or external events like a marketing campaign launching. This is where KEDA changes the equation.

What KEDA Brings to AI Workloads

KEDA (Kubernetes Event-Driven Autoscaling) extends HPA with 50+ built-in scalers that respond to external metrics instead of just CPU and memory. For AI inference, the most relevant scalers are:

• Prometheus — scale on any Prometheus metric: GPU utilization, KV cache hit rate, request queue depth, token throughput
• RabbitMQ / Apache Kafka — scale based on queue depth for async inference pipelines
• AWS CloudWatch — scale based on Bedrock or SageMaker metrics
• Azure Monitor — scale based on Azure OpenAI metrics
• Datadog — scale on any Datadog query for teams using Datadog APM
• CPU / Memory — still useful for non-GPU pods in the inference stack (tokenizers, preprocessors)

KEDA works as a Kubernetes Custom Resource Definition (CRD). You install it as a Deployment, define a ScaledObject that connects your inference Deployment to a scaler, and KEDA automatically manages the HPA for you.

What's New in KEDA 2.20.2

KEDA 2.20.2, released 2026-07-31, is a stability hotfix on the 2.20.x line — eleven fixes, no API changes, no CRD migration, no Helm chart bump. If you are on 2.20.1 this is a one-line helm upgrade; if you are on 2.20.0 the 2.20.1 release notes above cover the updateStatus data-race panic that this release also benefits from. The headline win is the dedicated HPAActive condition on ScaledObject — the fix for the long-standing "HPA metric gap flipped Ready to False" problem that showed up as spurious alerts on every noisy Prometheus scrape interval. The release also closes several production panics: concurrent map writes in the shared root CA CertPool, three nil-pointer dereferences on missing config fields, a gRPC reconnect-loop that flooded operator logs, and a MongoDB scaler leak. Headline fixes from the 2.20.2 release notes:

• Introduce a dedicated HPAActive condition on ScaledObject mirroring the HPA's own ScalingActive status (#7914) — Until 2.20.2, a transient gap in the HPA metric (a Prometheus scrape that returned no samples for a single poll, a Datadog point that hadn't propagated yet, a CloudWatch lag spike) flipped the KEDA-managed Ready condition to False for the duration of the gap, which on a busy inference fleet fired alerting rules that have nothing to do with actual readiness. 2.20.2 introduces a dedicated HPAActive condition that mirrors the HPA's own ScalingActive status separately, so transient HPA metric gaps no longer cascade into the Ready condition. The new condition also drives a metric you can alert on directly when you actually want to know about HPA-side gaps. For teams that already wired alert rules around ScaledObject Ready, this is the release where the noise floor on those alerts drops to actual readiness events. Pairs with the Kubernetes monitoring stack guide for the dashboard pattern that surfaces the new condition.
• Fix concurrent map writes panic in the shared root CA CertPool (#7910) — A Go data race in the shared root CA pool — the operator reuses a single x509.CertPool across scalers, and a scaler that reloaded its CA bundle while another scaler iterated the pool crashed the operator with fatal error: concurrent map writes. Until 2.20.2, a CA rotation on a Prometheus scaler could take down all of the operator's scalers (not just the one rotating). For teams that rotate CAs on a schedule (a common pattern with cert-manager + a custom CA), this is the fix that closes the cascade. Pairs with the eBPF observability guide for the broader network-side observability surface, and with the supply chain security for DevOps 2026 guide for the rotation pattern.
• Fix nil pointer dereference in AWS Secret Manager TriggerAuthentication when awsSecretManager.credentials is omitted (#7927) — A ScaledObject that used AWS Secret Manager TriggerAuthentication without an explicit credentials block and without podIdentity panicked the operator with a nil deref. Until 2.20.2, the only mitigation was to add a placeholder credentials block. 2.20.2 treats the missing field as a configuration error rather than a panic, returning a clean error to kubectl describe triggeredauthentication. For teams running IRSA-based AWS auth where the credentials come from the pod identity and the manifest intentionally omits the block, this is the line where the omit path stops crashing the operator.
• Fix nil pointer dereference in customScalingStrategy.GetEffectiveMaxScale (#7798) — The custom scaling strategy's optional customScalingQueueLengthDeduction field was being dereferenced without a nil check, so a ScaledObject that omitted the field panicked the metrics handler on every poll. 2.20.2 treats the omitted field as zero deduction. For teams using the custom scaling strategy for queue-length-driven inference autoscaling (the pattern this guide walks through), 2.20.2 is the line where the optional-field pattern stops crashing the operator on a representative subset of configurations.
• Fix nil pointer dereference in GetCurrentReplicas when the informer cache returns an undefaulted spec.replicas (#7863) — A race window between the Deployment/StatefulSet/ReplicaSet being observed by the informer cache and the replicas field being defaulted to 1 could return a nil value from the cache, which the metrics handler dereferenced. 2.20.2 treats nil as the Kubernetes default of 1. This is the same class of informer-cache race as #4389 / #4955, and the #7855 ScaledJob CRD validation fix lands alongside it. Pairs with the SRE best practices for AI/LLM systems guide for the informer-cache pattern.
• Restore gRPC reconnect backoff in the metrics service client (#7856) — Until 2.20.2, an unset Backoff in WithConnectParams disabled the reconnect backoff and produced a zero-delay reconnect loop that flooded operator logs when keda-operator-metrics-apiserver was unreachable. For teams running a separate metrics API server (a common split between the controller and the metrics surface), this is the fix that closes the log-flood failure mode when the metrics server is briefly down during a rollout.
• Share HTTP transports across scalers to reuse connection pools (#7789) — A perf fix for high-ScaledObject-count installs. Each scaler was opening its own HTTP transport per metric poll, which on a cluster with hundreds of ScaledObjects produced re-dial storms against the metric backends (Prometheus, CloudWatch, Datadog). 2.20.2 shares transports across scalers and reuses connection pools. The win is most visible on Datadog-backed workloads where the API rate limit was the binding constraint on poll throughput.
• Treat negative external metric values as zero to prevent incorrect HPA scaling (#7880) — Until 2.20.2, a negative external metric value (a transient reporting artifact on some Prometheus exporters, a CloudWatch point that returned a negative delta) was passed through to the HPA, which interprets negative values as "scale to zero." On a noisy metric pipeline this manifested as a ScaledObject hitting zero replicas on a single bad poll, then recovering on the next. 2.20.2 clamps negatives to zero. Pairs with the scalingModifiers fallback behavior from 2.20.0 and the observability 2026 guide for the metric-pipeline pattern.
• Azure Blob Storage Scaler: fix globPattern never matching when written in path-style with a leading / (#6492) — Azure blob names never have a leading /, but the scaler was treating a leading-slash globPattern as a literal prefix match, so a well-formed globPattern like /inference-batches/* matched nothing. 2.20.2 strips the leading slash. Useful for batch-inference teams running on Azure Blob triggers.
• MongoDB Scaler: disconnect the client when the initial Ping fails (#5612) — Until 2.20.2, a failed initial Ping left the background topology-monitoring connections opened by mongo.Connect alive — a slow connection leak that surfaced over weeks of uptime. 2.20.2 disconnects on ping failure. For long-lived inference platforms that key on MongoDB change streams (a less common but real pattern), this is the fix that closes the slow leak.

No ScaledObject / ScaledJob API changes, no CRD migration, no chart-schema bump versus 2.20.1. The Helm chart and bundled manifests have been republished at keda-2.20.2 from the kedacore/charts repo, and the operator image is at ghcr.io/kedacore/keda:2.20.2. If you pinned tag: 2.20.1 in your GitOps repo, bump to tag: 2.20.2 and re-apply — the upgrade path is the same as 2.20.0 → 2.20.1. The fixes are scoped to the metrics operator and the scaler surface; neither requires touching your ScaledObject manifests. Going forward the 2.20.x line is the branch the project ships hotfixes through, so future patch releases will follow the same pattern. For teams running the KEDA + Prometheus + Karpenter stack that this guide walks through, the HPAActive condition is the change that justifies the rebuild even before the panic fixes — it is the line where ScaledObject Ready stops being a noisy mirror of upstream metric gaps and starts reflecting actual readiness. The Kubernetes GPU operators guide covers the broader GPU node-provisioning surface that this stack plugs into, and the Kubernetes cost optimization guide walks through the autoscaling-side cost controls the HPAActive condition unlocks.

What's New in KEDA 2.20.1

KEDA 2.20.1, released 2026-06-08, is a two-bug stability hotfix on the 2.20.x line. If you are already on 2.20.0 you should pull 2.20.1 — the race condition it fixes is a real production panic, not a theoretical one, and the second fix restores an observability signal ScaledJob-based batch inference pipelines have been silently missing. The release is exactly the kind of upgrade that deserves a helm upgrade and a coffee, not a postmortem. The two fixes:

• Fix concurrent map read/write data race in fallback updateStatus (#7838, fix PR #7843) — A Go data race in pkg/fallback.updateStatus — one goroutine iterating a shared map while another wrote to it — that crashed the keda-operator with a fatal error: concurrent map iteration and map write panic. On a busy inference platform running multiple scalers per ScaledObject (Prometheus for queue depth, Kafka for batch jobs, CloudWatch for cost-aware scaling), the panic hit roughly five times a week. The visible symptom in production: a sudden restart of the keda-operator pod, followed by every ScaledObject flapping to zero replicas for ~30 seconds while the operator recovered and re-reconciled. On 2.20.1 a namespace + ScaledObject-name-keyed sync.Mutex in GetMetricsWithFallback makes the race impossible, and a TestUpdateStatusConcurrency regression test fails on -race without the fix and passes with it. If you have ever paged yourself for an unexplained ScaledObject-to-zero blip, this is most likely what you were looking at.
• Fix KEDAScalersStarted event not emitted for ScaledJobs (#7820) — The "Started scalers watch" event was being silently aggregated away for ScaledJob resources. The mechanism: the new events.k8s.io event recorder (the upgrade from the legacy core events resource that landed in 2.20.0) deduplicates events sharing the same (reason, action, regarding) triple, and KEDA happened to use the same event action for the per-scaler "scaler is built" event and the higher-level KEDAScalersStarted event — so the per-scaler events overwrote the aggregate. The operator-visible symptom: kubectl describe scaledjob <name> and kubectl get events never showed a KEDAScalersStarted reason for any ScaledJob (it worked fine for ScaledObjects). For batch-inference pipelines driven by ScaledJob — KEDA firing Argo Workflows, one-shot embedding generation, the "burst scale to drain a queue" pattern that 2.20.0's scalers were designed to enable — the bootstrapping signal that the scaler watch loop is healthy was invisible. 2.20.1 emits the event with a unique action field so the recorder keeps it distinct. The PR also fixes a flaky e2e test for the "accurate" ScaledJob scaling strategy that was supposed to drain an Azure Queue but didn't.

No ScaledObject / ScaledJob API changes, no CRD migration, no chart-schema bump versus 2.20.0. The Helm chart and bundled manifests have been republished at keda-2.20.1 from the kedacore/charts repo, and the operator image is at ghcr.io/kedacore/keda:2.20.1. If you pinned tag: 2.20.0 in your GitOps repo, bump to tag: 2.20.1 and re-apply. The fixes are scoped to the metrics operator and event aggregator; neither requires touching your ScaledObject manifests. Going forward the 2.20.x line is the branch the project ships hotfixes through, so future patch releases will follow the same pattern.

What's New in KEDA 2.20.0

KEDA 2.20.0 ships two new scalers aimed squarely at observability-driven autoscaling, plus a meaningful RBAC change tied to the Kubernetes 0.35 dependency bump. The headline additions for AI infrastructure teams running event-driven autoscaling on Kubernetes:

• New OpenSearch Scaler — Native scaling on OpenSearch cluster metrics (CPU, JVM heap pressure, search latency, pending tasks). For AI teams that store embeddings in OpenSearch and want inference workers to scale on query volume, this is the scaler we have been waiting on — it removes the awkward "scrape OpenSearch metrics into Prometheus, then use the Prometheus scaler" workaround.
• New Elastic Forecast Scaler — Scale on Elastic's ML-based forecast values rather than just observed values. This is useful for AI workloads with predictable daily or weekly traffic patterns (batch retraining, scheduled inference) where you want to pre-scale before the spike hits.
• scalingModifiers fallback behavior — When a modifier metric is unavailable, KEDA now falls back gracefully instead of failing the entire scaling decision. For teams running noisy metric pipelines, this prevents a single missing data point from pinning replicas at 0.
• AWS External ID for TriggerAuthentication — All AWS scalers (SQS, S3, Kafka MSK, CloudWatch, DynamoDB) now support the External ID field, which is required when assuming cross-account roles in AWS Organizations setups with SCP guardrails. Critical for multi-account AI platforms.

Upgrade note (RBAC): The Kubernetes 0.35 dependency bump moves KEDA event recording from the legacy core events resource to the events.k8s.io API group. If you deploy KEDA with custom or restricted RBAC, grant the operator create/patch on events.k8s.io/events before upgrading — otherwise ScaledObject event recording will silently fail. The bundled KEDA manifests and the official Helm chart already include the updated permissions, so this is only a concern for heavily customized installs.

Upgrade to 2.20.0 via Helm or your GitOps pipeline — the core ScaledObject API is unchanged, so no workload migration is required beyond the RBAC note above.

Installing KEDA

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda --create-namespace

Or via ArgoCD if you manage GitOps:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: keda
spec:
  project: default
  source:
    chart: keda
    repoURL: https://kedacore.github.io/charts
    targetRevision: 2.19.0
    helm:
      values: |
        resources:
          limits:
            cpu: "300m"
            memory: "128Mi"
          requests:
            cpu: "100m"
            memory: "64Mi"
  destination:
    server: https://kubernetes.default.svc
    namespace: keda

Scaling AI Inference with KEDA + Prometheus

The most powerful combination for AI inference is KEDA with the Prometheus scaler. This lets you scale based on any metric your inference server exposes — GPU utilization, request queue depth, token throughput, or custom business metrics.

Example: Scaling vLLM on Request Queue Depth

vLLM exposes a metric called vllm:scheduler_pending_tokens — the number of tokens waiting in the scheduling queue. When this exceeds a threshold, it means the GPU is backlogged and you need more replicas. Here's how to wire it up:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-inference-scaler
  namespace: inference
spec:
  scaleTargetRef:
    name: vllm-inference  # Your inference Deployment
  pollingInterval: 15   # Check every 15 seconds
  cooldownPeriod: 300    # Wait 5 minutes before scaling down
  minReplicaCount: 1
  maxReplicaCount: 10
  metricsServer:
    address: prometheus.monitoring.svc:9090
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring.svc:9090
      metricName: vllm_scheduler_pending_tokens
      query: |
        sum(vllm_scheduler_pending_tokens{model="$MODEL_NAME"})
      threshold: "8192"  # Scale up when 8K+ tokens queued

The key configuration decisions:

• pollingInterval: 15 — Aggressive enough to catch bursty AI traffic without causing thrashing. For latency-sensitive APIs, you can go as low as 5 seconds.
• cooldownPeriod: 300 — AI inference is GPU-bound and startup is slow (30-90 seconds for a vLLM pod). A 5-minute cooldown prevents scale-down during momentary dips. Tune down to 120 seconds if your inference pods start faster.
• threshold: 8192 — This is the sum of pending tokens across all replicas. Start conservative and adjust based on observed p99 latency. If latency spikes before hitting this threshold, lower it. If you're scaling up too aggressively, raise it.

Example: Scaling on GPU Utilization for Mixed Workloads

For mixed inference batches where GPU utilization is the real bottleneck:

  triggers:
  - type: prometheus
    metadata:
      metricName: gpu_utilization
      query: |
        avg(rate(DCGM_FI_DEV_GPU_UTIL_total{device="GPU-all"}[2m])) * 100
      threshold: "75"   # Scale up when avg GPU util > 75%

Note: GPU metrics require the DCGM Exporter running in your cluster with the Prometheus node_exporter scraping GPU metrics. See our GPU monitoring guide for setup instructions.

KEDA vs HPA vs VPA: When to Use Each

For AI workloads, these three autoscaling mechanisms serve different purposes:

Scaler	What it scales	Best for	AI relevance
HPA (CPU/Memory)	Pod replicas	Web APIs, CPU-bound workers	Low — GPU bottleneck not visible in CPU metrics
HPA (KEDA + Prometheus)	Pod replicas	AI inference, queue-driven workloads	High — scale on token queue, GPU util, request rate
VPA	Pod resource requests	Stateful workloads, databases	Low — AI workloads need fast scaling, not resource right-sizing
Karpenter	Node count + type	Any workload needing nodes	High — provision GPU nodes on demand
Cluster Autoscaler	Node pool size	Managed K8s (EKS/GKE/AKS)	Medium — simpler than Karpenter, less flexible

The recommendation for AI inference in 2026: Use KEDA for pod-level scaling (fast, event-driven), and Karpenter for node-level provisioning (dynamic, responsive to actual node pressure rather than just pending pods).

Karpenter: Dynamic GPU Node Provisioning

Traditional Cluster Autoscaler works at the node pool level — you pre-define node groups and it adds/removes nodes from those groups. This is limiting when your AI workloads need different GPU types at different times (an A100 for large batch inference, a T4 for small real-time requests).

Karpenter, developed by AWS and now CNCF, provisions exactly the right node type for each pending pod — no node pools, no pre-configuration. When a GPU pod can't schedule because no suitable node exists, Karpenter launches the cheapest available node that satisfies the pod's resources.

Karpenter Provisioner for GPU Nodes

apiVersion: karpenter.sh/v1beta1
kind: Provisioner
metadata:
  name: gpu-provisioner
spec:
  # Only provision nodes for pods that need GPUs
  requirements:
    - key: node.kubernetes.io/lifecycle
      operator: Exists
      values: [spot, on-demand]
    - key: nvidia.com/gpu
      operator: Exists
      values: ["1", "2", "4"]  # 1, 2, or 4 GPU nodes
    - key: karpenter.sh/capacity-type
      operator: In
      values: [spot, on-demand]
  limits:
    resources:
      nvidia.com/gpu: "8"   # Max 8 GPUs in cluster at once
      cpu: "64"
      memory: "256Gi"
  providerRef:
    name: default
  # TTL — recycle nodes after 24h to capture price changes
  ttlSecondsUntilExpired: 86400
  weight: 100

The key decisions for AI workloads:

• Spot + On-Demand mixing — AI training jobs (fault-tolerant) use spot instances. Inference APIs (latency-sensitive) use on-demand. Use node taints to route appropriately.
• GPU count limits — Set an upper bound on total GPUs to prevent runaway costs during an attack or misconfiguration. The GPU limit in the provisioner above (8 GPUs) acts as a circuit breaker.
• ttlSecondsUntilExpired: 86400 — Forces node recycling every 24 hours. This is important for spot instances because you get lower prices on new instances, and it ensures you're always running on the latest driver versions.

Tainting GPU Nodes for Appropriate Workloads

# In your Inference Deployment
spec:
  template:
    spec:
      tolerations:
      - key: "nvidia.com/gpu"
        operator: "Exists"
        effect: "NoSchedule"
      # Only schedule on spot GPU nodes for batch inference
      nodeSelector:
        karpenter.sh/capacity-type: spot
      containers:
      - name: vllm
        resources:
          limits:
            nvidia.com/gpu: "1"
            memory: "64Gi"
            cpu: "8"

Putting It Together: The AI Autoscaling Stack

A production AI inference cluster in 2026 needs three layers of elasticity:

1. Pod-level (KEDA) — Fast response to inference demand signals: queue depth, token throughput, concurrent requests. Scale within seconds.
2. Node-level (Karpenter) — Provision the right GPU node type when pod scaling exhausts current capacity. Responds in 30-60 seconds.
3. Cluster-level (CloudQuota) — Monthly FinOps guardrail. Set a maximum GPU count per cloud account to prevent runaway spend. This is a policy control, not an autoscaler.

Here's how the three layers interact for a traffic spike:

1. Traffic spike → KEDA scales inference pods from 2 → 6 replicas (15-30 seconds)
2. Existing GPU nodes are saturated → pending pods appear → Karpenter provisions a new GPU node (30-60 seconds)
3. If GPU count approaches the limit → Cluster Autoscaler or cloud quota prevents further node provisioning, protecting against runaway costs
4. Traffic normalizes → KEDA scales pods down after cooldown (5 minutes) → Karpenter terminates idle nodes after 24h TTL

The FinOps Perspective: Autoscaling Without Overspending

Autoscaling without cost controls is a liability. Every AI team has a story of a pod that scaled to 50 replicas on a weekend and ran up $4,000 in cloud costs before anyone noticed. The fix isn't to disable autoscaling — it's to add the right guardrails.

1. Set Namespace-Level GPU Quotas

apiVersion: v1
kind: ResourceQuota
metadata:
  name: inference-gpu-quota
  namespace: inference
spec:
  hard:
    nvidia.com/gpu: "8"    # Max 8 GPUs in inference namespace
    requests.nvidia.com/gpu: "8"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: inference-limits
  namespace: inference
spec:
  limits:
  - type: Container
    max:
      nvidia.com/gpu: "4"  # Single pod can't monopolize

2. Use Spot for Batch Inference, On-Demand for Real-Time APIs

Separate your inference workloads by SLA:

• Real-time APIs (p99 below 500ms) — Run on on-demand or reserved instances. You need guaranteed capacity and fast scaling.
• Batch inference / async processing — Run on spot instances with checkpointing. 70-90% cost savings with fault-tolerant code.

3. Monitor Your Autoscaler Efficiency

Track these metrics to ensure your autoscaling isn't creating waste:

Metric	What it tells you	Target
keda_scaler_metrics_value	What signal is triggering scale	Stable, not spiking wildly
keda_pod_scale_count	Actual replica count over time	Gradual changes, not oscillation
karpenter_nodes_terminated	Node churn rate	< 20% daily churn — too high means waste
GPU utilization avg	Are you overprovisioning GPUs?	> 60% for inference, > 80% for training
Monthly GPU cost	Total spend on GPU nodes	< forecast ± 10%

For GPU cost monitoring, see our guide to GPU monitoring for AI inference and Kubernetes cost optimization for the full FinOps stack — and if you are weighing this self-hosted setup against managed APIs, compare the per-token math with our LLM API cost calculator.

Common Failure Modes

KEDA Not Scaling Down After Traffic Drops

If replicas stay high after traffic normalizes, check cooldownPeriod — 300 seconds is conservative but prevents thrashing. Also verify your Prometheus query is returning values when traffic is low (some metrics go to zero, some don't).

Karpenter Not Terminating Idle Nodes

Karpenter waits until all pods are evicted from a node before terminating it. If you have long-running inference requests, nodes can appear "idle" but be waiting for requests to complete. Set ttlSecondsAfterEmpty: 60 to terminate nodes faster when they go empty.

Scale-Up Too Slow for Latency-Sensitive APIs

For APIs requiring below 200ms latency, KEDA's 15-second polling interval is too slow. Pre-scale your minimum replicas to handle normal peak load, and use KEDA only for overflow. Set minReplicaCount: 5 (or whatever your normal peak is) rather than 1.

Summary

AI inference on Kubernetes requires event-driven autoscaling — not CPU-based HPA. The winning combination in 2026 is:

• KEDA + Prometheus for pod-level scaling on actual AI metrics (queue depth, token throughput, GPU utilization)
• Karpenter for dynamic GPU node provisioning that responds to actual demand, not just pending pods
• Namespace GPU quotas as a hard cost guardrail against runaway scaling
• Spot/On-Demand separation to capture 70% savings on batch workloads without impacting API latency

The result is an inference cluster that scales as fast as your users need it to, procures exactly the GPU capacity required, and stops before it empties your cloud budget.