Fix Your Broken GPU Scheduling: The Kubernetes GPU Operator Setup That Actually Works

The GPU Operator Stack

Modern Kubernetes GPU support isn't a single component — it's a layered stack, and understanding each layer matters when something breaks at 2 AM.

┌─────────────────────────────────────────────────────────┐
│  Your ML Workload (training/inference/batch)             │
├─────────────────────────────────────────────────────────┤
│  Kubernetes Scheduler (filters + scores GPU nodes)      │
├─────────────────────────────────────────────────────────┤
│  NVIDIA Device Plugin ( advertises GPU resources )       │
│  DCGM Exporter ( exposes GPU metrics to Prometheus )     │
│  GPU Feature Discovery ( labels nodes with GPU info )  │
├─────────────────────────────────────────────────────────┤
│  NVIDIA Driver ( kernel module on each node )           │
├─────────────────────────────────────────────────────────┤
│  Underlying GPU hardware ( A100, H100, L40S, etc. )     │
└─────────────────────────────────────────────────────────┘

The NVIDIA GPU Operator manages all the Kubernetes-layer components automatically. It provisions the Device Plugin, DCGM Exporter, Driver Manager, and GPU Feature Discovery as operator-managed pods — so you don't have to track version alignment across a cluster manually.

Installing the GPU Operator

# Add the NVIDIA Helm repository
helm repo add nvdp https://nvidia.github.io/gpu-operator
helm repo update

# Create a namespace for the operator
kubectl create namespace gpu-operator

# Install the GPU Operator with Helm
helm install gpu-operator nvdp/gpu-operator \
  --namespace gpu-operator \
  --set driver.enabled=false \      # Drivers pre-installed on node
  --set toolkit.enabled=true \       # Container Toolkit for Docker/Containerd
  --set dcgmExporter.enabled=true \ # GPU metrics for Prometheus
  --set dcgmExporter.serviceMonitor.enabled=true

The driver is typically pre-installed via node OS packages (NVIDIA Driver CUDA), so you disable the driver's embedded install and just manage the Kubernetes-facing components. The Container Toolkit is critical — it injects the NVIDIA CUDA runtime into every pod that requests a GPU, so your training containers don't need to bundle their own CUDA libraries.

Verifying the Installation

# Check that the Device Plugin is running
kubectl get pods -n gpu-operator

# Query available GPU resources
kubectl describe node <gpu-node-name> | grep -A 10 "nvidia.com/gpu"

# Expected output:
# nvidia.com/gpu: 4
# nvidia.com/gpu.memory: 40Gi
# nvidia.com/gpu.product: NVIDIA-A100-SXM4-40GB

If nvidia.com/gpu shows 0 after installation, the Device Plugin isn't communicating with the GPUs. Common causes: missing kernel module (nvidia.ko), incorrect container runtime configuration, or the node was not rebooted after driver install.

Requesting GPUs in Pod Specs

GPU scheduling in Kubernetes requires explicit resource requests. There's no magic discovery — you must tell the scheduler you need GPU hardware.

apiVersion: v1
kind: Pod
metadata:
  name: pytorch-training-job
spec:
  restartPolicy: OnFailure
  containers:
  - name: training
    image: pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime
    command: ["python", "/app/train.py"]
    resources:
      limits:
        nvidia.com/gpu: "2"      # Request 2 GPUs
        memory: "64Gi"
        cpu: "16"
      requests:
        memory: "32Gi"
        cpu: "8"
    env:
    - name: CUDA_VISIBLE_DEVICES
      value: "0,1"               # Match allocated GPUs

Critical rule: Always set CUDA_VISIBLE_DEVICES to match the number of GPUs requested. Kubernetes assigns GPUs by index, and if your code enumerates GPUs differently than the scheduler, you can end up with two containers fighting over the same GPU while a third GPU sits idle.

Multi-Instance GPU (MIG) on A100/H100

MIG allows you to slice a single physical GPU into up to 7 independent instances, each with guaranteed QoS. On an A100 40GB, a 1g.5gb MIG profile gives you 7 slices × 5GB = 35GB, with remaining headroom for system overhead.

# Check MIG mode on the node
kubectl debug node/<node-name> -it --image=nvidia/cuda:12.1.0-base-ubuntu22.04 -- nvidia-smi -L

# Enable MIG mode in the GPU Operator Helm values
helm upgrade gpu-operator nvdp/gpu-operator \
  --namespace gpu-operator \
  --set driver.enabled=false \
  --set toolkit.enabled=true \
  --set dcgmExporter.enabled=true \
  --set mig.strategy=mixed   # "single" or "mixed" — mixed lets you use both MIG and full GPU

# Pod requesting a MIG slice (1g.5gb = 1/7 of A100)
apiVersion: v1
kind: Pod
metadata:
  name: inference-mig-slice
spec:
  containers:
  - name: inference
    image: nvcr.io/nvidia/tritonserver:24.04-py3
    resources:
      limits:
        nvidia.com/mig-1g.5gb: "1"   # MIG resource name

MIG is ideal for inference serving where you want guaranteed latency SLA. For training jobs that saturate the GPU, use full GPU instances — MIG introduces scheduling overhead that cuts into training throughput.

DCGM Exporter: GPU Metrics for Prometheus

DCGM (Data Center GPU Manager) exposes hardware-level metrics — GPU utilization, memory usage, temperature, power draw, NVLink throughput, and ECC errors. Without DCGM, you see pod-level resource usage but not what's actually happening inside the GPU.

The GPU Operator installs DCGM Exporter automatically, but you can also install it standalone:

helm install dcgm-exporter nvdp/dcgm-exporter \
  --namespace gpu-operator \
  --set serviceMonitor.enabled=true \
  --set serviceMonitor.interval=15s

Key DCGM Metrics to Monitor

Metric	Description	Alert Threshold
DCGM_FI_DEV_GPU_UTIL	GPU compute utilization %	< 20% sustained = underutilized
DCGM_FI_DEV_FB_USED	Frame buffer (VRAM) used MB	> 90% of total = OOM risk
DCGM_FI_DEV_POWER_USAGE	Current power draw Watts	> 90% of TDP = thermal risk
DCGM_FI_DEV_GPU_TEMP	GPU die temperature °C	> 83°C sustained = throttling
DCGM_FI_DEV_ECC_SBE_DBE_TOTAL	ECC error count	Any uncorrectable = replace GPU
DCGM_FI_DEV_NVLINK_RX_BYTES	NVLink receive throughput	Low = communication bottleneck

Grafana Dashboard for GPU Nodes

A practical dashboard panels configuration for production GPU monitoring:

# grafana-gpu-dashboard.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-dashboard
  namespace: monitoring
data:
  gpu-dashboard.json: |
    {
      "panels": [
        {
          "title": "GPU Utilization %",
          "type": "stat",
          "gridPos": {"h": 6, "w": 8, "x": 0, "y": 0},
          "targets": [
            {"expr": "DCGM_FI_DEV_GPU_UTIL", "legendFormat": "{{GPU}}"}
          ],
          "fieldConfig": {
            "defaults": {
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {"color": "green", "value": null},
                  {"color": "yellow", "value": 50},
                  {"color": "red", "value": 90}
                }
              }
            }
          }
        },
        {
          "title": "VRAM Usage (MB)",
          "type": "timeseries",
          "gridPos": {"h": 6, "w": 8, "x": 8, "y": 0},
          "targets": [
            {"expr": "DCGM_FI_DEV_FB_USED", "legendFormat": "{{GPU}}"},
            {"expr": "DCGM_FI_DEV_FB_FREE", "legendFormat": "{{GPU}} FREE"}
          ]
        },
        {
          "title": "Power Draw (W)",
          "type": "timeseries",
          "gridPos": {"h": 6, "w": 8, "x": 16, "y": 0},
          "targets": [
            {"expr": "DCGM_FI_DEV_POWER_USAGE", "legendFormat": "{{GPU}}"}
          ]
        }
      ]
    }

The most important metric for FinOps: GPU utilization. If your A100s are running at under 30% sustained utilization, you're wasting $15,000+/year per GPU on idle capacity. The response isn't always "buy fewer GPUs" — batching, data loading pipeline optimization, and mixed-precision training can often raise utilization significantly without buying new hardware.

GPU Scheduling Beyond Default Behavior

The Kubernetes scheduler places GPU pods on nodes with available resources, but the default behavior doesn't account for GPU memory fragmentation, NUMA topology, or multi-instance GPU slices. Understanding these nuances matters at scale.

GPU Resource Filtering

By default, Kubernetes treats GPUs as opaque resources — it knows a node has 4 GPUs but not which GPUs are partially used or whether they're on the same NUMA node as available CPU cores. For latency-sensitive inference workloads, NUMA alignment is critical: GPU-to-CPU data transfers across NUMA boundaries add 2-5μs latency, which compounds at high request rates.

# Node affinity to co-locate GPU pods with CPU resources on the same NUMA node
apiVersion: v1
kind: Pod
metadata:
  name: latency-sensitive-inference
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: topology.kubernetes.io/numa-node
            operator: In
            values:
            - "0"   # Prefer NUMA node with GPU 0
  containers:
  - name: inference
    image: nvcr.io/nvidia/tritonserver:24.04-py3
    resources:
      limits:
        nvidia.com/gpu: "1"

Time-Slicing GPUs for Multiple Small Workloads

For development and staging environments, you can time-slice a single GPU across multiple small pods. This is NOT recommended for production inference (QoS suffers) but is useful for maximizing utilization of development GPUs.

# Configure time-slicing via Device Plugin config
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4    # 4 pods can share 1 GPU (round-robin)
EOF

# Point the Device Plugin at this config
helm upgrade gpu-operator nvdp/gpu-operator \
  --namespace gpu-operator \
  --set driver.enabled=false \
  --set toolkit.enabled=true \
  --set dcgmExporter.enabled=true \
  --set config.storageClass=nvidia.com/gpu

Production Patterns: Training vs. Inference

GPU workloads fall into two fundamentally different operational categories, and your scheduling strategy should reflect that.

Training Jobs: Throughput-Optimized

Training jobs are batch-compute workloads. They saturate GPU memory with large batch sizes, run for hours to days, and the primary metric is samples-per-second throughput. Multi-GPU training uses NCCL for gradient synchronization across nodes.

# Distributed PyTorch training job with Tensor Parallelism
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: distributed-training
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      template:
        spec:
          containers:
          - name: pytorch
            image:registry.example.com/pytorch-training:2.3
            resources:
              limits:
                nvidia.com/gpu: "8"   # 8-GPU master node
            env:
            - name: WORLD_SIZE
              value: "8"
            - name: MASTER_ADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          nodeSelector:
            node.kubernetes.io/gpu-count: "8"
    Worker:
      replicas: 3
      template:
        spec:
          containers:
          - name: pytorch
            image: registry.example.com/pytorch-training:2.3
            resources:
              limits:
                nvidia.com/gpu: "8"

Key scheduling consideration for training: Use podantiaffinity to co-locate all training workers on the same node group, minimizing NCCL network traffic across hosts. NCCL over bandwidth-constrained inter-node links is a common training throughput bottleneck.

Inference Serving: Latency-Optimized

Inference workloads serve requests in real-time. They need low latency, not maximum throughput, and they benefit from different optimizations: continuous batching, GPU memory pre-loading for model shards, and priority scheduling.

# vLLM inference deployment with GPU memory optimization
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-inference
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - "--model=meta-llama/Llama-3-70b-instruct"
        - "--tensor-parallel-size=2"        # Split model across 2 GPUs
        - "--gpu-memory-utilization=0.90"  # Reserve 10% for KV cache
        - "--max-num-batched-tokens=8192"  # Continuous batching
        resources:
          limits:
            nvidia.com/gpu: "2"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-inference-svc
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 8000
  selector:
    app: vllm-inference

Node Pools and GPU Resource Management

In multi-tenant or multi-workload clusters, GPU node pools prevent expensive GPU nodes from being consumed by non-GPU pods.

# node-pool-gpu-training.yaml
apiVersion: v1
kind: NodePool
metadata:
  name: gpu-training-pool
spec:
  minNodes: 2
  maxNodes: 10
  scaleTarget:
    cpu: 80%
  nodeSelector:
    node.kubernetes.io/gpu-pool: training
  taints:
  - key: nvidia.com/gpu
    value: "true"
    effect: NoSchedule   # Only GPU workloads schedule here
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  labels:
    gpu-type: A100-40GB
    gpu-count: "4"

Then add the toleration to your training job pod spec:

spec:
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"
  nodeSelector:
    node.kubernetes.io/gpu-pool: training

This setup prevents a runaway data preprocessing job from consuming GPU node hours when it doesn't actually need a GPU.

FinOps: Cutting GPU Waste

GPU compute is the most expensive compute in cloud infrastructure. An idle A100 costs $1.56/hour on-demand on AWS. Here's how production teams cut that waste:

Spot Instances for Fault-Tolerant Training

Training jobs are checkpointed — they're inherently fault-tolerant. Spot instances give 60-70% discounts over on-demand.

# Karpenter provisioner for GPU spot instances
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: gpu-training-spot
spec:
  requirements:
  - key: node.kubernetes.io/instance-type
    operator: In
    values:
    - g5.48xlarge    # A10G GPUs, 4x per instance
  - key: capacity.kubernetes.io/gpu-count
    operator: In
    values: ["4"]
  - key: karpenter.sh/capacity-type
    operator: In
    values: ["spot"]
  limits:
    cores: 64
  ttlSecondsAfterEmpty: 300
  provider:
    spotFleet: true

Right-Sizing with DCGM Utilization Data

After 2 weeks of DCGM metrics, most clusters show GPU utilization under 50%. The response isn't always "fewer GPUs" — it's often fixing the bottleneck:

• Low GPU util + high CPU util → data loading bottleneck, increase num_workers in DataLoader
• Low GPU util + low CPU util → model is too small for the GPU, batch size too small
• Low GPU util + high memory bandwidth → compute-bound kernel, try attention optimization (FlashAttention)

Troubleshooting Common GPU Operator Issues

Issue: Pods stuck in Pending with GPU resource requests

# Diagnose: check if any node advertises nvidia.com/gpu
kubectl describe nodes | grep nvidia.com/gpu

# If 0 on all nodes, the Device Plugin isn't working
kubectl logs -n gpu-operator -l app=nvidia-device-plugin
# Common fix: reboot nodes after driver install
# Or: nvidia-smi on the node returns "No devices were found"

Issue: DCGM Exporter not exposing metrics

# Check DCGM Exporter pod logs
kubectl logs -n gpu-operator -l app=dcgm-exporter

# Verify metrics endpoint
kubectl exec -n gpu-operator deploy/dcgm-exporter -- curl localhost:9400/metrics | head

# Check Prometheus scrape target
kubectl get endpoints dcgm-exporter -n gpu-operator

Issue: GPU pods OOM but VRAM shows available

This is a CUDA OOM vs. system OOM confusion. The GPU has its own memory manager. Set nvidia.com/gpu.memory requests if using MIG, and monitor DCGM_FI_DEV_FB_USED per pod via the DCGM Prometheus exporter's pod-level metrics:

# Enable pod-level GPU metrics
helm upgrade dcgm-exporter nvdp/dcgm-exporter \
  --namespace gpu-operator \
  --set podLevelResources.enabled=true

HAMi (CNCF Incubating, Jul 2026)

HAMi (Heterogeneous AI Computing Middleware) was promoted from CNCF sandbox to incubating on 2026-07-15, making it the canonical CNCF-governed answer to vendor-neutral GPU sharing. The project started in 2022 as k8s-vGPU-scheduler, was renamed to HAMi in 2023, joined CNCF as a sandbox in early 2024, and graduated to incubating this July. The incubating status matters for production adoption: it gives you a vendor-neutral governance path under the CNCF TOC, not a single-vendor roadmap dictated by NVIDIA's release cadence.

The integration with the stack you have already built is straightforward. HAMi sits next to the NVIDIA Device Plugin as an extended scheduler, advertises the additional resource types listed below, and respects DCGM and the GPU Operator's own feature discovery. You do not need to choose between the GPU Operator and HAMi — they coexist. HAMi installs its own hami-device-plugin DaemonSet alongside the existing NVIDIA one, and a scheduler extender that scores nodes based on memory oversubscription policies.

The four sharing primitives

HAMi exposes four orthogonal sharing primitives. Most clusters use a mix — MIG for inference latency SLAs, time-slicing for dev/staging, memory oversubscription for batch training, and vGPU only where NVIDIA vGPU licensing is already in place.

Primitive	HAMi resource	NVIDIA Device Plugin equivalent	Best for
MIG	nvidia.com/mig-1g.5gb	nvidia.com/mig-1g.5gb (same name)	Latency-bound inference with QoS guarantees
vGPU	nvidia.com/vgpu-1	Not exposed by the stock plugin — requires NVIDIA vGPU licensed driver	vSphere-style hard partitions, regulated workloads
Time-slicing	nvidia.com/gpu (overprovisioned via replicas)	nvidia.com/gpu overprovisioned via sharing.timeSlicing.replicas	Dev/staging, small inference pods, CI runners
Memory oversubscription	nvidia.com/hami-gpu with a memory request higher than the physical VRAM	Not supported — Device Plugin reports only physical memory	Batch training, eviction-tolerant workloads, max utilization on idle VRAM

Side-by-side config diff for time-slicing with four replicas:

# BEFORE — NVIDIA Device Plugin ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4

# AFTER — HAMi NodeConfig (per-node)
apiVersion: v1
kind: ConfigMap
metadata:
  name: hami-scheduler-config
  namespace: hami-system
data:
  hami-scheduler-policy.json: |
    {
      "MigResource": ["nvidia.com/mig-1g.5gb"],
      "ResourceExist": ["nvidia.com/gpu", "nvidia.com/hami-gpu"],
      "ResourceDefault": "hami-gpu",
      "DefaultMem": 0,
      "DefaultCores": 0,
      "CustomDeviceConfig": [
        {"DeviceName": "NVIDIA H100", "MemRange": [4096, 81920], "CoresRange": [1, 100]},
        {"DeviceName": "NVIDIA A100", "MemRange": [1024, 40960], "CoresRange": [1, 100]}
      ]
    }

Migration cookbook: nvidia.com/gpu → nvidia.com/hami-gpu

Switching existing workloads is a label-and-restart job, not a rewrite. The scheduler extender honors requests to both nvidia.com/gpu (legacy, routes to NVIDIA Device Plugin) and nvidia.com/hami-gpu (routes to HAMi's oversubscription-aware scheduler).

# Step 1: install HAMi (alongside the existing GPU Operator)
helm repo add hami-charts https://project-hami.github.io/HAMi
helm install hami hami-charts/hami \
  --namespace hami-system --create-namespace

# Step 2: label workloads you want to opt into oversubscription
# (untouched pods keep using nvidia.com/gpu with no migration risk)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: batch-trainer
spec:
  template:
    metadata:
      labels:
        hami-scheduler: "true"        # Tells HAMi to manage this workload
    spec:
      containers:
      - name: trainer
        resources:
          limits:
            nvidia.com/hami-gpu: "1"  # Was: nvidia.com/gpu: 1
            nvidia.com/hami-gpu-mem: "20000"   # Request 20 GiB; HAMi enforces eviction
            nvidia.com/hami-gpu-cores: "50"    # Optional, 0-100 percentage

Run the migration in three waves: (1) dev/staging namespaces first to validate the eviction policy behaves as expected, (2) internal batch jobs second because they tolerate mid-task eviction, (3) customer-facing inference last and only after two weeks of clean eviction-policy behavior in staging. The gotcha is hami-gpu-mem: if you set it higher than the physical VRAM, HAMi will oversubscribe and evict under pressure. If you set it to the physical total, you have not gained anything over the bare Device Plugin.

Heterogeneous scheduler: one manifest, three vendors

The killer feature is that HAMi abstracts the device model. A single pod spec can request nvidia.com/hami-gpu and the scheduler extender picks the right backing primitive on whatever hardware the chosen node has:

# Heterogeneous AI cluster — NVIDIA + AMD + Ascend in one pool
# Node A: H100 80GB (NVIDIA MIG + vGPU)
# Node B: MI300X 192GB (AMD ROCm + MPS)
# Node C: Ascend 910B 64GB (Huawei CANN)

# Same pod spec schedules across all three:
apiVersion: v1
kind: Pod
metadata:
  name: mixed-fleet-inference
spec:
  nodeSelector:
    hami-scheduler: "true"
  containers:
  - name: inference
    image: your-inference-image
    resources:
      limits:
        nvidia.com/hami-gpu: "1"      # or AMD/Ascend equivalent
        nvidia.com/hami-gpu-mem: "8192"
        nvidia.com/hami-gpu-cores: "30"
      env:
      - name: HAMI_DEVICE_VENDOR
        valueFrom:
          fieldRef:
            fieldPath: metadata.labels['gpu-vendor']   # 'nvidia' / 'amd' / 'ascend'

This is why HAMi's CNCF promotion matters for AWS Trainium / Inferentia adopters specifically — the same scheduler pattern will extend to custom silicon in the same heterogeneous pool. For workloads that need to co-schedule across the mixed fleet (a common LLM serving pattern when one silicon is over-subscribed), see Kubernetes GPU scheduling for ML workloads for the topology-aware extender config.

Observability: DCGM per-slice, OTel gpu.*, per-tenant billing

HAMi extends the existing observability layer rather than replacing it. Three hooks matter:

• DCGM per-slice metrics. Each HAMi-allocated slice gets a synthetic GPU UUID that DCGM Exporter picks up automatically. Existing DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_FB_USED panels just work — you get per-tenant visibility for free. Filter by the HAMI_DEVICE_UUID env var to attribute slices back to a pod.
• OTel gpu.* resource attributes. The HAMi scheduler extender stamps pods with gpu.vendor, gpu.model, gpu.memory.bytes, and gpu.slice.id resource attributes. With the OTel Collector and the resourcedetection processor on, these flow into every span your training loop emits — invaluable when correlating a stalled epoch to the actual GPU allocation.
• Per-tenant billing. Combine HAMi's HAMI_DEVICE_UUID + Kubernetes namespace labels + your chargeback system to bill tenants by actual GPU-seconds consumed (not just pod-wall-time). For a multi-tenant cluster running shared inference, this is the difference between billable hours that customers believe and billable hours that finance can reconcile. See Kubernetes cost optimization for the per-namespace rate model; the LLM cost arithmetic that drives the per-request side is the same calculation our LLM API cost calculator does on the inference-token side.

The FinOps question HAMi answers most cleanly: should I buy another H100, or oversubscribe the existing ones? On an A100 80GB running five inference replicas at 30% utilization each, HAMi's memory oversubscription can collapse three of those replicas onto one physical card with eviction-policy QoS — turning a $30k/year GPU purchase into a $0/year scheduling change. The math is in the same per-tenant-spend ledger that powers FinOps for Kubernetes GPU; HAMi is the mechanism, not the model.

Summary

GPU operators on Kubernetes are mature enough for production, but the operational complexity is non-trivial. The key takeaways:

1. GPU Operator manages the stack automatically — use it instead of installing components manually
2. DCGM Exporter is non-negotiable for production — you can't FinOps what you can't measure
3. MIG is for inference serving latency guarantees — use full GPUs for training throughput
4. Spot instances + Karpenter cut GPU costs 60-70% for fault-tolerant training workloads
5. NUMA alignment matters for latency-sensitive inference — don't ignore it at scale
6. Node pools with taints prevent GPU waste on non-GPU workloads
7. HAMi (CNCF incubating since Jul 2026) is now the canonical CNCF-governed layer for vendor-neutral GPU sharing — memory oversubscription, MIG, vGPU, and time-slicing under one scheduler, with first-class hooks for DCGM per-slice metrics, OTel gpu.* resource attributes, and per-tenant billing. It runs alongside the GPU Operator, not in place of it.

The gap between "GPU cluster that runs" and "GPU cluster that's production-ready" is DCGM monitoring, proper node pool isolation, and right-sized resource requests. Get those three right and you'll be ahead of most teams running AI on Kubernetes.