Technical solution

NVIDIA GPU Operator

On Kubernetes clusters we manage every GPU node through the NVIDIA GPU Operator. It installs and reconciles the whole NVIDIA software stack as containers, so a new node joins the cluster ready to run GPU workloads and a failed component is restored automatically.

What the Operator manages

Component	Role
GPU driver	Kernel driver delivered as a container, or a pre-installed host driver when the customer's OS policy requires it.
Container Toolkit	Lets containerd start containers with GPU access.
Device Plugin	Advertises `nvidia.com/gpu` (and MIG profiles) to the Kubernetes scheduler.
GPU Feature Discovery + NFD	Labels nodes with GPU model, memory, driver and CUDA versions so jobs land on the right hardware.
MIG Manager	Applies Multi-Instance GPU layouts from a node label.
DCGM Exporter	Publishes GPU telemetry to Prometheus.
Validator	Runs CUDA checks after every install or upgrade before the node accepts work.

Reference install

We pin the chart version in Git and install with Helm. The version below is a placeholder. Each project uses the release validated for its GPUs, OS and kernel.

shell

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --version <validated-version> \
  --set mig.strategy=mixed \
  --set dcgmExporter.serviceMonitor.enabled=true \
  --wait

Common variations:

--set driver.enabled=false when the host OS already ships a validated NVIDIA driver.
--set toolkit.enabled=false when the container toolkit is baked into the node image.
Registry overrides (for example driver.repository) for offline sites. See images & model artifacts.

Verify a node

shell

kubectl get pods -n gpu-operator
kubectl get nodes -o custom-columns=NODE:.metadata.name,GPUS:.status.allocatable.nvidia\.com/gpu

kubectl run smi --rm -it --restart=Never \
  --image=nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04 \
  --overrides='{"spec":{"containers":[{"name":"smi","image":"nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04","command":["nvidia-smi"],"resources":{"limits":{"nvidia.com/gpu":1}}}]}}'

Sharing GPUs: MIG and time-slicing

Multi-Instance GPU (hard isolation)

On Hopper and Blackwell GPUs, MIG splits one GPU into isolated slices with their own memory and compute. We use it for multi-tenant inference and notebooks. The layout is set per node with a label, and MIG Manager applies it:

shell

# example: seven 1g.10gb slices on each H100 80GB in this node
kubectl label node gpu-node-01 nvidia.com/mig.config=all-1g.10gb --overwrite

Time-slicing (soft sharing)

For development clusters where isolation matters less than density, the device plugin can advertise each GPU several times:

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4

shell

kubectl patch clusterpolicies.nvidia.com/cluster-policy --type merge \
  -p '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config","default":"any"}}}}'

Time-slicing gives no memory isolation. One job can exhaust GPU memory for every other job on that GPU. We do not use it for production tenants.

Multi-node fabric

For training across nodes we pair the GPU Operator with the NVIDIA Network Operator, which configures InfiniBand or RoCE NICs and GPUDirect RDMA. Acceptance includes an NCCL all-reduce test across all nodes with the measured bus bandwidth recorded in the handover report.

Upgrades

Chart and driver versions live in Git. Every change is reviewed and tested on one node first.
Driver upgrades roll one node at a time: cordon, drain, upgrade, validate, uncordon.
We follow NVIDIA's support matrix for each GPU generation and Kubernetes version and don't skip validation.

Monitoring and alerts

DCGM Exporter feeds Prometheus and Grafana. Default alerts we ship:

Signal	DCGM field	Why it matters
XID errors	`DCGM_FI_DEV_XID_ERRORS`	Driver-reported GPU faults. Some codes mean the GPU needs a reset or RMA.
Temperature	`DCGM_FI_DEV_GPU_TEMP`	Cooling problems show here before throttling hurts throughput.
Utilisation	`DCGM_FI_DEV_GPU_UTIL`	Finds idle GPUs and scheduling gaps.
Memory used	`DCGM_FI_DEV_FB_USED`	Capacity planning and out-of-memory investigations.

技术方案

NVIDIA GPU Operator

在 Kubernetes 集群上,我们通过 NVIDIA GPU Operator 统一管理所有 GPU 节点。它以容器方式安装并持续校正整套 NVIDIA 软件栈:新节点加入集群即可运行 GPU 任务,组件故障时会自动恢复。

Operator 管理哪些组件

组件	作用
GPU 驱动	以容器方式交付内核驱动;客户操作系统有要求时,也可使用宿主机预装驱动。
Container Toolkit	让 containerd 启动的容器可以访问 GPU。
Device Plugin	向 Kubernetes 调度器上报 `nvidia.com/gpu` 及 MIG 规格。
GPU Feature Discovery + NFD	给节点打上 GPU 型号、显存、驱动与 CUDA 版本标签,让任务调度到合适的硬件。
MIG Manager	根据节点标签应用 MIG 切分方案。
DCGM Exporter	把 GPU 监控指标发布到 Prometheus。
Validator	每次安装或升级后先跑 CUDA 校验,通过后节点才接收任务。

参考安装方式

Chart 版本固定在 Git 中,通过 Helm 安装。下面的版本号是占位符,每个项目使用针对其 GPU、操作系统与内核验证过的版本。

shell

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --version <validated-version> \
  --set mig.strategy=mixed \
  --set dcgmExporter.serviceMonitor.enabled=true \
  --wait

常见变体:

宿主机已预装经验证的 NVIDIA 驱动时:--set driver.enabled=false
节点镜像已内置容器工具包时:--set toolkit.enabled=false
离线机房需改用本地镜像仓库(如 driver.repository),见镜像与模型管理。

验证节点

shell

kubectl get pods -n gpu-operator
kubectl get nodes -o custom-columns=NODE:.metadata.name,GPUS:.status.allocatable.nvidia\.com/gpu

kubectl run smi --rm -it --restart=Never \
  --image=nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04 \
  --overrides='{"spec":{"containers":[{"name":"smi","image":"nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04","command":["nvidia-smi"],"resources":{"limits":{"nvidia.com/gpu":1}}}]}}'

GPU 共享:MIG 与时间片

MIG 多实例(硬隔离)

Hopper 与 Blackwell GPU 支持 MIG,可把一张卡切成多个拥有独立显存与算力的实例,适合多租户推理与开发环境。按节点打标签即可,由 MIG Manager 执行:

shell

# 示例:把该节点每张 H100 80GB 切成 7 个 1g.10gb 实例
kubectl label node gpu-node-01 nvidia.com/mig.config=all-1g.10gb --overwrite

时间片(软共享)

开发集群更看重密度而非隔离时,可让 Device Plugin 把每张卡上报为多份:

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4

shell

kubectl patch clusterpolicies.nvidia.com/cluster-policy --type merge \
  -p '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config","default":"any"}}}}'

时间片没有显存隔离。一个任务可能占满显存,影响同卡上的其他任务。生产租户不使用时间片。

多节点互联

跨节点训练时,我们把 GPU Operator 与 NVIDIA Network Operator 配合使用,由后者配置 InfiniBand 或 RoCE 网卡及 GPUDirect RDMA。验收包含全节点 NCCL all-reduce 测试,实测总线带宽写入交接报告。

升级策略

Chart 与驱动版本保存在 Git 中,每次变更先经评审,并在单个节点上测试。
驱动逐节点滚动升级:cordon → drain → 升级 → 校验 → uncordon。
严格遵循 NVIDIA 针对各代 GPU 与 Kubernetes 版本的支持矩阵,不跳过校验。

监控与告警

DCGM Exporter 接入 Prometheus 与 Grafana。默认告警:

指标	DCGM 字段	为什么重要
XID 错误	`DCGM_FI_DEV_XID_ERRORS`	驱动上报的 GPU 故障,部分错误码意味着需要重置或返修。
温度	`DCGM_FI_DEV_GPU_TEMP`	散热问题会在降频影响吞吐之前先在这里出现。
利用率	`DCGM_FI_DEV_GPU_UTIL`	发现闲置 GPU 与调度空档。
显存占用	`DCGM_FI_DEV_FB_USED`	容量规划与显存溢出排查。