Technical solution
Images & model artifacts
GPU clusters pull large images (CUDA, PyTorch, driver containers) and even larger model weights. We keep both inside your environment, so nodes start fast, builds are reproducible, and nothing unverified reaches production.
Design
| Layer | What we deploy |
|---|---|
| Registry | Harbor (CNCF, open source) on your hardware in Singapore, with SSO and per-team projects. |
| Proxy caches | Pull-through projects for nvcr.io, docker.io, ghcr.io, quay.io and registry.k8s.io. Each image is fetched from the internet once. |
| Scanning | Trivy scans every pushed image. Critical CVEs block promotion to the production project. |
| Signing | Images are signed with cosign in CI. An admission policy (Kyverno) only admits signed images in production namespaces. |
| Retention | Rules keep the last N releases per repository and remove untagged layers on a schedule. |
| Model storage | S3-compatible object storage (MinIO or Ceph RGW) with versioned buckets and SHA-256 checksums for every weight file. |
Point containerd at the cache
Workloads keep their normal image names. containerd on each node resolves them through Harbor:
server = "https://nvcr.io" [host."https://registry.example.internal/v2/nvcr-proxy"] capabilities = ["pull", "resolve"] override_path = true
The same pattern repeats for each upstream registry. If Harbor is down, containerd falls back to the upstream server, unless the site is fully offline.
Image rules we enforce
- Production manifests reference images by digest (
@sha256:…), never by a moving tag such aslatest. - Base images come from NVIDIA NGC (CUDA, PyTorch, Triton) or an approved internal base, rebuilt monthly for security patches.
- Each team pushes with its own robot account. Nobody shares the admin password.
Model weights
- Fetch once. Weights are downloaded from the source (for example Hugging Face) by a controlled job. Access tokens live in a secrets manager, not in scripts.
- Record. We store the model name, revision, licence and SHA-256 of every file.
- Store. Files go to a versioned bucket in your object storage.
- Serve. Inference pods load weights from the bucket or a node-local cache, never directly from the internet.
Offline (air-gapped) sites
For sites without internet access we prepare a transfer bundle outside the site and import it through your approved channel:
# copy a pinned image set into the site registry skopeo sync --src yaml --dest docker images.yaml registry.example.internal/mirror # verify a model file after transfer sha256sum -c model.sha256
The GPU Operator chart is then pointed at the site registry with its *.repository values. Driver images are matched to the exact OS and kernel on your nodes.
技术方案
镜像与模型管理
GPU 集群要拉取体积很大的镜像(CUDA、PyTorch、驱动容器),模型权重更大。我们把两者都留在你的环境内:节点启动快、构建可复现,未经验证的内容进不了生产环境。
整体设计
| 层 | 部署内容 |
|---|---|
| 镜像仓库 | Harbor(CNCF 开源项目),部署在你位于新加坡的服务器上,支持 SSO,按团队划分项目。 |
| 代理缓存 | 为 nvcr.io、docker.io、ghcr.io、quay.io、registry.k8s.io 建立代理项目,每个镜像只从公网拉取一次。 |
| 漏洞扫描 | Trivy 扫描所有推送的镜像,存在严重漏洞的镜像不能进入生产项目。 |
| 镜像签名 | CI 中用 cosign 签名,准入策略(Kyverno)只允许已签名的镜像进入生产命名空间。 |
| 保留策略 | 每个仓库保留最近 N 个版本,定期清理无标签的镜像层。 |
| 模型存储 | S3 兼容对象存储(MinIO 或 Ceph RGW),桶开启版本管理,每个权重文件记录 SHA-256 校验值。 |
让 containerd 走缓存
业务侧的镜像名称不用改,由各节点上的 containerd 经 Harbor 解析:
server = "https://nvcr.io" [host."https://registry.example.internal/v2/nvcr-proxy"] capabilities = ["pull", "resolve"] override_path = true
其他上游仓库照此配置。Harbor 故障时,containerd 会回退到上游 server(完全离线的机房除外)。
镜像规范
- 生产环境清单一律按摘要(
@sha256:…)引用镜像,不使用latest这类会变化的标签。 - 基础镜像来自 NVIDIA NGC(CUDA、PyTorch、Triton)或经批准的内部基础镜像,每月重建以打安全补丁。
- 每个团队用各自的机器人账号推送镜像,不共用管理员密码。
模型权重
- 只下载一次。由受控任务从源头(如 Hugging Face)下载;访问令牌存放在密钥管理系统中,不写进脚本。
- 登记。记录模型名称、版本、许可证以及每个文件的 SHA-256。
- 存储。文件存入对象存储中开启版本管理的桶。
- 加载。推理 Pod 从桶或节点本地缓存加载权重,不直接访问公网。
离线(物理隔离)机房
机房无法联网时,我们在外部准备好传输包,再通过你批准的渠道导入:
# 把固定版本的镜像集同步到机房内仓库 skopeo sync --src yaml --dest docker images.yaml registry.example.internal/mirror # 传输后校验模型文件 sha256sum -c model.sha256
随后通过 *.repository 参数让 GPU Operator 使用机房内仓库;驱动镜像与节点的操作系统和内核版本精确匹配。