Technical solution

Model serving

We run open or custom models on your GPUs behind one OpenAI-compatible endpoint, so your applications switch from a public API by changing a base URL.

Stack

vLLM for most LLMs: paged KV cache, continuous batching, prefix caching, tensor and pipeline parallelism.
Triton Inference Server or TensorRT-LLM when a project needs mixed model types or maximum throughput on NVIDIA hardware.
Gateway in front: per-team API keys, rate limits, usage metering and request logs kept inside your environment.
Autoscaling on queue depth, with a minimum warm replica so the first request isn't slow.

Sizing: start from memory

Model weights need roughly parameters × bytes per parameter of GPU memory. The KV cache for context and concurrency comes on top, often 20–60% more.

Model	Precision	Weights	Typical fit
8B	BF16	≈ 16 GB	1 GPU, or a MIG slice for light traffic
70B	FP8	≈ 70 GB	1× H200 141 GB, or 2× H100 80 GB
70B	BF16	≈ 140 GB	2× H200 or 4× H100 (TP=2 / TP=4)
405B	FP8	≈ 405 GB	One 8-GPU H200 or B200 node (TP=8)

Final sizing comes from a benchmark with your prompts, context lengths and concurrency, not from this table.

Reference launch

shell

vllm serve /models/llama-3.1-70b-instruct-fp8 \
  --served-model-name llama-70b \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --port 8000

Applications call it like any OpenAI-compatible API:

shell

curl https://llm.example.internal/v1/chat/completions \
  -H "Authorization: Bearer $TEAM_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-70b","messages":[{"role":"user","content":"Hello"}]}'

What we measure at acceptance

Metric	Meaning
TTFT	Time to first token. What users feel as responsiveness.
Inter-token latency	Time between streamed tokens after the first one.
Throughput	Output tokens per second across all concurrent requests.
p95 / p99 latency	Tail latency at your target concurrency.

Targets for each metric are written into the statement of work before deployment starts.

技术方案

模型推理服务

我们在你的 GPU 上运行开源或自研模型,并统一提供 OpenAI 兼容接口。你的应用从公有 API 迁移过来,只需改一个 base URL。

技术栈

vLLM:适用于大多数 LLM,支持分页 KV cache、连续批处理、前缀缓存、张量并行与流水线并行。
Triton Inference Server 或 TensorRT-LLM:项目需要多种模型类型并存,或在 NVIDIA 硬件上追求极限吞吐时使用。
统一网关:按团队发放 API Key、限流、用量计量,请求日志保留在你的环境内。
弹性扩缩:按队列深度扩缩,并保留至少一个预热副本,避免首个请求过慢。

容量规划:从显存算起

模型权重大约需要 参数量 × 每参数字节数 的显存,上下文与并发所需的 KV cache 另算,通常再多 20–60%。

模型	精度	权重	常见配置
8B	BF16	≈ 16 GB	1 张卡;流量小时可用一个 MIG 实例
70B	FP8	≈ 70 GB	1× H200 141 GB,或 2× H100 80 GB
70B	BF16	≈ 140 GB	2× H200 或 4× H100(TP=2 / TP=4)
405B	FP8	≈ 405 GB	一台 8 卡 H200 或 B200 节点(TP=8)

最终配置以你的真实提示词、上下文长度与并发压测结果为准,而不是这张表。

参考启动命令

shell

vllm serve /models/llama-3.1-70b-instruct-fp8 \
  --served-model-name llama-70b \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --port 8000

应用侧像调用任何 OpenAI 兼容 API 一样使用:

shell

curl https://llm.example.internal/v1/chat/completions \
  -H "Authorization: Bearer $TEAM_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-70b","messages":[{"role":"user","content":"Hello"}]}'

验收指标

指标	含义
TTFT	首 token 延迟,即用户感受到的响应速度。
Token 间延迟	流式输出时,首个 token 之后每个 token 的间隔。
吞吐	所有并发请求合计每秒输出的 token 数。
p95 / p99 延迟	目标并发下的长尾延迟。

各项指标的目标值在部署开始前写入 SOW。