One platform, every GPU cloud. Deploy H100s to B300s across multiple providers. On-demand, no lock-in, from a single dashboard.
Remote
Joined January 2023
- Tweets6.6K
- Following13
- Followers117K
- Likes4.8K
You're renting a GPU in a data center you've never seen. One question matters more than price: can the people running that rack read your data?
NVIDIA TEE encrypts the GPU's memory so the operator running that server can't read your model weights or inputs while the job runs. But TEE can't vet the building. That's a separate guarantee: Tier 3+ certification.
Before trusting a TEE-enabled node:
- Confirm TEE is enabled on the node, not just available on the chip
- Confirm the facility is Tier 3+ certified
- Confirm the node came from a certified partner, not a reseller
- Confirm you can audit both before data leaves your network
We've sourced exactly this: confidential inference on H200, NVIDIA TEE enabled, from a Tier 3+ certified partner.
Running inference you can't expose to the host? Request a quote for TEE-enabled nodes.
DeepSeek R2 emits up to 40,000 thinking tokens where a standard LLM emits about 400. That's 100x the KV cache pressure, and it's the actual engineering problem behind serving a reasoning model, not the parameter count.
MLA compression is what makes it survivable: roughly 512 dimensions plus a 64-dim RoPE key instead of full per-head attention. Over a 30,000-token reasoning chain at FP16, that's about 2.1 GB of KV cache through MLA versus 15+ GB for a comparable GQA architecture.
Full precision needs 8x H200. FP8 fits the same workload on 8x H100.
A100 has no FP8 hardware support at all. H100 does, and it's the single biggest reason the fine-tuning gap is so lopsided: about 2.1x faster than A100 on BF16 Llama 3 70B fine-tuning, 3.4x once you move to FP8.
Memory bandwidth tells a smaller but real story too: 3,350 GB/s on H100 versus 2,039 on A100, a 1.64x gap that shows up directly in memory-bound inference. NVLink 4.0 on H100 runs 900 GB/s bidirectional against A100's 600 GB/s, which shows up as gradient-sync overhead once you scale past a few GPUs.
An inference team serving two production models was paying for idle GPU-hours every night their traffic dropped.
They moved both services onto Spheron's Tier 3+ certified network, kept the 99.9% uptime SLA they already had, and switched from a fixed monthly reservation to per-minute billing.
No migration to a new framework, no change to their serving stack, full root access on the same GPU class they were already running.
The traffic pattern didn't change. The bill for the hours nobody was calling the model did.
A 128K-token system prompt on Llama 3.1 70B took 11.2 seconds to first token, cold. Sharing the KV cache across nodes cut that to 1.5.
That's LMCache running alongside vLLM: a 95%+ cache hit rate against roughly 60% for vLLM's own prefix cache alone.
At 200 concurrent requests with 80% prompt reuse, throughput on a 4x H100 cluster goes from 1.4 req/s to 21.3, a 15.2x gain.
It needs real infrastructure to get there: 2-4TB of NVMe per node at 6+ GB/s sequential read, plus 256-512GB of CPU DRAM per node. A different place to spend the budget, not a free lunch.
Three ways to pay for the same GPU, and most teams only know about one:
- On-demand, billed per minute, no commitment, cancel anytime
- Spot, priced below on-demand for interruptible workloads
- Reserved, for teams that know their utilization months out and want the lowest rate locked in
Add stablecoin settlement (USDC or USDT) on top and a finance team negotiating a reserved commitment isn't waiting on a wire transfer or a currency conversion to lock in capacity. The billing model should match the workload, not the other way around.
Same model, same GPU, same quantization. Different kernel, 10.9x different speed.
Qwen2.5-32B-Instruct, AWQ, H200:
- Marlin kernel: 741 tokens/s
- vLLM default AWQ kernel: 68 tokens/s
Cost per million output tokens drops from $19.61 to $1.80 with Marlin.
The weights are identical. Only the code running them changed.
Full numbers (plus GPTQ and GGUF) here
Running AWQ or GPTQ in prod? Tell us your stack, we'll check the config.
MI300X: 192 GB HBM3, 1,307 TFLOPS dense FP16. H200: 141 GB HBM3e, about 989 TFLOPS dense. AMD wins capacity and raw dense compute on paper.
Serving DeepSeek R1 offline flips the practical result: H200 does 6,311 tokens/sec, MI300X does 4,574. The deciding factor isn't the spec sheet, it's whether your model and KV cache fit comfortably inside 141 GB. If it does, H200's software maturity wins. If it doesn't, MI300X's extra 51 GB is the only option on the table.
4-bit quantization doesn't mean 4x cheaper. The kernel you pick changes cost by 10x on its own.
On H200 serving Qwen2.5-32B, AWQ with the Marlin kernel hits 741 tokens/sec; vLLM's default AWQ kernel on the same weights: 68. Same format, 10.9x apart. GPTQ Marlin reaches 712 vs 277 on GPTQ's default kernel.
Cost per million output tokens on H200 at $4.80/hr: AWQ Marlin ~$1.80, GPTQ Marlin ~$1.87, default AWQ kernel ~$19.61.
GGUF Q4_K_M holds closest to FP16 quality; GPTQ Marlin costs 10.4 HumanEval points. Pick the format for your accuracy floor, then pick Marlin regardless.
PCIe Gen5 x16 moves data CPU to GPU at about 64 GB/s. NVLink moves data GPU to GPU at 900 GB/s. A 14x gap, and it's the reason multi-GPU training either scales cleanly or doesn't.
Gradient sync, activation checkpointing, tensor-parallel shards: all of it crosses the GPU-to-GPU link, not the CPU link. Put a job on hardware where that path runs through PCIe switches instead of NVLink and the wall clock tells you immediately.
Cheapest serving kernel isn't the same question as best quality, and the quantization comparison makes that obvious. FP16 baseline perplexity is 6.56 with 56.1% on HumanEval.
GGUF Q4_K_M holds closest: 6.74 perplexity, 51.8% HumanEval. GPTQ Marlin, despite being the fastest and cheapest per million tokens, lands worst: 6.97 perplexity and 45.7% HumanEval, a 10.4-point drop from baseline.
The format that wins your latency benchmark can be the one that costs you the most accuracy. Check both numbers before you pick one for production.
