Pinned Tweet
Launching Now: On-Demand Access to H100s,5090, H200s, B200s & more
No more waiting lists.
No long-term contracts.
No vendor lock-ins.
Just instant access to the GPUs you need—from SXM5 InfiniBand to PCIe—all in one place.
⚡ Spin up 100+ H100s or reserve a few H200s for training
🧠 Choose across providers without switching tabs
💸 Pay only when you run workloads
Your infra team’s dream stack is live.
Fire up your first cluster now
app.spheron.ai
On Llama 3 70B at batch 16, H100 running W4A8 generates 2,900 tokens/s. A100 running INT4 manages 1,100 tokens/s, a 2.6x gap that has nothing to do with raw bandwidth and everything to do with H100's FP8 tensor cores, hardware A100 simply doesn't have.
W4A8 needs vLLM 0.4+, TensorRT-LLM 0.10+, or PyTorch 2.1+ with transformer_engine. If your serving stack is still on an older vLLM build, you're leaving most of that throughput on the table regardless of which GPU you rent.
1
1
63
MI300X ships 192GB of HBM3 on a single GPU. A model that needs 2x H100 to fit now fits on one card, which changes your tensor-parallelism math before you've written a line of ROCm-specific code.
The tradeoff shows at the edges: FlashAttention 3 is Hopper-and-up only, so MI300X runs Flash Attention 2 through MIOpen instead, and TensorRT-LLM has no ROCm port planned, full stop. At batch 64-128 the H100-vs-MI300X throughput gap narrows to 5-10% anyway, so the capacity advantage often outweighs the missing kernel support for memory-bound serving.
1
79
The bandwidth ladder most people never see written down:
- DDR4 dual-channel: ~50 GB/s
- DDR5 quad-channel: ~90 GB/s
- PCIe Gen5 x16 (CPU to GPU): ~64 GB/s
- RTX 4090 GDDR6X: ~1,008 GB/s
- NVLink (GPU to GPU): 900 GB/s
- H100 HBM3: 3,350 GB/s
- H200 HBM3e: 4,800 GB/s
- B200 HBM3e: 8,000 GB/s
Every step up this ladder is a different bottleneck moving out of the way. Know which one your workload is actually hitting before you upgrade the wrong component.
1
82
A100 and H100 both ship 80GB of VRAM and support up to 7 MIG instances. If you're choosing based on capacity, there's no difference to choose on.
The real split is precision support: H100 has FP8 tensor cores, A100 doesn't. But getting W4A8 working on H100 needs vLLM 0.4+, TensorRT-LLM 0.10+, or PyTorch 2.1+ with transformer_engine installed. It doesn't come free with the hardware.
Match your framework version to the card before you benchmark anything.
2
3
148
Quantizing a reasoning model isn't a free lunch, and the size of the bill depends entirely on which precision you pick.
FP8 on a reasoning-model-class workload costs under 1% accuracy on MATH-500 and under 2% on AIME, for a 1.6-1.9x throughput gain over FP16. Drop to INT4 and the accuracy hit jumps to 6-12% on MATH-500 and 10-18% on AIME.
For reasoning chains, FP8 is the quantization floor. INT4 trades too much reasoning quality for the extra speed.
2
85
Same model, same precision, one generation jump: H200 hits 31,712 tokens/s serving Llama 2 70B. H100 hits 22,290 on the identical benchmark.
That's 42% more throughput from the HBM3e upgrade alone, 4,800 GB/s against H100's 3,350 GB/s. Nothing about the compute changed, the memory did.
If your serving workload is bandwidth-bound, the generation jump is the upgrade, not a bigger batch size.
1
3
110
H100 beats MI355X by 20-30% at batch 1-4. Push to batch 64-128 and that gap shrinks to 5-10%.
At batch 32 on Llama 3.1 70B FP16, estimated numbers from MLPerf data land close: MI355X near 18,000 tokens/s against H100 SXM5 near 19,500. The CUDA advantage is real at low concurrency and mostly disappears once you're running production-size batches.
If your workload runs small batches, the ecosystem gap matters more than the benchmark footnote suggests.
2
84
A 30,000-token reasoning chain on a 61-layer model needs about 2.1GB of KV cache with MLA compression. The same chain on a comparable GQA architecture needs 15GB or more, over 7x the memory for the same context.
MLA gets there by storing roughly a 512-dimension vector plus a 64-dim RoPE key per token, instead of full per-head key and value matrices.
That's the difference between fitting a long reasoning trace on one card and needing to shard it across several.
L40S's Transformer Engine promises 2x L40's FP8 throughput on paper. Real benchmarks land at 1.5-1.8x, because bandwidth saturates before the extra compute gets used.
Same 48GB of GDDR6 ECC at 864 GB/s on both cards. The real split: L40S hits 362 TFLOPS dense FP16, L40 hits 181. Serving Llama 3.1 8B at batch 8, L40S in FP8 reaches 504 tokens/s against L40's 168 tokens/s in FP16.
Pick L40S for FP8-quantized 7B-13B inference.
Pick L40 only when you're power-constrained and don't need FP8.
1
4
128
H100's bandwidth advantage over A100 looks massive on paper: 3,350 GB/s vs 2,039 GB/s, a 1.64x gap. In practice that number does a lot of work for memory-bound inference, and almost none of it on the PCIe variant.
H100 PCIe ships HBM2e at 2,000 GB/s, which narrows the advantage over A100 PCIe to about 1.03x. The real gap shows up in FP8: A100 has no FP8 tensor hardware at all, while H100 hits 3,958 TFLOPS FP8, which is why H100 runs Llama 3 70B fine-tuning 3.4x faster with FP8 enabled.
If you're buying for raw bandwidth, check which variant you're actually getting. If you're buying for FP8 throughput, SXM vs PCIe barely matters.
3
100
DeepSeek R2 can emit up to 40,000 thinking tokens where a standard LLM emits about 400. That's 100x the KV cache pressure before you've generated a single answer token.
MLA compression is why this is servable: it stores roughly a 512-dimension vector plus a 64-dim RoPE key per token instead of full per-head matrices. Run the math on a 61-layer model over a 30,000-token reasoning chain and MLA needs about 2.1GB of KV cache. A comparable GQA architecture needs 15GB or more for the same chain.
Full precision still needs 8x H200 SXM5 (1,128GB) to hold the model. FP8 brings that down to 8x H100 SXM5 (640GB), with the checkpoint under 600GB.
1
1
83
Shared system RAM tops out at 50-100 GB/s. Dedicated VRAM runs 2,000-4,800 GB/s. That's a 20-100x gap, and it decides whether your model spills gracefully or crawls.
An M4 Max with unified memory hit 28 tokens/s on Llama 3 70B at 4-bit. An RTX 4090 spilling the same model into system RAM managed 10 tokens/s, less than half the speed, despite GDDR6X bandwidth around 1,008 GB/s when it isn't spilling.
Once a model exceeds dedicated VRAM, wall-clock training time stretches 2-5x. Check your model's footprint against your card's VRAM before a run, not after it OOMs three epochs in.
69
Same model, same quantization, one machine runs 2.8x faster, and it's not the GPU with more raw compute that wins.
Llama 3 70B at 4-bit: an M4 Max on unified memory hits 28 tokens/s. An RTX 4090 spilling into system RAM manages 10 tokens/s on the identical model. The 4090 has far more raw FLOPS. It loses because once weights don't fit in dedicated VRAM, you're bottlenecked by system RAM bandwidth, not compute.
Size the model to the GPU's VRAM first. Compute is the second question.
2
2
115
NVIDIA's L40S Transformer Engine promises roughly 2x FP8 throughput over L40 (733 TFLOPS dense FP8 vs 362 TFLOPS dense). Real workloads don't hit that multiplier.
Both cards carry the same 48GB GDDR6 ECC at 864 GB/s, so the gap is pure compute, not memory. On Llama 3.1 8B at batch size 8, L40S FP8 reaches 504 tokens/s against L40 FP16's 168 tokens/s.
Isolate just the FP8 effect and the real speedup measures 1.5-1.8x, not 2x, because bandwidth saturates first.
Pick L40S for FP8-quantized 7B-13B inference. Pick L40 only when you're power-constrained (300W vs 350W TDP) and FP8 isn't on the table.
84
11pm. Fine-tuning job needs CUDA 12.4. Managed VM ships 12.1. You can't touch the driver.
That's a day lost to a support ticket.
Full root access fixes it:
- Install the exact CUDA, ROCm, or driver version your framework needs
- Patch the kernel yourself
- Run bare metal, nothing abstracted between your code and the GPU
Spheron deploys VM or bare-metal instances with full root access, sourced from Tier 3+ certified data center partners. You pick the driver version, not the provider.
Result: a ten-minute fix instead of a lost day.
Comment if a locked-down VM has ever blocked your deploy.
87
vLLM 0.32: same H100, 4x more concurrent users.
At a 4,096-token context, one H100 went from ~107 to ~428 concurrent users. No new hardware.
Why: NVFP4, a 4-bit format, shrinks the KV cache, the memory holding past conversation tokens. Now supported on:
- A100
- H100
- H200
Smaller cache, more users, same GPU you already rent.
Caveat: the ratio shifts at longer contexts, and 4-bit quantization trades precision for memory. Test on your own workload first.
Running vLLM? Bump the version before buying more GPUs.
1
1
92
Spheron AI retweeted
We use AI in every part of Spheron: code, code review, content, posting, signal-hunting on socials, emails, onboarding and daily customer updates.
Video was the last gap. After months of testing, we cracked it.
Meet our new AI avatar hire 😎
She’s the first of a few presenters for our explainer videos. Not low-effort AI content, but real explainers worth your time.
Episode 1: why @spheronai is all boxes and blue.
Made end to end with AI. If you like it, show some love 🙌
Spot GPU pricing looks cheap until you get reclaimed.
H100 spot: $2.20/hr vs $2.64/hr on-demand. B300 spot: $5.84/hr vs $11.47/hr on-demand. Rates dated 2026-11-01, subject to availability and region.
Getting reclaimed costs you: lost queue spot, checkpoint reload, new GPU wait. All billed per minute, 20 min minimum.
3-week job (504 hrs) on H100 spot: $1,108.80 before a single restart. Two or three reclaims later, reserved starts looking cheap.
When to pick what:
- Job under a week: spot wins outright.
- Multi-week, any preemption risk: reserved breaks even fast.
- Zero tolerance for restarts: reserve from day one.
Reserved clusters: 8 to 512+ GPUs, InfiniBand on request, 24-48hr turnaround.
81
NVIDIA claims Rubin runs 7.5x faster than GB300 NVL72. The reason most coverage skips: it split one GPU job into two chips.
LLM inference has two phases. Prefill reads your prompt (compute-heavy). Decode generates tokens one at a time (memory-heavy). NVIDIA built a chip for each, per its published specs:
- Rubin CPX: prefill, 30 PFLOPS NVFP4
- Rubin GPU: decode, 50 PFLOPS, 288GB HBM4 at 22 TB/s
Vera Rubin hasn't shipped. No pricing yet. We'll publish independent benchmarks once it does. Compare today's H100, H200, B300 rates now.
1
135



















