The bottlenecks around our GPUs

Fig. 1: Conceptual illustration of a silver GPU connected through a narrow interconnect to surrounding infrastructure on a charcoal drafting grid.

Fig. 2: the bottleneck is everything around the gpu: 56.9 GB/s, 600 W, 2.00x vs 1.35x, 5.7 days

Every slow job we measured hit a limit around the GPU: the connection feeding it, its power cap, a prompt-cache miss, or the upload carrying a checkpoint out. The card itself was fast in each case. We measured these limits where the owner actually runs the hardware.

GPU discussions tend to focus on the card as the hard part of the last mile. In our runs, the work was getting everything around a working card to keep up. We need to build that surrounding system for renters.

Fig. 3: Figure 1 from the bottleneck is everything around the gpu: 56.9 GB/s, 600 W, 2.00x vs 1.35x, 5.7 days

The connection

We cross-checked the card's copy engine four ways. Host to device measured 56.9 GB/s; device to host, 56.6. Device-to-device reads were 38.2 to 39.0, and simultaneous transfers ran at 37.5 to 37.8 each way. Those are healthy numbers for this card, and the check takes seconds.

A card running at x8 or Gen3 shows up here first as a low host-to-device number. Acceptance checks elsewhere follow the same approach. NVIDIA's dcgmi levels put a time budget on each check: r1 under 2.5 s; r2 under 2.5 min on 4 GPUs or 10.5 on 8; r3 under 10 / 35 min; and r4 under 45 min / 2.25 h. The targeted stress test passes at 0.95 of a per-GPU target. It doesn't require a perfect score. Crusoe flags a GPU rated for Gen5 x16 but running at Gen3 x8 as critical.

Bad silicon, slots, and wiring can all show up in this early check. The longest soak runs 24 to 72 hours; the connection check takes under 2.5 seconds. Start there.

For a renter asking whether a card is good, these four transfer measurements give more information than one benchmark score. Run the cheap checks first, then use the long checks to confirm.

Fig. 4: Figure 2 from the bottleneck is everything around the gpu: 56.9 GB/s, 600 W, 2.00x vs 1.35x, 5.7 days

The power cap

A 12-minute BF16 burn on our H200 NVL sustained 538.6 TFLOP/s over 178 iterations with 0 mismatches. Peak power was 601 W at 78 C. With both cards burning at once, they measured 548.0 and 562.2 TFLOP/s, a 2.6 percent spread. Both were near 597 W, with temperatures between 76 and 80 C.

SWPowerCap fired on 144 of 144 samples. The card was hitting its own 600 W power limit; it wasn't thermally throttling. That supports the sustained result: the card's design limit was holding it back while our cooling kept up. Under load, the SM clock ranged from 855 to 1320 MHz, a 35 percent spread. A power-bound card can't sustain its boost clock, so that clock is a poor number to promise a renter.

The useful promise is the measured rate under the card's own limit: 538.6 TFLOP/s across 178 iterations with 0 mismatches. Cooling wasn't the constraint in that run.

The prompt cache

One sentence sent to a model produced a turn billed for 25,267 uncached input tokens. The first call sent 42,036 tokens. A second call in the same 15-second turn sent 50,937, including 8,183 that were byte-identical to the first call. That model's cache floor is 1,024 tokens. The mean classifyStructured prompt is 678, so it never qualified.

Sending the same content twice costs 2.00x uncached, compared with 1.25 + 0.10 = 1.35x cached. By the fourth send, it's 4.00x versus 1.55x. Cache content you'll send more than once; content sent once gets no benefit. In the vendor's own testing on a 100-step task, automatically clearing stale tool results cut token use by 84 percent.

The trace showed a prompt-layout problem behind the double billing. Adding cache breakpoints wouldn't fix it. Moving volatile content out of the cached prefix would, without new hardware or a migration. A cache decision happens per prompt and pays on the second send, which makes it the cheapest change in this stack. Here, a few hundred tokens kept the prompt below the threshold.

Fig. 5: Figure 3 from the bottleneck is everything around the gpu: 56.9 GB/s, 600 W, 2.00x vs 1.35x, 5.7 days

Checkpoint uploads

A full fine-tune checkpoint needs 14 bytes per parameter under the ZeRO paper's accounting: 16-bit weights plus 32-bit optimizer state. The optimizer state is larger than the model itself, and the checkpoint saves both.

An 8B checkpoint with optimizer state is about 112 GB, or about 15.6 hours to upload at 16 Mbps. A 70B checkpoint is about 980 GB and takes about 5.7 days on the same link. For comparison, a 100 GB training dataset arrives in about 4.4 hours at 50 Mbps. A LoRA adapter for an 8B model is about 185 MB and uploads in about 1.5 minutes.

At 1 Gbps, the same 8B checkpoint uploads in 15 minutes. That's nearly two orders of magnitude of difference from the 16 Mbps link.

Small adapters move fine over our current connection. LoRA renters are well served today, but large full fine-tunes need a faster link or a volume that keeps checkpoints in the building. Full checkpoints otherwise take hours to leave. Renters uploading from home go through the tunnel and are limited by their own upload speed.

Pull data onto the machine and push results to the renter's bucket. Training can finish well before that upload does. Our take is to price the upload instead of the GPU hours.

Fig. 6: Figure 4 from the bottleneck is everything around the gpu: 56.9 GB/s, 600 W, 2.00x vs 1.35x, 5.7 days

Waiting for weights

Of 58 crowdsourced storage ideas, 26 concern waiting for weights. Among the 11 strongest ideas, 10 concern that wait. The full catalog is 1.66 TB. The main storage complaint is cold starts and weight loading; capacity is secondary.

We also collected 102 posts from 88 people. Of those posts, 23 ask for a ready environment, while groups of 14 and 13 describe lost checkpoints or interrupted runs. The strongest request is a working environment on first boot. Losing finished work is the most painful failure.

How we measured it

These measurements serve different purposes. The copy-engine matrix cross-checks the Python numbers; we recorded it without applying a score. The burn verifies the card under sustained load. Cache costs come from multiplication, and checkpoint transfer times assume line rate. None is a comparative benchmark.

The connection and power results come from our own hardware: a copy-engine matrix read on every burn and a 12-minute BF16 battery run on the H200 NVL with both GPUs idle. The cache example comes from one production turn and its trace. Checkpoint sizes use the ZeRO paper's 14 bytes per parameter. The storage findings come from 58 ideas and 102 posts by 88 people.

Across these runs, the GPU worked while the surrounding system set the pace. Connection checks take seconds, the power cap enforces itself for free, caching pays on the second send, and checkpoint uploads can take a week. Each is cheaper to address than the GPU, and ignoring each can cost more than the GPU. Even the checkpoint delay depends on a link that's already there.

The measurements put recipes and a container cache first on our list, followed by volumes and snapshots. For renters, removing those waits is where we can compete.

The numbers come from seven internal docs: the fleet copy-engine matrix, host acceptance research, fleet consolidation battery run, prompt-cache trace, cache arithmetic, training-fit table, and crowdsourced training learnings. The figures reproduce their numbers verbatim.

---

Originally published by An (@0xaqt) on X on October 15, 2026.