Six hours on an 8-GPU box: 463k tokens/s and two hot slots

Fig. 1: Conceptual illustration of a silver GPU cooling assembly with a restrained blue diagnostic trace on a charcoal drafting grid.

Fig. 2: six hours on an 8-gpu box: the speed, the heat, and what the dashboard got wrong

One 8-card RTX PRO 6000 box trained a 1.38B-parameter model from scratch in 6 h 23 min. Pretraining held a median 463k tokens/s, and no sample showed a heat slowdown. Two cards still ran out of thermal margin.

GPUs 2 and 7 ran about 14 °C hotter than the coolest card and dipped to −3 °C of margin. The owner dashboard called all 8 cards Healthy. Its utilization, temperature, memory and PCIe readings were close to the node's own logs, but the verdict missed what the thermal-margin counter was reporting. The dashboard code now reads that counter too.

This was test 1 of the B3IQ lab notebook: nanochat's speedrun on 8 RTX PRO 6000 Blackwell Server Edition cards, bare metal, on the night of 2026-10-14. B3IQ is a customer-owned neocloud; owners run their own jobs on boxes we operate. I compared the owner dashboard with the node's logs throughout the run.

Pretraining processed 8.76B tokens over 8,352 steps in 340.6 minutes, or 5.7 hours. The whole recipe took 6 h 23 min from the start of pretraining. It completed without errors, and the kernel log stayed empty. The only clock limit the cards reported was their normal 600 W power cap.

Fig. 3: one box, six hours: what the cards said

What ran

Nanochat's public recipe pretrains a GPT-style model on web text, tests it, fine-tunes it into a chat model, then tests that. All 8 GPUs stay busy communicating during training.

The first try crashed. The recipe's prebuilt attention kernel supports compute capabilities 8.0 and 9.0a; this card is 12.0. A one-line switch made nanochat's loader fall back to PyTorch's built-in attention. With the recipe's default attention pattern, a test ran at 280k tokens/s. Full attention reached 464k, 1.66x faster, so we switched.

Pretraining started at 23:10 UTC: 1.38B parameters, FP8 math, 8,352 steps of 1M tokens each. It finished at 04:50 after 340.6 minutes. Chat fine-tuning started at 04:58, and the main pipeline finished at 05:33. For the first 3 minutes of the fine-tune, the GPUs sat idle while the dataset downloaded.

Fig. 4: the run, station by station

Throughput over 8,352 steps

The median step rate was 463k tokens/s, with 90% of steps between 462k and 464k. Benchmark checks and checkpoint saves pause the run; those pauses don't appear in the 5-minute medians. Between 23:10 and 04:50, the run processed 8.76B tokens in 340.6 minutes.

The per-step rate stayed steady even while GPUs 2 and 7 were at their thermal margin. With all 8 cards communicating throughout training, one card slowing for heat or power would have bent the throughput curve. We saw no such drop.

Fig. 5: 463k tokens per second, steady for 5.7 hours

How hard the cards worked

nvidia-smi reported 98.5–99.2% utilization. That means a kernel was running almost all the time; it doesn't tell you how much work the hardware did inside each kernel.

Nvidia's finer GPM hardware counters split that time up. We read them once a minute on all 8 cards from 01:55, covering the last 2 h 55 min of pretraining. The compute units had work 82.5% of the time. Tensor cores, the matrix units that do the training math, ran 36.0% of the time. Each compute unit was 25.4% full with threads, and memory moved 48.9% of its capacity. PCIe traffic was 3.9 GB/s out and 4.0 in per GPU, about 6% of a Gen5 x16 link each way.

Fig. 6: how hard the cards worked, six counters

Separately, the model math works out to 306 TFLOPS per GPU: 463k tokens/s × 5.28 GFLOPs per token ÷ 8 GPUs. The 5.28 comes from nanochat's own formula: 6 FLOPs per matrix-multiply parameter plus the attention term. I worked it out by hand because the recipe has no peak figure for this card and prints 0% for it.

Which peak you use changes the percentage. At the 2.3 GHz the cards actually held, 188 SMs × 1,024 FLOPs per clock gives 443 TFLOPS of dense BF16. The run used 69% of that. Against the Server Edition's rated 2,430 MHz boost, or 468 TFLOPS, it used 65%.

Against the FP8 peak, it used 35%. That peak is twice the BF16 one, 886 against 443 TFLOPS, but only the large matrix multiplies run in FP8. Attention and the rest stay in BF16. At this model size, the GPUs also spend a fair share of time waiting on memory and on each other. For comparison, Aurora Infra's plain BF16 matrix-multiply benchmark sustains about 376 TFLOPS on this card at 600 W. Our 306 TFLOPS of model math is about four fifths of that bare-matmul rate.

Power stayed around 523–550 W per card, swinging within each 2.3-second step and peaking at the 600 W board limit. Clocks held 2.31–2.35 GHz. The power cap appeared in 82% of samples and was the only reported clock limit. None showed a heat slowdown.

Fig. 7: power, clocks and the three peaks

Two hot slots

Six cards averaged 67–77 °C during pretraining. GPUs 2 and 7 averaged 81 °C, peaking at 86 and 87 °C. These are passively cooled server cards, so the chassis fans do all the cooling. Two slots running much hotter than the rest points at airflow, but we still don't know whether a weak fan or slot position caused it. This box has no ipmitool, so I couldn't read the chassis fans.

The card's own thermal margin, temperature.gpu.tlimit, is more useful here than core temperature. Nvidia defines it as the margin in degrees to the card's maximum operating temperature. At 0 or lower, the card may start cutting its clock for heat. GPUs 2 and 7 kept touching zero and dipped to −3 °C, with 16 and 22 samples at or below zero. GPU 0, the coolest card, never fell below 13 °C of margin.

The two hot cards reached −2 °C of margin at core temperatures anywhere from 77 to 87 °C. A fixed Celsius threshold can't catch that range. The dashboard's alert line was 90 °C, above the highest logged temperature of any card: 87 °C on GPU 7.

Fig. 8: eight cards, two hot slots, and the margin counter

Even at −3 °C of margin, the clocks showed only the normal 600 W power cap. This was a nighttime run; a warmer room would likely push those two cards into a heat slowdown. At midnight Eastern, the dashboard's temperature chart showed them at 81 and 82 °C against 67 °C for the coolest card. The temperature difference was visible on the page.

Fig. 9: the dashboard's temperature chart at midnight eastern: GPU 2 at 81 °C and GPU 7 at 82 °C, alert line at 90 °C

Checking the owner dashboard

The B3IQ owner dashboard is the signed-in page where the machine's owner sees its health tab and GPU cards. Every 10 minutes, a script compared its stored samples with the node's logs. We averaged both sides over the same 10-minute window because individual readings swing within each training step. At every stage change, I also reloaded the page and read it.

Fig. 10: the comparison: both sides of the dashboard check

Across seven steady windows, utilization, temperature, memory and PCIe traffic matched the cards' readings within about 2 points, 1–2 °C and 0.2 GB/s. Power read 10–32 W high on the page in every window, about 5%. Both sides read the same power.draw field. I don't know why they disagree yet.

Fig. 11: dashboard versus the cards' own logs, seven windows

The page also goes quiet in a background tab. It deliberately skips its 60-second refresh while hidden and catches up when shown. A visible page runs one heartbeat behind, about a minute, because the host software reports once a minute.

All 8 GPU cards said Healthy while GPUs 2 and 7 sat at their thermal margin. The dashboard's measurements matched the cards; its health verdict missed the margin counter. The code now reads that counter: a card with 3 °C or less of margin counts as at or near its heat limit, and at 0 it is a fault.

Fig. 12: the owner dashboard with all 8 gpus at 100%: 8 under load, none throttling

What the model could do

The pretrained model scored 0.2707 on CORE, an average of 22 standard tests covering reading, common sense, trivia and simple reasoning. Nanochat scores the original OpenAI GPT-2 checkpoint at 0.2565. Our model kept improving through the last step: validation loss fell from 3.17 to 0.704 bits per byte. Quick CORE checks read 0.201 at step 2,000, 0.215 at 4,000, 0.260 at 6,000 and 0.284 at 8,000. Those checks use fewer questions per test, so they read a little higher than the full run.

Fig. 13: validation loss and the quick CORE checks over 8,352 steps

After chat fine-tuning, the generic chat model scored 66.3% on ARC-Easy, 52.4% on ARC-Challenge, 38.3% on MMLU, 3.3% on GSM8K and 11.0% on HumanEval, for a ChatCORE of 0.247.

Asked why the sky is blue, it correctly named Rayleigh scattering, then said the atmosphere absorbs red light and that the sky is blue "even on a cloudy day". It gives fluent, confident answers that are only partly correct, which is typical for a model this size. The box beat nanochat's GPT-2 reference on CORE overnight. Its chat model still made those mistakes.

Fig. 14: what the chat model scores

Limits of the comparison

The 463k tokens/s and 69% of peak describe this model size and batch, with FP8 on the large matrix multiplies and full attention. A larger model or a different recipe will behave differently. The 35% FP8 figure is the share of the FP8 peak this run used. These are results for the training job, not a general benchmark of the card.

The duration also needs context. We used nanochat's default of 12 tokens per parameter, 8.76B tokens. Its own speedrun trains the same depth to 8 tokens per parameter in about 1.5 hours on 8 H100s, which have about twice this card's BF16 peak and NVLink between them. A C++ port reached CORE 0.2592 on the same card class in 135 minutes. We trained on more tokens and got a higher score; this run doesn't establish that our box was faster.

The dashboard comparison covers seven averaged ten-minute windows on one box. Individual one-second readings can differ. The thermal observations are also from one chassis at night, and the fan-versus-slot-position question is still open. This was a 1.4B model trained on 8.76B tokens. Its CORE score beat nanochat's GPT-2 reference, while its chat answers remained fluent, partly correct and confident.

The next run is serving on the same box. Tell us which test you want after that. Before we can explain why GPUs 2 and 7 run hot, we need to read the chassis fans; that would settle whether a weak fan or slot position is responsible. The other unresolved measurement is the page's consistent 10–32 W power overread.

How it was measured

One box: 8 RTX PRO 6000 Blackwell Server Edition cards, bare metal, 2026-10-14 23:10 to 2026-10-15 07:02 UTC. Nanochat at commit 92d63d4: depth 24, 1,536 wide, 2,048-token context, FP8 on 145 of 158 linear layers, 1M-token batches, attention through PyTorch SDPA with the full-attention pattern, PyTorch 2.9.1 and NCCL 2.27.5 in the recipe's own environment.

Per-GPU readings came from nvidia-smi every 30 s: utilization, memory, temperature, power, clock and clock reasons. Thermal margin was read every 60 s from 00:13; hardware counters every 60 s from 01:55 with nvidia-smi dmon --gpm-metrics. dmesg -W watched for GPU, PCIe and memory errors and stayed empty. The dashboard check read the stored samples behind the health tab every 10 minutes and compared 10-minute averages with the node's log. Owner-measured on B3IQ, on an 8-card RTX PRO 6000 box.

Sources

• nanochat at the commit we ran, the recipe, its flops formula and its gpt-2 reference score - nanochat at commit 92d63d4
• the c++ port's 135-minute run on 8 rtx pro 6000 - 135-minute C++ run
• nvidia rtx pro 6000 blackwell server edition - RTX PRO 6000 specifications
• nvidia rtx blackwell pro gpu architecture whitepaper, peak rates - Blackwell architecture whitepaper
• nvidia-smi, the t.limit sensor and the gpm counters - nvidia-smi documentation
• dcgm's definitions of sm activity and tensor activity - DCGM metric definitions
• aurora infra's 600 w vs 300 w bench - 600 W vs 300 W measurements

---

Originally published by An (@0xaqt) on X on October 17, 2026.