Two nights of training on an 8-GPU box

Fig. 1: Conceptual illustration of an open server with eight silver accelerator modules on a charcoal drafting grid.

Fig. 2: Two nights of training on an 8-GPU box: the detector, the tiny models, and the 8-bit step

One 8-card RTX PRO 6000 box trained YOLO11x from scratch on COCO for the full 600 epochs in 19 h 10 min. Across all 19 hours, the cards averaged 86% utilization and 415 W per card, with no heat slowdown. The best score was 0.527 mAP50-95 at epoch 412; Ultralytics publishes 0.547.

On YOLO11m, eight cards were 6.4× one card: 409 s to 64 s per epoch. The missing 1.6× showed up in power use, which fell from 344 to 220 W per card. The busy column missed it.

Model size changed the bottleneck. With 32 tiny models running, the GPUs stayed at 10–14% while the CPUs reached 94%. The same tiny model fell from 94.6% to 9–15% accuracy after INT8. Two early layers carry values about 6× wider than the rest.

We build and host these boxes at b3iq, a neocloud that belongs to its customers. Owners run their own jobs on hardware we keep fast and boring. We'd spent weeks tuning this box's PCIe switch and drivers and checking the qualification numbers. Now we wanted to see it finish training a model.

This is test 2 of the lab notebook, after the six-hour language-model pretrain of test 1. We wanted to push eight cards as hard as they would go, for as long as it took, and record every number.

The work covered three jobs over two nights. Night one started with forty small vision models trained side by side, the workload shape we expect most from edge vision. We then shrank each model to 8-bit numbers, as a camera board requires.

On night two we ran the standard YOLO detectors on the standard COCO benchmark, one card against eight. Then we trained the largest YOLO11 from nothing for its full 600-epoch schedule, a whole day's job. The machine was one 8-card RTX PRO 6000 Blackwell Server Edition box, bare metal, with 96 GB per card, 128 CPU cores and 756 GB of memory. All the data is public: COCO 2017 has 118,287 training photos and 80 kinds of everyday object; Food-101 has 101 dishes and 101,000 photos.

The detector trained at full tilt for 19 hours. Its best score was two points under the published number on an untuned recipe. We found problems in the software around the box: the recipe overfit after hour 13, the 8-bit conversion collapsed, and copying checkpoints off the box was unreliable.

Fig. 3: One box, two nights: what the runs said

Training setup

A model starts as millions of random numbers. You show it a photo and the right answer, nudge the numbers toward the answer, and repeat for every photo. An epoch is a pass through the full dataset: 118,287 photos for COCO, 115 s on this box for YOLO11x.

Accuracy is measured on photos the model never trained on. For detection, the score is mAP50-95, from 0 to 1; the published YOLO11 models top out at 0.547. From scratch means random starting weights, with no pretrained model.

A checkpoint is the model's weights saved to disk mid-run: best.pt for the best so far, last.pt for the latest. INT8 means rounding a model's numbers to 8 bits so it fits a camera board's chip. It usually costs a few points of accuracy, sometimes everything.

The six runs, in order: eight tiny models at once, one per card; 32 at once, four per card; INT8 of every one of them, with six calibration variants on the one that broke; YOLO11m from scratch on eight cards and then on one; a segmentation model and a transformer detector; and YOLO11x for 600 epochs.

Fig. 4: Two nights, six runs

Night one: forty tiny models

Food-101 was cut into "stores" of 12 dishes, 9,000 photos per store, with one MobileNetV3-Small (1.5M weights) per store. Eight stores trained in 218 s and 32 stores in 204 s, so forty models in under eight minutes, at 86.9–94.8% accuracy on their 12 dishes.

Even with 32 models running, this workload didn't keep the GPUs busy. The 128 CPU cores decoded JPEGs and augmented photos at 94% busy. The cards read 10–14% busy and used an average of 6.5 GB of 96 GB each.

The next night, a 57M-weight detector on 640-pixel photos kept each card at 86% mean over 19 hours, 415 W mean and 34–43 GB. The CPUs were at 16%, and the box processed 1,030 photos/s. The bottleneck followed the model: CPU decode starved the tiny models; the big one kept the cards busy.

Fig. 5: Who was busy: 32 tiny models against one detector

During the 32 small jobs, the dashboard's health tab showed uneven GPU utilization between 0% and 59%, with the host CPU bar at 94%. Before adding cards for small-model work, fix the photo pipeline: decode on the GPU (nvJPEG, DALI) or pre-decode once to tensors. The box's 128 cores and 756 GB are what make 32 jobs at once possible at all.

Fig. 6: The owner dashboard during the 32 small jobs: GPUs between 0% and 59% and uneven, 4 to 5 GB of 96 per card at this reading, host CPU 94%, load 51 on 128 cores

Fig. 7: Meme, a man in a suit explaining to another across a desk: no matter what CPU and GPU you have, there will always be a bottleneck

Night one: the 8-bit conversion

MobileNetV3-Small, 1.53M weights, scored 94.6% in float on its 12 dishes and 9–15% after INT8, near chance, since 1 in 12 is 8%. Every tool lost accuracy: ONNX Runtime with six calibration variants, PyTorch's fx post-training quantization, and three epochs of quantization-aware training (training with the rounding simulated, 29%). The prepared graph with the fake rounding switched off still scored 94.6%. The failure came from the model's numbers, regardless of which tool we used.

Fig. 8: Meme, the half-finished horse drawing: the carefully shaded rear is labelled in float, 94.6%, and the scribbled head and legs are labelled in INT8, 9 to 15%

The first two layers' activations, the values passed between layers, span −141 to +143 over 512 calibration photos. Every later layer sits within ±25, about 6× narrower. Eight bits give 256 levels per tensor. Keeping those two layers in float restores 89.2%, which locates the damage in them. We did not isolate whether the range, the weights or both caused it.

Fig. 9: Why the 8-bit step collapsed: two early layers carry values about 6× wider than the rest

Two fixes worked. Leaving the stem and block 1 in float and quantizing the rest restored 89.2%, shrinking the model from 5.9 MB to 1.8 MB. A different architecture, MobileNetV2 at 2.24M weights, gave 97.1% in float and 96.4% in INT8 with per-channel scales (95.0% with one scale per tensor), from 8.5 MB to 2.5 MB, with no tricks.

All of it was measured on the CPU, on store 7 of the small-model night. For a team shipping to a camera board, this can surface late. On the box it took about eleven minutes because six variants ran side by side.

Fig. 10: Three ways to shrink the same store model

Night two: one GPU to eight, 6.4×

YOLO11m from scratch on COCO took 409 s per epoch on one card and 64 s on eight. 409 ÷ 64 = 6.4, against the 8× ideal of 51 s. Per-card power fell from 344 W in the one-card run to 220 W in the eight-card run, with the same model and batch of 32 per card. The per-card watts point to less work per busy second behind the missing 1.6×.

We think the cards were waiting on CPU-side augmentation (the random edits made to each photo before a card sees it) and on synchronization between cards. We did not run a test to isolate the two. The busy column never showed the gap: 71% on one card, 75% on eight. NVIDIA defines that number as the share of time one or more kernels was running, so it counts time, not effort.

Fig. 11: One GPU to eight: 6.4×, and where the rest went

The other two short runs that night used different models. YOLO11m-seg from scratch reached 0.133 mask mAP after 10 epochs at 90 s each. RT-DETR-l, a transformer detector started from pretrained weights, ran at about 530 photos/s with the cards at 49% and 186 W. Memory was 15 to 28 GB of 96 GB on each of the four short runs, so there was room for bigger batches.

Night two: nineteen hours flat out

YOLO11x started at 06:35 UTC on 2026-10-18 and finished at 01:45 the next day. It completed 600 epochs at 115 s each, 69,018 s of wall-clock time, with no restarts.

A sidecar read the cards every 30 s and collected 18,400 samples (2,300 reads of 8 cards). Per-card means were 86% busy and 415 W, with a 459 W peak and about 64 kWh for the GPUs over the run. The hottest card peaked at 76 °C, with per-card maxima from 63 to 76 °C. The card's own thermal margin never read below 8 °C.

Throttle flags appeared on 0.24% of samples: a power-cap flag on GPU 2 for 15 samples, and the rest in the first 90 s. GPUs 2 and 7 ran warmest on this chassis, as they did in test 1, this time with headroom.

Fig. 12: Nineteen hours flat out: the numbers

The health tab's GPU cards at one reading during the run: 8 under load, none throttling, 77 to 100% per card, 416 to 440 W of 600 W, every card healthy.

Fig. 13: The owner dashboard's GPU cards during the run: 8 under load, none throttling, 77 to 100% per card, 416 to 440 W of 600 W, every card marked healthy

At hour 15, the dashboard's 24-hour view showed one flat block from the start of the run (06:35 UTC; the chart's own axis is local time). Utilization, memory, temperature, heat margin and power were all level.

Fig. 14: The owner dashboard's 24-hour health charts 15 hours into the run: utilization, memory, temperature, heat margin, power and PCIe width, one flat block from the start of the run

The 7-day charts, read on 2026-10-20 and cropped to the plot area from Oct 14 on, put the two tests side by side. The language-model pretrain of 2026-10-14/15 sits near 100% utilization, about 520 W per card and averages near 75 °C across the eight cards; this run sits at about 85%, about 420 W and a flat 65 °C, under the 90 °C alert line.

Fig. 15: The owner dashboard's 7-day utilization chart, read on 2026-10-20 and cropped to the plot from Oct 14 on: the Oct 14 to 15 pretrain near 100%, the Oct 18 to 19 YOLO11x run at about 85%

Fig. 16: The owner dashboard's 7-day power chart, read on 2026-10-20 and cropped to the plot from Oct 14 on: about 520 W per card in the pretrain, about 420 W in the YOLO11x run, the dashed 600 W limit line above

Fig. 17: The owner dashboard's 7-day temperature chart, read on 2026-10-20 and cropped to the plot from Oct 14 on: the average of 8 GPUs near 75 °C in the pretrain with the hottest card near 80, a flat 65 °C through the YOLO11x run, the dashed 90 °C alert line above

Best checkpoint: hour 13

The best score came at epoch 412, hour 13: 0.527 mAP50-95. The run kept going for six more hours and ended at 0.490. The logged values were 0.272 at epoch 5, 0.374 at 10, 0.466 at 25, 0.483 at 50, a dip to 0.474 at 100, then 0.503 at 200, 0.517 at 300, 0.527 at 400, 0.522 at 500 and 0.490 at 600.

Train losses kept falling to the end (mosaic augmentation switches off for the last ten epochs by default). Validation losses turned up after about epoch 400, and mAP followed them down. The model was overfitting, learning the training photos instead of the task.

Fig. 18: The model you ship is from hour 13

Fig. 19: The run's results.png: ten panels of train and validation losses, precision, recall, mAP50 and mAP50-95 over 600 epochs

The best.pt file held the hour-13 model. The trainer rewrites best.pt whenever the validation score improves and last.pt every epoch. That left us with the 0.527 model to keep, even though the run ended at 0.490.

The run also saved periodic checkpoints with optimizer state: 20 GB on the box's NVMe by the end, 1.8 GB copied off. A hand-rolled puller on a laptop did the copying, and it died once. A training box needs durable checkpoint storage off the box, written as the checkpoints appear.

Fig. 20: Meme, the bell-curve format: the two ends say its just overfitting, the crying middle says classifier accuracy is a complex subject with many different interactions and failure modes

The two points to the published 0.547 look like a recipe gap. We ran an untuned recipe and it overfit after epoch 412; Ultralytics' 0.547 is from its own run of the same model at the same 640-pixel size, with a recipe we did not reproduce. Our read is that heavier augmentation closes the gap; that is a hyperparameter change, and it is untested here.

Fig. 21: What 0.527 looks like: predictions on held-out COCO photos; the confident boxes are trains, planes, zebras, pizza and vases at 0.9 and kites at 0.8, with low-confidence boxes around them

Limits of the runs

0.527 is one run of an untuned recipe, and 0.547 is Ultralytics' number from its own run; this is not a reproduction of their result, and nothing here says the gap is hardware. The 6.4× is YOLO11m at batch 32 per card, one run each, 10 epochs on eight cards against 2 on one; a bigger batch or a faster photo pipeline could move it. The tiny-model numbers are for 1.5M-weight models with JPEG decode on the CPU; a decoded-tensor pipeline might load the cards differently. The INT8 collapse is one architecture on one dataset, measured on the CPU; it says MobileNetV3-Small has this problem here, not that INT8 is unsafe. The thermals are one chassis over one 19-hour run, not a rating. And the 19-hour time is one box of this card class on this recipe, not a speed record for the model.

Job records for the next run

The box completed a benchmark-grade training job. The dashboard's health curves cover the run, and its activity log records SSH events. It doesn't record what ran, when, or which checkpoints were written. We had to assemble that history by hand for this report.

This 128-core box let us try six calibration variants side by side in minutes. Whether a model survives INT8 depends on its architecture. Check the model before promising GPU utilization.

Two questions stay open: whether heavier augmentation closes the last two points to 0.547 on this box, and whether GPU-side JPEG decode gets a box full of 1.5M-weight models off the CPU ceiling.

We measured everything on the box itself, on public data (COCO 2017 and Food-101), at b3iq. A sidecar read every card every 30 s, collecting 18,400 samples over the long run. We captured the owner dashboard's health tab at each stage.

Most GPUs live in someone else's building, and you rent time on them. At b3iq, we think the people who use GPUs should own them and still have someone run the boring parts. You own a bare-metal box like this one; we keep it fast and boring. The dashboard in these screenshots is the owner's view. If you want one, b3iq.org.

Tell us which test you want next.

---

Originally published by An (@0xaqt) on X on October 20, 2026.