GPU Server Burn-In Test: DCGM Level 3 on 8 RTX PRO 6000

A new GPU server should prove itself before it carries real work. This is the test we run before we hand over a whole server, with the numbers from our last run on 20 October 2026: eight NVIDIA RTX PRO 6000 Blackwell Server Edition cards, NVIDIA DCGM 4.7, run level 3.

The test plan in five steps

Take an inventory with nvidia-smi. All cards present, ECC enabled, the power limit you expect, and a PCIe Gen5 x16 link on every card. Compare the serial numbers with your paperwork.
Run dcgmi diag -r 3. This is NVIDIA's own hardware diagnostic for data centre GPUs.
Log every card while it runs: temperature, power draw and the clock event reasons that tell you why a card slowed down.
Read the ECC and Xid counters when the run ends.
Check idle power once the load is gone. Our run caught its only problem here.

What level 3 ran on our server

Test	What it checks	Result
software	The system is ready to run CUDA applications	Fail on 8 of 8: persistence mode off (see below)
memory	GPU memory integrity, using about 75% of the memory	Pass 8 of 8
diagnostic	Large matrix multiplies while writing and reading across the frame buffer	Pass 8 of 8
pcie	Host-to-GPU and GPU-to-GPU bandwidth, latency and replays	Pass 8 of 8
nvbandwidth	Bandwidth between the GPUs and the host	Pass 8 of 8
memory_bandwidth	Memory bandwidth of each card against a minimum	Pass 8 of 8
targeted_stress	Constant compute load at a target GFLOPS	Pass 8 of 8
targeted_power	Drives each card towards its power limit and holds it there	Pass 8 of 8

DCGM has four run levels. Level 1 takes seconds and checks the setup, level 2 adds a medium set, level 3 runs the long hardware diagnostics, and level 4 adds a memory test and a power pulse test. Each level includes the ones below it.

NVIDIA's guide gives level 3 under 35 minutes on an 8-GPU Hopper system. On our eight RTX PRO 6000 cards it took about two hours. For most of that time no card drew more than about 150 W; full power came at the start and again at the end. Budget a couple of hours, not half an hour.

Temperatures and errors during the run

We logged every card every 10 seconds. The hottest card peaked at 86°C and the coolest at 81°C, with the highest reading at 604 W. The only limiter the cards reported was the software power cap, which is what you expect at the 600 W limit. There was no thermal slowdown.

After the run the counters read 0 ECC errors and no Xid errors on all eight cards. For longer thermal tests, fan modes and power caps on the same server, see what an 8x RTX PRO 6000 server really draws.

The one failure: persistence mode and 440 W at idle

The software check failed on every card with the same message: persistence mode is disabled. That is not a hardware fault. Ubuntu's nvidia-persistenced service starts the daemon with --no-persistence-mode, so the setting is off by default.

It still costs money. With persistence off, our idle cards sat in performance state P0 at about 90 W each, 721 W for the eight. With it on, they settle in P8 at about 35 W each: 279 W for the eight after a reboot. That is about 440 W, or roughly 10.5 kWh a day, on a server doing nothing.

NVIDIA's driver documentation explains the setting: "The GPU is typically deinitialized after the last client exits." Persistence keeps the GPU initialised, which also avoids a start-up delay of seconds before each CUDA job.

To turn it on now, as root:

nvidia-smi -pm 1

To keep it after a reboot, override the service so it starts without --no-persistence-mode. Put this in /etc/systemd/system/nvidia-persistenced.service.d/persistence-mode.conf:

[Service]
ExecStart=
ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced --verbose

Then run systemctl daemon-reload and systemctl restart nvidia-persistenced, and check with nvidia-smi --query-gpu=persistence_mode,pstate,power.draw --format=csv.

gpu-burn or DCGM?

gpu-burn is an open-source "multi-GPU CUDA stress test". It puts every card under full load for as long as you ask, which makes it good for thermal tests over hours. We used it in July to find the right fan mode for this chassis.

DCGM checks more. It tests memory, PCIe, bandwidth and power behaviour and gives a pass or fail per card. Use gpu-burn to see whether the cooling holds, and DCGM to catch a weak card.

Beyond the GPUs

Disks. Let the first RAID sync finish before you benchmark storage. A sync in progress makes the disks look slow.
Network. Test the open ports from outside the data centre, not from the server itself.
Remote access. Check that the BMC gives you power control, a remote console and virtual media.

If you rent a whole server

When we hand over a dedicated GPU server, we run this plan first and send the results with the access details: the DCGM report, peak temperatures and the ECC and Xid counters. You get root and BMC access from the first day, so you can run your own tests as well.

Questions

How long does DCGM level 3 take on eight GPUs? NVIDIA gives under 35 minutes on an 8-GPU Hopper system. On our eight RTX PRO 6000 cards it took about two hours.

Does a "Fail" in the software section mean the hardware is bad? Not necessarily. In our run the only failure was persistence mode, a setting. Every hardware test passed.

Do I need root? To install DCGM and to change persistence mode, yes. By default DCGM itself drops privileges and runs the diagnostics under an unprivileged service account.

Should I leave DCGM installed? On your own server, the DCGM exporter can keep sending GPU metrics to Prometheus. On a server we hand over, we remove DCGM after the test so that nothing of ours stays on the customer's system.

Sources

NVIDIA, DCGM Diagnostics user guide: run levels, test descriptions and durations.
NVIDIA, Driver Persistence: persistence mode and the persistence daemon.
gpu-burn on GitHub.
NEOXIS measurements on our 8 × RTX PRO 6000 Blackwell Server Edition server, 20 and 21 October 2026: dcgmi diag -r 3 with DCGM 4.7 and driver 580.178.04, nvidia-smi log every 10 seconds, idle power before and after enabling persistence mode.