Headless setup for a local-AI box

Running a GPU box without a monitor, from display output and the NVIDIA driver to persistence mode, a power limit at boot, SSH, BIOS settings and serving a model on the LAN.

An inference box earns its keep sitting in a corner with no screen, serving models to the laptop you actually work on. Getting there takes a handful of one-time steps: keep the display off the GPUs, install the driver, make the power settings survive a reboot, reach the box over SSH, set the BIOS so it comes back after a power cut, and expose the model server to your network without exposing it to the internet. The examples assume Ubuntu Server, which is what the sources document; other distributions differ in the package commands, not in the ideas.

Keep the display off the GPUs

Every GB a desktop or console uses on a card is a GB a model can’t, and the VRAM margins in the VRAM sizing guide assume the cards drive no display. So give the screen to something else, even if you plan to plug one in only during setup.

Integrated graphics. The Ryzen 9 9950X on the 24 GB and 48 GB builds has integrated AMD Radeon graphics, and the ASRock X870E Taichi has one HDMI output and two USB4 ports for it. ASRock’s manual notes that only the CPU’s embedded graphics can be displayed through the USB4 ports. Plug the monitor in there.
A server board’s own VGA. EPYC has no integrated graphics. The ASRock Rack ROMED8-2T on the 96 GB build has a VGA port driven by its ASPEED AST2500 BMC, plus a dedicated IPMI LAN port for the BMC’s remote console (manual PDF page 10). Its BIOS has an “OnBrd/Ext VGA Select” item to choose between onboard and external VGA (manual PDF page 57); leave it on onboard so all four 3090s stay free.
A desktop with no integrated graphics. On the Starter build’s host, if the CPU has no graphics, the 3060 drives the screen during setup. Once SSH works, unplug the monitor and check nvidia-smi -q: NVIDIA’s documentation notes that Display Active can read Enabled even with no monitor attached, which means memory is still allocated for a display.

One oddity to know about: owners on NVIDIA’s forum report a headless RTX 3090 idling higher with no monitor attached (about 25 W) than after one was plugged in once (13 W). The power and noise guide covers idle draw and how to measure it.

Install the driver and check it

Ubuntu’s server documentation recommends its ubuntu-drivers tool, which by default installs pre-built, signed drivers that work with Secure Boot. For a compute box it uses the server (“ERD”) driver branch:

Update the system with sudo apt update && sudo apt upgrade, then reboot with sudo reboot, as Ubuntu’s documentation says to.
List the drivers for your hardware: sudo ubuntu-drivers list --gpgpu.
Install the recommended one with sudo ubuntu-drivers install --gpgpu, which picks the best match for your hardware. Ubuntu’s page also shows how to pin a branch from the list and which utilities package goes with it; its examples use an older branch number, so take the number from your own list, not from the page.
Reboot with sudo reboot, then check: cat /proc/driver/nvidia/version shows the running driver, and nvidia-smi should list every card. On an RTX 3090 each should show 24576 MiB of memory.

Ubuntu warns that drivers installed from other sources can overwrite these and may break Secure Boot, so pick one route and stay on it. After any driver upgrade, nvidia-smi reports “Failed to initialize NVML: Driver/library version mismatch” until you reboot; Ubuntu’s troubleshooting section says rebooting loads the new kernel module and brings the versions back in sync. Then run nvidia-smi -q once per card and read the PCIe link generation and width under load; the PCIe lanes and risers guide explains what to expect, and the used RTX 3090 checklist has a half-hour burn-in test worth running on every used card before you rely on it.

Persistence mode and the power limit at boot

Two settings don’t survive a reboot, and a headless box reboots unattended.

Persistence mode. With it on, NVIDIA’s documentation says the driver stays loaded even when no client is using the GPU, which avoids the driver load delay each time a CUDA program starts. Without it, the GPU state unloads when the last client exits. nvidia-smi -pm 1 turns it on as root, per GPU or for all; it resets to disabled after every reboot. NVIDIA calls that kernel setting legacy and recommends its persistence daemon, nvidia-persistenced, which keeps the device files open from user space; nvidia-smi uses the daemon if it is running, so the same command works either way.
The power limit. nvidia-smi -pl 300 sets each card’s limit in watts, within the minimum and maximum nvidia-smi -q reports. It needs root, and per NVIDIA’s documentation the limit goes back to its default after a driver unload, as well as after a reboot. The multi-GPU Tiers depend on it; the PSU sizing guide explains why.

Puget Systems reapplies both at boot on a quad-RTX 3090 machine with a systemd service that runs a small script: it turns on persistence mode with nvidia-smi, then sets the limit, and is enabled with systemctl enable so it runs at every boot. A minimal version of the same idea needs no script. It keeps Puget’s -pm 1 line because that works whether or not the daemon is installed; if you run NVIDIA’s recommended nvidia-persistenced instead, you can drop that line. Save these lines as /etc/systemd/system/nv-power-limit.service:

[Unit]
Description=NVIDIA persistence mode and power limit
After=systemd-modules-load.service
ConditionPathExists=/usr/bin/nvidia-smi
[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -pm 1
ExecStart=/usr/bin/nvidia-smi -pl 300
[Install]
WantedBy=multi-user.target

Change the 300 to your own limit before saving. It must sit between the Min and Max Power Limit that nvidia-smi -q -d POWER shows for each card: 300 for the 3090 Tiers, 250 for four cards on a US 15 A circuit (the 96 GB build explains why), and for the Starter’s RTX 3060, which NVIDIA rates at 170 W, no more than 170 and within the range the card reports. systemd allows several ExecStart lines only for Type=oneshot, which is why it is set. Start it once with sudo systemctl start nv-power-limit.service, check sudo systemctl status nv-power-limit.service, then sudo systemctl enable nv-power-limit.service so it runs at every boot. The unit has two weak spots. After=systemd-modules-load.service orders it, but doesn’t guarantee the driver is ready, and if /usr/bin/nvidia-smi is missing, for example halfway through a driver change, ConditionPathExists makes systemd skip the unit rather than fail it. Either way the cards run at their default limit and nothing complains. So after every boot and every driver update, check that nvidia-smi -q -d POWER shows your limit on every card. Puget’s own script also compares the limit with the card’s maximum before applying it; NVIDIA’s documentation requires a value between the card’s minimum and maximum power limits.

Remote access over SSH

Install the server with sudo apt install openssh-server. Ubuntu’s documentation puts local settings in snippet files under /etc/ssh/sshd_config.d/, which override the main file, and says to check the configuration with sudo sshd -t before restarting the service. Set up key-based login from your laptop first, and only then turn off password logins. Ubuntu’s warning applies with full force on a headless box: if SSH is your only way in and you break its configuration, you may lock yourself out. Keep a monitor and keyboard within reach the first time, or, on the ROMED8-2T, the BMC’s remote console.

That BMC has its own login. ASRock’s manual gives the IPMI web interface’s default username and password as admin and admin (PDF page 76). Change both before the IPMI port goes anywhere near a network, and keep it off the internet entirely.

Waking a box you shut down

The power and noise guide suggests shutting the box down or suspending it when idle, which on a headless box raises the question of how it comes back.

ROMED8-2T: the BMC. ASRock’s BMC guide for the board has a Power Control page with Power On, Power Off and Power Cycle for the host (BMC guide PDF page 111), reachable from the IPMI web interface while the machine is off. The board also lists Wake-On-LAN for its LAN controller (manual PDF page 10), with a BIOS item, “PCIE Devices Power On”, that enables wake on LAN (manual PDF page 61).
X870E Taichi. Its manual doesn’t mention Wake-on-LAN, and we have not confirmed it for this board. Check ASRock’s BIOS documentation for the board before you rely on waking it remotely; otherwise someone has to press the button.

BIOS settings that matter

Power on after a power cut. The ROMED8-2T’s “Restore AC Power Loss” item sets what happens when mains returns: [Power Off] stays off, [Power On] boots (manual PDF page 59). A headless box should be set to [Power On], or every outage means a trip to the power button.
Above 4G Decoding. The ROMED8-2T’s manual describes it as decoding 64-bit capable devices in the address space above 4G (PDF page 58). Its manual names no Resizable BAR setting, and we make no claim about it.
Per-slot link speed. If a card on a riser misbehaves, the ROMED8-2T can force any slot to Gen3; the PCIe lanes and risers guide covers when.
The X870E Taichi. Its manual covers the hardware and leaves the BIOS menus out, so we can’t name its settings here. Look for the same three ideas in ASRock’s BIOS documentation for the board: restore on power loss, above-4G decoding and the primary display set to the integrated graphics.

Serving a model on the LAN

llama.cpp’s server listens on 127.0.0.1 by default, which reaches only the box itself, and on port 9931 per its README. To reach it from your laptop, give --host the box’s LAN address, or 0.0.0.0 for every interface, and pick a port with --port. Then:

Set an API key. --api-key (or --api-key-file) makes clients authenticate; it has no default, so without it anyone on your network can use the server. The README notes that the health endpoint stays public even with a key set.
Don’t open it to the internet. No port forwarding on your router, no --host 0.0.0.0 on a machine with a public address. The README’s own advice for a public deployment is an API key plus a reverse proxy, which is beyond a home box; for a local network it recommends setting --cors-origins to your front end’s origin.
Leave the agent tools off. The README marks --tools, --agent and the MCP server options “do not enable in untrusted environments”: they expose file read and write, and in the tools list a shell command, over the API.
Firewall the port. With --host 0.0.0.0 the server listens on every interface, so also restrict the port to your LAN with a host firewall. Ubuntu’s firewall documentation shows ufw allowing a port only from a subnet, as in sudo ufw allow proto tcp from 192.168.0.0/24 to any port 9931, then sudo ufw enable. Allow SSH from the LAN the same way before you enable it, or you lock yourself out.

To reach the box from outside your home, connect to your home network over a VPN first, and keep both SSH and the model server unforwarded on your router. Nothing on this box should face the internet directly.

Sources

Last reviewed 2026-11-12