Agent Runtime Server Setup
Running agents evergreen on a server came with its own set of problems. Your server has a fixed amount of RAM. When a runaway program consumes it all, Linux panics and starts killing processes almost at random (the OOM killer), sometimes killing the wrong thing, like your SSH connection, taking the whole box down with it. The setup should have a layered defense so that a single greedy process slows itself down rather than crashing everyone.
This is about agent runtime setup, not agent interface or inference setup. Three things are different. For example, for Claude, agent inference is Anthropic's server, agent interface can be your phone, but agent runtime should be some file system accessible to Claude (not the model, but the program) local binary (which can be separate from the interface layer).
Layers
1. A big overflow area (swap)
Think of RAM as your desk and swap as a filing cabinet next to it. When the desk fills up, rarely-used papers move to the cabinet instead of being thrown on the floor.
I set up a 128GB swap file on its SSD. Slower than RAM, but slow beats crashed. One gotcha baked into the setup: the file must be created with dd (which writes the bytes directly), not fallocate (which creates a file with holes that Linux refuses to use as swap).
2. A compression buffer in front of swap (zswap)
Before pages go to the slow filing cabinet, Linux compresses them (with zstd) and keeps them in a corner of RAM, like vacuum-packing clothes before boxing them. Most of the time, the swapped data never actually hits the disk, so things stay fast.
zswap was chosen over its cousin zram because zswap's usage is billed to the process that caused it, which matters for Layer 4.
3. A polite bouncer (earlyoom).
The kernel's own OOM killer only acts at the very last second, when the system is already unresponsive. earlyoom is a small daemon that watches memory and steps in earlier: when both RAM and swap drop below 5% free, it politely asks the fattest process to quit (SIGTERM), and below 2% it forces the issue (SIGKILL). It has a do-not-kill list protecting the things that must survive: systemd, sshd, tailscaled, docker/containerd, and earlyoom itself.
4. Straitjackets for known-heavy jobs (cgroup caps).
Before starting something you know is hungry, wrap it.
That runs the command inside a memory-limited box. If it exceeds 8GB, the kernel kills only that box; the rest of the server never notices. This is the best layer because the blast radius is exactly one job.
5. Bodyguards for the way in
The SSH and Tailscale daemons get an OOM score of -500, which tells the kernel, "If you must kill someone, pick almost anyone else first." You can always get back in to fix things.
6. Plus tuning knobs (sysctls)
Swap eagerly into the compressed tier (swappiness=100), start background cleanup earlier so the box never stalls waiting for memory (watermark_scale_factor=125), and keep a bigger emergency reserve (min_free_kbytes).