Red Oak logo
A self-hosted AI workstation with a glowing network above it

Local AI Infrastructure

This site runs on local AI. Not as an experiment. As the operating model.

The pages you're reading were written, reviewed, and published by an agent running on hardware I own. The models are on four GPU boxes in my building. The data stays on my disks. The AI side of it is fully local: inference, data, and workflow all run on the home network and a Tailscale mesh. The site itself is served through Cloudflare, which is just delivery, not the brains.

What runs on it

  • The blog pipeline: posts drafted, reviewed, and published by the agent
  • Client work: site builds, audits, automation
  • Scheduled jobs: nightly crawls, benchmark runs, report generation
  • Email and research workflows through the agent fleet

The point is not that I can run a model at home. The point is that a real business runs on it.

The hardware

Four inference nodes, all running the same model file (same sha256 on every box, which makes upgrades a one-command job):

  • Two NVIDIA nodes: RTX 3090, 24GB each, llama.cpp on CUDA, 204,800-token context
  • Two AMD nodes: Radeon R9700, 32GB each, llama.cpp on ROCm, 262,144-token context

That's 112GB of GPU memory across four consumer parts. No data center. No procurement cycle. The boxes sit on the network next to everything else, and a fifth host runs the gateway.

The model

Qwen3.8-27B, Unsloth Q4_K_XL quantization, one 17.66GB file. Same weights on all four nodes.

One model, four replicas, beats a zoo of small models for a business workload. When a new quant or a new llama.cpp build comes out, I test it on one node, benchmark it, then roll it across. The last rebuild was a security call, not a performance call: the running build was 296 releases behind and carried an unpatched RCE in the completion endpoint. We rebuilt to the current release the same day.

None of this came together in a weekend. I've put a lot of hours into building, testing, and tuning the stack: swapping quantizations, benchmarking builds, adjusting context limits. I did it because I wanted to actually learn how it works and get the best experience possible out of it. The numbers on this page are the payoff for that time.

The routing

A LiteLLM gateway sits in front of the four nodes. It exposes two auto-routers that tier each request by complexity: routine work goes to a thinking-off lane for speed, hard problems get routed to a thinking-on lane where the model reasons before it answers. Each node has both lanes.

The router overhead is negligible. If a node dies, the request lands on another one. I've watched that happen under load, and the fleet just keeps going.

The gateway isn't just a router. It adds response caching and compression, which cuts redundant inference and bandwidth. It's also the security boundary: every request passes through an authorization key, and I can monitor usage, so I can see exactly what's consuming the fleet.

The numbers

Measured on this hardware, not from a vendor page (November 2026):

  • 70–110 tokens per second warm, through the router
  • 150–220 tps at peak on short contexts
  • ~60–63 tps on the 24GB CUDA nodes in fair A/B benchmarks
  • Long-context decode slows to 20–37 tps around 170k tokens. That's the attention cost. It's the one number on this page that goes the wrong way, and I'm not hiding it.

The marginal cost of a query is electricity. The hardware is already paid for. That changes what you're willing to automate.

What it doesn't do

Local AI is not everything, and I don't sell it that way.

The frontier commercial models still win the hardest reasoning tasks, and by a real margin. I tried local image generation and moved it to a commercial service, because the local hardware doesn't match the results and I don't care enough about the savings to pretend otherwise.

Running this is also an ops discipline. It's not set-and-forget: builds get patched, models get swapped, context limits get tuned. That's the trade. You own the whole stack, including the maintenance.

That's my experience, not a verdict. Plenty of businesses are fine on a commercial API. If your usage is light and your data is clean, the cloud is a good deal.

What this means for your business

The pattern here is the service. I audit what you actually run, figure out which of it belongs on your own hardware, build the stack, and keep it running. The business case writes itself once you see the token bill you're paying today.

The full argument, including what happens when commercial AI prices go up, is in the post: Why Local AI Matters for Business.

I'd like to talk about yours. Let's talk.