I honestly don’t think anyone knows what they are really doing in any of the spaces I’m about to talk about. It’s weird for me to talk about these things myself, because I know enough to know what I don’t know, and I don’t like writing down things I don’t feel like I can be an authority on.
But I suppose there’s some value to train-of-thought. So here we go. This isn’t an organized article, it’s just some lessons from all the crazy stuff I’ve been doing for the last two years or so.
Quants and making models fit in local environments
zOmg this amazing model runs locally on a SINGLE Nvidia 6700xtwtfbbq! - no, it doesn’t. Well, it loads, and it runs, but it’s entirely useless in whatever configuration was used to get the benchmark.
The tradeoffs when running these things locally are very real. Quantizing the weights of the model itself and the “KV cache” lead to massive quality issues that are not immediately obvious, and the whole thing is basically useless (for agentic work) unless it has a big context window (which eats up more GPU memory).
My targets: ~128k of context, 8-bit KV cache, and 8-bit model weights. Q6 at the absolute worst for weights, and never quant the cache lower than 8 bit. It just falls on it’s face at long context without these things. Run a smaller model if you have to.
Dense models are smarter than MoE models all day every day for long context / multi-turn. MoE is fast and super useful for one-shot tasks. The DGX Spark is an MoE monster, but it’s mostly useless for ~27b+ dense models. Therefore run your delegation agents on a spark / MoE, and your orchestration agent on real GPUs and dense models.
K8s are not for homelabs.
Use anything else. K3s is infinitely better. I run k8s because that’s what I run at work; don’t be like me. K3s uses traefik. Traefik is so much better than nginx for basic ingress / proxy stuff.
My current single-node cluster
I have an intel 10900 with a radeon GPU running my k8s cluster. It is single-node right now, but at some point it may be multiple nodes. I am using Flux CD, and I am slowly moving services from the NAS / docker-compose setup to the k8s cluster. Duckiesays.com is already migrated.
I have a Hermes Agent running on the cluster with inference using the GPU in the node… gemma4-12b in this case. The agent is called “echo” and all she does is … k8s administration. The local inference endpoint is also used for Duckiesays, and will likely be leveraged for other things as needed.
That’s right. Kubernetes that administers itself. What could possibly go wrong?
I do most of my k8s administration by telling echo what I want and then looking at the changes made in git. Flux rolls out the changes eventually. It’s… it’s occasionally pretty nasty, especially when I write my requirements poorly. But most of the time it’s amazing.
“Worse still,” I also have a generic (qwen3.6-27b) AI assistant that can talk to echo. So I can tell my generic assistant to tell the other agent to deploy stuff. Multi-agent workflows! Not really.
Agent execution is by far a more impactful use case than agentic design.
IaC revolution is here
Infrastructure-as-code has always been interesting, but there’s always been a trade-off… “I could write some IaC for this, and that would take a while, or I could just go configure the server.” With the advent of agents, writing the IaC code itself is stupid fast, and once it’s in place, doing things is downright human… just tell your named bot to do the job.