From one agent to a team: what I learned about building AI agents at home

I have spent the last seven weeks grinding on building an AI agent platform in my home lab and the main thing it taught me is large agents with a large tool kit is the wrong design. A team of narrow, purpose-built agents, each with a small toolset and hard limits works best.

My goal

I wanted to learn what it takes to run AI agents responsibly, not just prompt them. My day job is troubleshooting systems rather than building them, so I set three requirements before working with Claude to write any code.

• Local first. Models run on my own GPU. Cloud models are a narrow exception, not the default.
• Controllable. Every agent gets its own credential, its own budget, and a kill switch I can pull without touching anything else.
• Observable. If an agent does something odd, I can trace one message end to end and see exactly what happened.

The first agent was a personal assistant I called Rocket. I told myself it was the first of many. I did not expect the first lesson to arrive in four days.

The initial infrastructure

I kept it simple, and nearly every layer traces back to something I wanted to learn or something I already had. The first working version took about four days.

Compute. Kubernetes is a growing part of my day-to-day work, so I wanted real hardware to learn it on, and it had to be cheap. I bought a used lot of five Dell OptiPlex 3020M Micros, the tiny small-form-factor PCs, for $158.96 total, about $32 each. Each has an i3-4160T, 12 GB of RAM and no GPU. Three came with a 500 GB hard drive and two came without one. I run k3s with one control node and no high availability. I don't need HA here, and I would rather fully understand a simple cluster than half-understand a resilient one.

Delivery. I already manage my infrastructure as code: I update git and it pushes from there. I wanted the agents managed the same way. Argo CD deploys them from a manifests repo, and their code lives in a separate repo. Every change is a reviewed merge request, and a code commit can never deploy itself.

Gateway. This one came out of working it through with Claude. We went back and forth and settled on LiteLLM in front of every model. Each agent gets its own key, budget and rate limit, so revoking one key blinds that agent without touching the rest.

Model. This has been a moving target, and I expect it to stay one. After researching it with Claude, gemma4:12b was a good starting point for the card I had, a RTX 5060TI I bought in January 2026 in preparation for AI projects, before the prices went crazy. It has native tool calling and a real system role. I have since paired it with a cheap refurbished Quadro RTX 4000 that adds 8 GB, which deserves its own article. Combined 24 GBs of RAM model split across cards.

Interface. I already had Discord, and it was already configured for notifications from my infrastructure-as-code pipeline and for Renovate updates. Rocket lives where I already look. His connection only goes out, so nothing listens and nothing dials in, and there is no inbound path to defend.

Observability. Logs, metrics and traces flow into Loki, Prometheus and Tempo, and I view them in Grafana. One message becomes one trace across the agent and the gateway.

The pattern behind these choices is that I picked boundaries I could revoke, and a way to see inside every one of them.

The beta phase: trials and tribulations

Rocket, my first agent. On day four, I asked him to do a review of my infrastructure. He gave me a confident, detailed review. My dashboards were shelfware and my certificates were rotated by hand. Everything he told me was a hallucination. He had no tools, and his instructions told him to go search when he did not know something. A model's job is to complete the request in front of it. So when it can't do what it was asked, it writes what the action would have produced.

I gave him four read-only tools that same afternoon, and within the hour I noticed he had turned into an infrastructure bot with a personality attached. A model with tools goes looking for reasons to use them. Two days later I made him a front desk that files tickets for specialists, and removed his access to the cluster entirely.

Then I over-hired. I stood up about two dozen specialist agents in a single morning, named after characters from different franchises. Within days I could not tell who was who or what each one was for. On October 4 I cut the roster to an IT team, a librarian and Rocket, and set a rule: add one team at a time, and only after the previous team is fully built.

The failures kept coming, and each taught something specific:

• Fake tool calls. He reported tasks that did not exist and "closed" one by writing text instead of calling the tool. The examples in his prompt showed tool names in brackets, and he copied the shape.
• A runaway reply. One answer ran to 11,468 tokens over nearly five minutes and only stopped when the context filled.
• A field the model could not see. A tool parameter named "description" was never filled in, in 23 of 23 attempts. Renamed "details", it was filled in 22 of 22.
• Mutes he never made. He confirmed two alert mutes, three times, with no tool call at all. The real silences had not changed. I caught it by reading Alertmanager, not the chat.

Every one of these became a written, blameless postmortem.

Lessons learned, and what I changed

The day I retooled Rocket I wrote down the thought that now drives everything else:

"Agents with lots of power/actions they can perform is what makes them weak. If we do many agents with each having a task it's very good at it works better."

I think of it the way I think about people. Give one person too many responsibilities and they turn in average or poor work on all of them. Give them a small set of specialized tasks and they do those very well. Agents behave the same way.

My hardware and model choices made the limit more obvious. A 12-billion-parameter model on a single 16 GB card works inside a context window of about 16,000 tokens, so every extra instruction and every extra tool competes for the same small space. That pushed me toward narrow agents sooner than a bigger setup might have.

Here is what that turned into.

1. One job per agent, with a small toolset. Rocket went from an operator to a front desk with seven ticket tools and no cluster access. The specialists each do one thing and can reach only what that thing needs.
2. Put controls in code, not in prompts. Telling a model not to change anything is a request. Giving it a credential that cannot change anything is a control. The IT team's update agent, for example, only recommends a merge, and a CI job merges the exact commit that was reviewed, and only if the agent actually fetched the changelog and quoted it.
3. Check the log, not the answer. If no tool call was logged, the answer was a guess, however plausible it sounds. Every agent is traced, so I can verify any claim after the fact.
4. Match the reasoning mode to the job. Thinking off is right for chat: replies are ten times faster. For diagnosis it is wrong. On one investigation task the same model with thinking off took 17 calls and about 8 wrong queries in about a minute. With thinking on it took 3 calls and almost none wrong, in about four and a half minutes. I accept the slower runs for the agents that investigate.
5. Add one team at a time. The next team does not start until the last one is fully built and watched.
6. Write every failure down. I keep a blameless postmortem for each incident, close to fifty of them, plus knowledge articles for what they teach. They are how I avoid relearning a lesson.

Where I am today

The platform now runs 22 agents in five groups, each added one team at a time. This is the live roster as of November 9, pulled from the platform itself. Each team is named after one franchise, which is how I fixed my problem of not being able to keep track of who is who.

Front desk

• Rocket, Chief of Staff. The one I talk to on Discord. Takes requests, files tickets for the specialists, relays results, and handles alert mutes and maintenance windows. Yes I have his persona based off the character from Guardians of the Galaxy for my own amusement.
  
  Examples:

IT team (The Matrix)

• Morpheus, IT Director. Leads the team.
• Oracle, Observability. Runs the scheduled sweeps of the lab and opens tickets for what it finds.
• Trinity, Level 2. First-line diagnosis: investigates an issue and works out the cause.
• Neo, Level 3. Takes the hard escalations and config fixes, on a cloud model. Has an approval gate before requests reach him.
• Architect, IaC and GitOps. Reviews update merge requests and checks for breaking changes in the update against our config. A CI job does the merging.
• Hamann, Facilities and Recovery. Backup status, restore drills and disk health.
• Trainman, Network. Watches the network gear, read-only.

Librarian

• TheIntersect, Librarian. Keeps the written library of postmortems, knowledge articles and decisions current. Manages docs.soe, our internal documentation site.

Procurement and research (Futurama)

• Fry, Researcher. Makes the case for an option: the new, the promising, the evidence for. (Optimist)
• Bender, Researcher. Makes the case against: failure reports, fake reviews, the well-tested cheap choice. (Skeptic)
• Leela, Editor. Compiles the findings into one brief and arbitrates between the researchers.
• Hermes, Buyer. Shortlists three options and prepares a recommendation card. Never buys.
• Farnsworth, Fit engineer. Checks whether a candidate will work in my lab and reads the primary documents.
• Morbo, Advocate. Argues one side of a question from its own sources.
• Linda, Advocate. Argues the other side.

News desk, print (Superman)

• Lois, Investigations Editor. Deep dives on incidents.
• Clark, Data Desk. Owns the numbers in the nightly edition.

News desk, broadcast (Anchorman)

• Ed, Producer. Runs the weekly SOE Mornings show. A weekly podcast written of the past weeks events in the home lab (BPMs, Uptimes, Issues, etc.)
• Veronica, Writer. Writes the script.
• Fantana, Fact-Checker. Checks the script against its sources.
• Ron, Comedy Punch-up. Adds the jokes. Only the broadcast gets the silliness.
  

Two things matter more than the roster. Routine updates are reviewed overnight and merged by a CI job, so the lab largely maintains itself. And every agent runs on its own key, behind a network policy, with a trace I can follow.

I will be honest about it. This is a one-person lab, not production, and nothing depends on it but me. I set a target that the first-line diagnosis agent should resolve at least three quarters of issues herself and escalate the rest, but as of October I did not yet have a clean way to measure that, and building the measurement is current work.

This is a process of constant iteration. It reminds me of training someone new in a technical role: introduce the job in bite-sized chunks and give them specific tools to make them efficient, rather than throwing them into the deep end.

What's next

This piece is the overview and what I've built so far. I plan to write a deeper article on each part, with the decisions, the failures and the numbers. Along with building out entirely new teams of agents as the existing teams mature and need less tweaking.

• The platform and Rocket. The infrastructure choices and the first agent, in full.
• Hiring and naming. How I over-hired, and the rules I set afterward.
• The IT team. How one alert becomes a diagnosis, a proposed fix, and a reviewed change.
• The news desk. How a site writes and checks its own nightly edition.
• Procurement and research. Agents that watch topics and recommend without ever buying.