I’ve written about building a bilingual AI agent for my family over four weekends — the features, the warmth, the moment a “did you know?” became a conversation with my kids. This is the companion piece for the engineers: what’s actually under the hood, and the choices I’d defend. It’s a homelab story as much as an AI one.
Why run it all myself
One decision shaped everything else: the family’s data stays at home. The chat where we plan holidays, track spending, and where my kids ask questions — none of that was going to be a training signal for someone else’s model, or a row in a vendor’s database. That single constraint — privacy first — is what pushed the whole thing onto local, open-source models running on hardware I own. Everything downstream is a consequence of it.
The other two reasons were quieter but real: control (I can change any behaviour without waiting on a roadmap) and learning (running the full stack yourself teaches you things a hosted API never will).
The homelab, honestly
It started the way these things do: running on my Mac. That’s fine until you want the assistant up at 7am whether or not your laptop is open. So it grew — a couple of small, low-power mini-PCs became always-on workers, with the Mac staying in the mix for the heavier model work. They talk to each other over a private home network (a mesh VPN, so nothing is exposed to the open internet), and the family chat reaches them through a single, narrow, authenticated entry point.
Nothing exotic — deliberately. A homelab for a family assistant should be the kind of thing that survives a power cut and a distracted owner, not a rack that needs babysitting. Boring, always-on, and cheap to run beats clever every time.
The models — local and open-source
There isn’t one model; there’s a small ensemble, each doing what it’s good at:
- A small local model for the everyday, high-volume work — turning the day’s facts into a warm briefing, quick classification, translation.
- A Hermes model as the planning and tool-use brain — the part that decides “this looks like it needs a reminder” and proposes an action.
- An embedding model for memory — so the assistant can recall commitments and facts by meaning, not just keywords.
- A larger local model for the parts that need more spark — the jokes, the word of the day, the “did you know?” that has to be genuinely fresh.
The honest trade-off: local models are slower than the big hosted ones, and you own the operations — the updates, the memory limits, the 2am “why is it quiet.” I took that deal happily, because for a family assistant the currency isn’t raw capability. It’s trust. And trust is much easier to earn when the data never leaves the house.
Guardrails, in layers
Pointing generative models at your own family means guardrails aren’t a feature you add later — they’re the frame you build inside. Mine are layered so no single failure is catastrophic:
- Propose, don’t act. Every action with a real-world effect becomes a proposal a human approves. The model can suggest anything; it can only do what someone tapped Approve on — and it’s logged.
- Never transact. For things like travel, the assistant finds and surfaces options and hands off a link. It never holds a card or completes a purchase.
- Kid-safe generation. Anything aimed at the children passes an age-appropriateness filter, with a curated fallback so a bad model moment never reaches them.
- Rate limits and a safety preamble on every model call — the mundane hygiene that stops a loop or a bad prompt from doing damage.
Defence in depth, but pointed at a gentle goal.
Privacy by design, not by policy
Privacy here isn’t a paragraph in a settings page; it’s in the data model:
- Home-only data. The substrate lives on my hardware. What leaves the network is limited to what genuinely must (a weather lookup, a public search) — never the family’s conversations.
- Direct messages never enter the family digest. A private note to the assistant is control-plane only; it can’t surface in a summary the whole family sees. That boundary is enforced in code and guarded by a test, not left to good intentions.
- Homework stays private; effort stays visible. The kids’ graded study — quizzes, marks, dictation, the mistakes they made — lives in their own DM and never reaches the family digest. A parent sees that the work got done, never what they struggled with. Same seam as the digest boundary: content private, signal visible, separated in code.
- Real forgetting. Remove a member and their data becomes a tombstone — the system won’t silently reactivate it. Deletion means deletion.
These are the same seams that matter in any serious AI or data platform; I just got to build them where the stakes were personal.
How the architecture scales
Features pile up fast when the users live with you. A few structural choices kept it maintainable:
- Ground the facts, then generate. A deterministic step reads real data from the database; the model only ever phrases those facts. This one split kills a whole class of hallucination and makes every feature testable.
- Best-effort, isolated jobs. The periodic work — digests, reminders, checklists, the daily “did you know?” — each runs in its own savepoint. One failing job can’t poison the rest.
- Feature flags everywhere. Every capability ships behind a switch, defaulting off. New things go live deliberately, and roll back instantly.
- Run a hobby like production. Strict typing, real tests, database migrations, CI that refuses to ship broken code. Overkill for a family bot — until the “incident” is your family not getting their morning briefing.
The choices I’d defend
If there’s a through-line, it’s boring where it can be, strict where it matters. Local models over hosted ones (privacy). A human gate over model confidence (safety). Surface-and-hand-off over transact (restraint). Deterministic data over generated data (truth). None of these are the flashiest option. All of them are the trustworthy one — and trust is the only metric that matters when your users are the people you love.
What’s next
The roadmap is honest and small: better local models as the open-source frontier keeps moving; a few more everyday features the family keeps asking for; and, increasingly, writing up the patterns — because the propose-don’t-act gate, the ground-then-generate split, and privacy-in-the-data-model are the same instincts that make any AI or data product worth trusting at scale. The homelab is where I get to practise them with the stakes turned all the way up.
Part of a personal DIY hobby, tinkered together on weekends for my own family, built entirely with open-source, self-hosted models.
Leave a Reply