At home, my family’s AI assistant runs entirely on hardware I own: a couple of small, always-on mini-PCs and a Mac, running local, open-source models. I made that choice for a family, not a company. But over the past year I’ve watched enterprise leaders arrive at the same architecture, for reasons that sound almost word for word like mine. The weights are coming home. The interesting question is not whether that’s right in principle. It’s which of your workloads should follow them.

Why a family assistant runs on local models
I described the setup in The Homelab Behind the Family AI. One constraint drove everything: the family’s data stays at home. The chat where we plan holidays and where my kids ask questions was never going to become someone else’s training signal. That single rule put the whole stack on local, self-hosted, open-source models — a small model for the everyday work, a tool-use model for planning, an embedding model for memory, and a larger local model when something needs more spark.
Two quieter reasons came with it. Predictable cost: the hardware is paid for, and a chatty week costs the same as a quiet one. Control: I can change any behaviour, pin any model version, and nothing changes underneath me because someone else shipped an update.
The enterprise version of the same reasons
Swap the family chat for customer records, clinical notes or trading data, and the reasons scale up without changing shape.
- Sovereignty. Where data is processed, and under whose jurisdiction, is now a board-level question in many regions. “Our data never leaves our environment” is a much easier sentence to defend than a page of contractual assurances.
- Regulated data. For some data, sending it out for inference is either forbidden or so heavily conditioned that the paperwork outweighs the benefit.
- Cost you can forecast. Agent workloads are spiky and hard to predict, as I covered in What Agents Cost to Run. Owned or reserved capacity turns a variable bill into a planned one, at the price of utilisation risk.
- Control. A model you host doesn’t change on someone else’s schedule. You decide when to upgrade, and you test before you do.
The surveys point the same way, with the usual caveat that the most visible ones are sponsored by companies selling the answer. A private-cloud vendor’s 2026 outlook, a survey of 1,800 senior IT decision-makers fielded in early 2026, found that public cloud’s share of production AI inference had fallen from 56% to 41% in a year. A sovereignty study by an analyst firm, sponsored by an open-source vendor and covering 1,940 respondents, found 26% preferring open-weight models they control against 18% preferring proprietary frontier models. Neither is neutral, and I wouldn’t bet a strategy on either number. But the direction matches what you hear in any room of CIOs and CDAIOs.
Workload triage: three buckets
“Should we self-host?” is the wrong question, because it treats AI as one workload. It’s dozens. Sort them into three buckets.
Must stay in your walls. Anything touching regulated or highly sensitive data, anything where residency is a legal requirement, and anything where a leak would be existential. These run on models you host, on infrastructure you control, full stop. They are usually narrower than people expect: classification, extraction, summarisation and retrieval over internal data, all of which open-weight models handle well today.
Can roam. Work on public or low-sensitivity data, internal drafting with nothing confidential in it, and experiments. Run these wherever is cheapest and fastest this quarter, and expect that to change.
Needs the biggest model available. A small set of genuinely hard reasoning tasks where the best model really does matter. Be honest about how small this set is. In my experience most “we need the frontier” claims turn out, under an eval, to be “we never tested anything smaller.” Where the need is real, decide deliberately what data may go with the task, and strip the rest.
The triage is a data classification exercise as much as an AI one. If your organisation already knows which data is sensitive and where it may live, you’re most of the way there. If it doesn’t, that is the real project.
Portability is the real requirement
Whatever the buckets hold today, they will move. Regulation changes, a new open-weight model closes a capability gap, a price drops, a contract ends. The architecture that survives is the one that lets a workload change buckets without a rewrite. That takes three things.
- A routing layer. Applications call a capability (“extract the fields from this contract”), not a specific model endpoint. A thin layer in the middle decides which model serves it, under which data policy. Moving a workload becomes a configuration change.
- Evals that travel. You can only move a workload with confidence if you can prove the new model does the job as well as the old one. That is the three-tier harness from The Model Upgrade That Quietly Breaks You, and it applies just as much to moving from one model to another as to upgrading in place.
- No prompt lock-in. The value should live in your data, your entity graph, your rules and your evidence, not in prompts tuned to one model’s quirks. That’s the argument of The Model Is the Least Defensible Part of Your Agent, seen from the infrastructure side: if the model is interchangeable, make sure your platform treats it that way.
The honest caveats
Running models is operations work. I own the updates, the memory limits and the quiet nights when something stops responding. At home that’s a hobby. In an enterprise it’s capacity planning, GPU scheduling, patching, monitoring and an on-call rota. Self-hosting moves cost from a bill to a team, and that team has to exist.
Open-weight isn’t open data. Having the weights tells you very little about what the model was trained on, how it was filtered, or what licence terms apply to its use. Read the licence, check the provenance, and evaluate it on your own cases before trusting it with anything that matters. Running a model in your walls protects your data from leaving. It doesn’t make the model itself trustworthy.
Utilisation is the hidden cost. Owned capacity is cheap when it’s busy and expensive when it’s idle. The triage helps here too: steady, high-volume workloads are the best candidates to bring home first.
The question to take away
The debate between open and closed models will run for years, and it’s mostly the wrong debate for building products and digital platforms. The right one is about placement and portability: knowing which AI and Data workloads must stay in your walls, and having a platform that can put them there.
Could you move your most sensitive AI workload in-house within a quarter — and if not, what exactly stops you? Whatever the answer is — no routing layer, no evals, prompts welded to one model, no idea where the data may live — that is your next investment, whichever way the weights end up going.
A companion to the Agentic AI series.
Related: The Homelab Behind the Family AI · The Substrate War — Will Your Data Platform Become Your System of Record? · What Agents Cost to Run.
Part of a personal DIY hobby, tinkered together on weekends for my own family, built entirely with open-source, self-hosted models.




Leave a Reply