A gateway so my data knows which AI it can talk to

I use AI models constantly — local ones on my own GPU, and cloud ones when a problem needs more capability than 8 GB of VRAM can hold. That split creates a problem nobody’s default setup solves: not all of my data should be allowed to reach all of those models. Financial records I hold as a trustee should never leave my machine. Personal notes can go to a vetted provider with a decent privacy posture. A generic coding question can go to whoever’s cheapest.

Most people handle this with discipline: remember not to paste the wrong thing into the wrong chat box. I wanted it handled the way any other security boundary is handled — by a system that makes the wrong thing impossible, not merely discouraged.

One door, three tiers

The design is a self-hosted LLM gateway: a single proxy running on my own machine that every AI call — from my scripts, my apps, my experiments — must pass through. Nothing in my fleet talks to a model provider directly. One egress door.

Behind that door, data is classified into three tiers, and each tier gets its own virtual API key:

• Restricted — fiduciary, financial, family. Its key can only resolve to local models, running on my own GPU. Requests to any cloud provider don’t fail politely — the key simply has no right to reach them.
• Internal — personal but not sensitive. Its key reaches a shortlist of vetted providers, chosen on data-handling terms, not vibes.
• Public — generic work. Its key can use everything, routed for cost.

The application never decides where data may go. The key it holds decides. A tool that processes restricted data is handed the restricted key, and from that moment the strongest guarantee in the system isn’t a policy document — it’s an authorization error.

“Prove it” is the whole point

The part of this project I’m most attached to isn’t the routing — it’s the canary. After deployment, I tested the boundary the way an attacker would: take the restricted key and ask for a broad-licence cloud model. The gateway’s answer is a hard 403. Take the public key, ask for the local model: 200. Every meaningful edge got the same treatment, and re-running that canary is now part of touching the config at all.

It sounds obvious, but it changed how I think about every system I run: a security property you haven’t tested is a hope, not a property. The first version of my persistence setup had exactly one clever assumption in it, and the assumption was wrong; the canary habit is what caught that class of thing.

Some other lessons that survived contact with reality:

• Fallbacks are a boundary risk. Routers love to retry a failed request against a different model. If a restricted request can “fall back” to a cloud model, the whole tier system is decorative. Every fallback chain has to stay inside its source tier — and the restricted tier’s fallback list is deliberately empty.
• Logs become a data store. A proxy that logs prompts has quietly become a second copy of everything, including the sensitive tier. Content redaction in the spend logs isn’t optional; metadata is enough.
• Supply chain counts. The gateway itself is the most privileged process in the system — it holds every provider credential. Version-pinned, signature-verified images only; no :latest.

The local tier is what makes it honest

A tier that says “this data never leaves the machine” only means something if there’s a capable model on the machine. The local tier runs open-weight models on my own GPU, and for a while it also fronted a genuinely frontier-class open model streamed off NVMe — slow enough to be an overnight oracle rather than a chatbot, but entirely mine. That’s the quiet thesis of the whole build: sovereignty isn’t refusing the cloud, it’s being structurally selective about it — cloud capability where the data permits, owned capability where it doesn’t, and a machine-enforced line between the two.

It’s been running for weeks now as ordinary infrastructure: my experiments authenticate with whichever tier key matches their data, new tools inherit the rules for free, and I’ve stopped having to remember anything. That’s what security that works feels like — boring.