Your scribe vendor, your coding assistant, and three teams with corporate cards are already calling language models. A BYOLLM gateway is how a health system turns that sprawl into one enforced path: models you chose, under a business associate agreement you hold, with a record an auditor will accept.
What a BYOLLM gateway actually is
BYOLLM stands for bring your own large language model. As a product feature, it means an application does not bundle a model. You point it at a model endpoint you control, and the application supplies only the workflow. Warp, Salesforce, Grain and a long list of enterprise tools now ship this setting, usually on their top tier, because their regulated buyers asked for it.
A BYOLLM gateway is the thing those settings point at. It is a single network endpoint, deployed inside your own cloud account, that speaks a familiar API shape, usually OpenAI compatible, and sits between every AI caller in your organization and every model those callers are allowed to reach. Three terms get mixed up constantly, so it is worth being precise:
- BYOK means you paste your own provider key into someone else’s product. Billing moves to you. Control does not.
- BYOLLM means you supply the endpoint or the weights. The application never decides which model runs.
- The gateway is the endpoint itself: the place where identity, data classification, redaction, routing, guardrails, budgets, and audit logging get enforced once, for everything.
The distinction matters because BYOK alone solves a procurement problem and leaves the compliance problem intact. If a vendor holds your key and decides which model to call, you have moved the invoice without moving the control. The gateway is what converts a billing arrangement into a technical safeguard.
The one sentence versionA BYOLLM gateway is the chokepoint where a health system can prove, per request, which model touched which class of data, under which agreement, in which region, on whose authority.
Why healthcare needs one more than anyone else
Every regulated industry wants model governance. Healthcare has a specific structural problem that makes the gateway close to mandatory: the compliance boundary is drawn around a contract, the business associate agreement, and that contract is scoped to named services and configurations rather than to vendors in general.
As of late 2026, Anthropic, OpenAI, Microsoft Azure OpenAI, Google Cloud Vertex AI and AWS Bedrock all offer BAA coverage for specific products and configurations. That phrase, specific products and configurations, is where the engineering work lives. Bedrock is included in the AWS business associate addendum for HIPAA eligible accounts when the workload runs in HIPAA eligible regions, which means eligibility is a property of your account configuration and your region choice, not a property of the logo on the model. A developer who swaps one model identifier for another inside the same provider console can move a workload outside your coverage without a single line of policy changing.
That is not a hypothetical. These are the five paths we find during assessments:
- Vendor defaults. An ambient documentation product ships with the vendor’s own pooled model. The signed BAA covers the vendor, and the vendor’s subprocessor list covers the model provider, but nobody on your side can enumerate which model saw which note.
- Shadow AI. A revenue cycle analyst pastes a denial letter with a member ID into a consumer chatbot. There is no BAA. There is also no log.
- Model drift inside an approved provider. A team adopts a new preview model in the same console. The provider is covered. That specific model, in that specific region, may not be.
- Logs and traces. The prompt was fine. The observability pipeline copied the full prompt body, including a patient name, into a third party logging SaaS with no BAA.
- Retrieval and tool calls. The user prompt contained no PHI, but the retrieval step pulled a chart excerpt into context, and the agent then called an external tool with it.
Each of these is an application level mistake. None of them can be reliably solved at the application level, because there are forty applications and the list grows every quarter. They are solved at the network level, once, by a component every caller has to pass through.
The reference architecture
Here is the shape we deploy. It is deliberately boring. The novelty budget in a healthcare AI program should be spent on clinical workflow, not on the plumbing that keeps you out of a breach notification.
Figure 1. The gateway sits inside your VPC and handles seven concerns in a fixed order. Callers never address a model endpoint directly. The evidence plane is a first class output, not a side effect of logging.
Four properties make this architecture work, and dropping any one of them collapses it:
- It runs in your network boundary. A hosted gateway that proxies PHI through a vendor’s infrastructure adds a business associate to your chain and a region you do not control. Self hosted or in-VPC deployment is the baseline for Tier A traffic.
- Order is fixed. Redaction happens before routing, routing happens before egress, and the audit event is emitted whether the request succeeds, is blocked, or times out. Blocked requests are the ones investigators care about most.
- There is no second door. If an application can still reach a model endpoint directly, the gateway is documentation, not a control.
- It is OpenAI compatible at the edge. This is pragmatic rather than ideological. Most vendor BYOLLM settings and most agent frameworks expect that shape, so compatibility is what lets you adopt the gateway without rewriting forty integrations.
Step 1: Inventory every call path you already have
Start by assuming the answer to “are we sending PHI to models today” is yes, and go looking for the proof. Five sources find close to everything:
- Egress logs. Query your firewall or proxy for the model provider domains over the last ninety days. Group by source workload and by volume.
- Expense data. Search card transactions and the AP system for provider names. Individual developer subscriptions show up here and nowhere else.
- Vendor contracts. Pull every active software agreement that mentions artificial intelligence, machine learning, or subprocessors. Note which vendor offers a BYOLLM or bring your own endpoint setting, and which tier it requires.
- Identity provider. List OAuth grants and SAML applications matching AI tooling.
- Your own code. Grep the monorepo for provider SDK imports and base URLs.
Record each path as a row: owner, application, data class it touches, provider, model, region, BAA status, monthly spend. That register is the single most useful artifact of the whole program. It is also what turns an abstract architecture conversation with your privacy office into a concrete, prioritized one.
Field noteIn most mid to large provider organizations we assess, the inventory turns up between three and five times as many live model call paths as the AI governance committee had on its list. The gap is almost never malicious. It is a product team enabling a feature flag inside a tool that was approved before the tool had AI in it.
Step 2: Write the data class routing table
This is the core design decision, and it is a policy decision before it is an engineering one. Every request that enters the gateway must carry a data class claim, and the data class determines which model tier the request may reach. Developers do not choose models. The table chooses models.
Figure 2. Four data classes, each with its own gateway action, allowed model tier, and logging mode. Default deny is the entry that makes the rest enforceable.
In practice this becomes a small declarative policy file that both your platform team and your privacy office can read. Keep it in version control, require a joint approval on changes, and make the gateway load it at runtime:
<span class="c-com"># gateway-policy.yaml</span>
<span class="c-key">tiers</span>:
<span class="c-key">A</span>:
<span class="c-key">description</span>: <span class="c-str">"PHI eligible, BAA covered, in-account"</span>
<span class="c-key">models</span>:
- <span class="c-str">"bedrock:us-east-1/anthropic.claude-*"</span>
- <span class="c-str">"azure-openai:eastus/gpt-*"</span>
<span class="c-key">require</span>: [<span class="c-str">baa_asserted</span>, <span class="c-str">no_train_flag</span>, <span class="c-str">private_networking</span>]
<span class="c-key">B</span>:
<span class="c-key">description</span>: <span class="c-str">"De-identified traffic only"</span>
<span class="c-key">models</span>: [<span class="c-str">"vendor-direct:*"</span>]
<span class="c-key">require</span>: [<span class="c-str">redaction_verified</span>]
<span class="c-key">C</span>:
<span class="c-key">description</span>: <span class="c-str">"Self hosted, never leaves the VPC"</span>
<span class="c-key">models</span>: [<span class="c-str">"vllm:internal/*"</span>]
<span class="c-key">classes</span>:
<span class="c-key">phi_raw</span>: { <span class="c-key">allow</span>: [<span class="c-str">A</span>], <span class="c-key">redact</span>: <span class="c-str">off</span>, <span class="c-key">log_body</span>: <span class="c-str">never</span> }
<span class="c-key">limited_data</span>: { <span class="c-key">allow</span>: [<span class="c-str">A</span>,<span class="c-str">C</span>], <span class="c-key">redact</span>: <span class="c-str">direct</span>, <span class="c-key">log_body</span>: <span class="c-str">hashed</span> }
<span class="c-key">deidentified</span>: { <span class="c-key">allow</span>: [<span class="c-str">A</span>,<span class="c-str">B</span>,<span class="c-str">C</span>], <span class="c-key">redact</span>: <span class="c-str">safe_harbor</span>, <span class="c-key">log_body</span>: <span class="c-str">full</span>, <span class="c-key">retain_days</span>: <span class="c-num">90</span> }
<span class="c-key">non_patient</span>: { <span class="c-key">allow</span>: [<span class="c-str">A</span>,<span class="c-str">B</span>,<span class="c-str">C</span>], <span class="c-key">redact</span>: <span class="c-str">detect_only</span>, <span class="c-key">log_body</span>: <span class="c-str">full</span> }
<span class="c-key">default</span>:
<span class="c-key">class</span>: <span class="c-str">phi_raw</span> <span class="c-com"># unclassified traffic is treated as the most sensitive</span>
<span class="c-key">on_no_tier_available</span>: <span class="c-str">reject</span>Two details decide whether this holds up under review. First, the default. An unclassified request must be treated as raw PHI, not as harmless, because the failure mode of the opposite default is a silent disclosure. Second, the detector runs even on traffic declared non patient. A hit on that path does not just log a warning. It reclassifies the request and reroutes it, which is what catches the analyst who pasted a denial letter into what they thought was a general purpose assistant.
Step 3: Pick the substrate your BAA actually covers
Tier A is where the compliance weight sits. The practical choice for most health systems is one of the three hyperscaler paths, with a direct provider API as a fourth option and self hosting as a fifth. The differences that matter are less about model quality than about what your legal team can point to.
| Tier A option | How coverage works | Choose it when | Watch for |
|---|---|---|---|
| AWS Bedrock | Listed as a HIPAA eligible service under the AWS BAA, with models from several providers reachable inside your own account | You are AWS native, want multiple model families behind one control plane, and need private networking patterns that keep traffic inside the VPC | Eligibility depends on account configuration and region. Anthropic as a subprocessor should be acknowledged explicitly in the agreement chain |
| Azure OpenAI | Covered under the Microsoft BAA through the enterprise agreement, with prompts and completions staying in your tenant | Your identity, data estate and security tooling are already Microsoft centric, or you need the OpenAI model family specifically | Per region model availability moves faster than your approval process. Pin deployments, not model names |
| Google Vertex AI | Covered as a service under the Google Cloud BAA, with Gemini models and native handling of healthcare data formats | You are on GCP, lean on BigQuery for analytics, or want tight coupling to the Cloud Healthcare API for FHIR workloads | FedRAMP High is not generally available for Vertex, which rules it out for some federal and contractor work |
| Direct provider API | Anthropic and OpenAI both offer BAAs for eligible API customers under specific configurations | You want the newest model the day it ships, or you are running a narrow high value workload with a dedicated agreement | Verify scope per endpoint. Coverage on the messages endpoint does not automatically extend to every beta surface |
| Self hosted open weights | No third party processes the data, so no additional BAA is required for inference | Bulk de-identification, embeddings, classification, and any workload with absolute residency requirements | You now own GPU capacity planning, patching, and model evaluation. Quality gap on complex clinical reasoning is real |
Most organizations end up with two Tier A substrates rather than one, because clinical workloads and analytics workloads have different latency and cost profiles. That is fine. It is exactly the case the gateway exists to handle, and it is far better than the alternative, where two substrates means two sets of controls maintained by two teams.
The trap worth namingHIPAA eligible is not the same as HIPAA compliant. Eligibility means the service can be used with PHI under your BAA. Compliance is a property of your whole deployment: the configuration, the access controls, the audit trail, the training, and the contracts. A gateway does not make you compliant. It makes the technical safeguards enforceable and the evidence producible, which is the part engineering actually owns.
Step 4: Build the redaction layer, and its failure mode
Redaction is where most healthcare gateway projects either earn trust or lose it. The goal is not to remove every trace of sensitivity. The goal is to strip the eighteen Safe Harbor identifiers from traffic that is leaving Tier A, in a way that is reversible for the caller and irreversible for the model provider.
Use deterministic tokens, not black boxes
Replacing a name with [REDACTED] destroys the model’s ability to do the task. Replacing it with a stable pseudonym preserves coreference, which is what clinical summarization depends on. Map each detected entity to a token that is consistent within a request, store the mapping encrypted and short lived, then rehydrate on the way back:
<span class="c-com"># simplified gateway redaction path</span>
detections = detect(prompt, detectors=[<span class="c-str">"clinical_ner"</span>, <span class="c-str">"pattern_mrn_ssn_dob"</span>, <span class="c-str">"structured_fhir"</span>])
mapping = {}
<span class="c-key">for</span> ent <span class="c-key">in</span> detections:
token = stable_token(ent.type, ent.value, request_id) <span class="c-com"># PERSON_1, DATE_2, MRN_1</span>
mapping[token] = ent.value
prompt = prompt.replace(ent.value, token)
<span class="c-key">if</span> residual_scan(prompt).hits: <span class="c-com"># second, independent detector</span>
<span class="c-key">return</span> reclassify_to_tier_a(request) <span class="c-com"># fail closed, do not fail open</span>
response = route(prompt, tier=policy.tier_for(request))
<span class="c-key">return</span> rehydrate(response, mapping) <span class="c-com"># caller sees real values, provider never did</span>Assume the detector is imperfect, and design for that
Named entity recognition on clinical text misses things. It misses unusual names, it misses identifiers embedded in free text, and it misses PHI inside scanned documents entirely unless you are also running OCR aware masking. Three design choices contain that risk:
- Two detectors, not one. A statistical model plus deterministic patterns for the structured identifiers, with a residual scan after the first pass.
- Fail closed. A residual hit reroutes the request to Tier A rather than letting it continue to a model outside your BAA. Rerouting is cheap. A disclosure is not.
- Never treat redaction as de-identification for release. Gateway redaction is a technical safeguard that reduces what leaves your boundary. Formal de-identification under Safe Harbor or expert determination is a separate, documented process with its own sign off.
Run the whole layer in log only mode for several weeks before you enforce it. Publish the precision and recall numbers to your privacy office using your own clinical text. That report, more than any architecture diagram, is what gets the enforcement switch approved.
Step 5: Treat the audit log as the product
The HIPAA Security Rule requires audit controls: mechanisms that record and examine activity in systems containing electronic PHI. For AI traffic, meeting that requirement means being able to reconstruct a model interaction months later without the log itself becoming an unnecessary PHI repository. Those two goals pull against each other, and the resolution is to log structure and identity richly while logging content narrowly.
A workable event shape:
{
<span class="c-key">"event_id"</span>: <span class="c-str">"01JB7Z4K9Q..."</span>,
<span class="c-key">"ts"</span>: <span class="c-str">"2026-10-23T14:02:11.418Z"</span>,
<span class="c-key">"actor"</span>: { <span class="c-key">"user"</span>: <span class="c-str">"u_8841"</span>, <span class="c-key">"app"</span>: <span class="c-str">"ambient-scribe"</span>, <span class="c-key">"purpose"</span>: <span class="c-str">"treatment"</span> },
<span class="c-key">"subject"</span>: { <span class="c-key">"mrn_hash"</span>: <span class="c-str">"sha256:9f1c..."</span>, <span class="c-key">"encounter_hash"</span>: <span class="c-str">"sha256:02ad..."</span> },
<span class="c-key">"policy"</span>: { <span class="c-key">"data_class"</span>: <span class="c-str">"phi_raw"</span>, <span class="c-key">"tier"</span>: <span class="c-str">"A"</span>, <span class="c-key">"version"</span>: <span class="c-str">"2026.10.02"</span> },
<span class="c-key">"gen_ai"</span>: {
<span class="c-key">"system"</span>: <span class="c-str">"aws.bedrock"</span>,
<span class="c-key">"request.model"</span>: <span class="c-str">"anthropic.claude-...-v2:1"</span>,
<span class="c-key">"region"</span>: <span class="c-str">"us-east-1"</span>,
<span class="c-key">"usage.input_tokens"</span>: <span class="c-num">3184</span>,
<span class="c-key">"usage.output_tokens"</span>: <span class="c-num">612</span>
},
<span class="c-key">"redaction"</span>: { <span class="c-key">"mode"</span>: <span class="c-str">"off"</span>, <span class="c-key">"detector_hits"</span>: <span class="c-num">0</span>, <span class="c-key">"residual_scan"</span>: <span class="c-str">"clean"</span> },
<span class="c-key">"content"</span>: { <span class="c-key">"prompt_sha256"</span>: <span class="c-str">"c41d..."</span>, <span class="c-key">"body_stored"</span>: <span class="c-key">false</span> },
<span class="c-key">"outcome"</span>: { <span class="c-key">"status"</span>: <span class="c-str">"ok"</span>, <span class="c-key">"latency_ms"</span>: <span class="c-num">1840</span>, <span class="c-key">"cost_usd"</span>: <span class="c-num">0.0197</span> },
<span class="c-key">"integrity"</span>: { <span class="c-key">"hmac"</span>: <span class="c-str">"..."</span>, <span class="c-key">"prev_event_hash"</span>: <span class="c-str">"..."</span> }
}Five properties to insist on:
- Hash the patient identifiers. You can still answer “every model call touching this patient” by hashing the MRN at query time. You are not storing a searchable patient index in your observability stack.
- Record the resolved model identifier and region, not the alias the application asked for. The alias tells you intent. The resolved identifier tells you what your BAA has to cover.
- Chain and sign the events. An audit trail that can be edited silently is worth very little during an investigation. Hash chaining plus HMAC gives you tamper evidence without a blockchain conversation.
- Log blocked and failed requests with the same fidelity. A denial is a security event and often the most informative row in the table.
- Follow the OpenTelemetry GenAI semantic conventions for the model attributes. It costs nothing at build time and makes your traces legible to tooling you have not bought yet.
Break glass access to the stored content, where you keep any, should require dual approval, generate its own audit event, and notify a named owner. If your platform cannot do that, it is not ready for Tier A traffic.
Step 6: Make vendors come to your endpoint
The gateway only pays for itself once third party AI products route through it, because that is where most of your PHI exposure lives. This is a procurement motion as much as an engineering one, and it works best as a short standard clause plus a short technical checklist.
Add to the security review, for any product with AI features:
- Does the product support a customer supplied, OpenAI compatible inference endpoint, and on which tier?
- Which features silently fall back to the vendor’s own models when the custom endpoint errors or times out? Fallback behavior is the most common gap, and it is almost never in the datasheet.
- Does the vendor’s embedding, reranking, speech to text, or safety classifier traffic also respect the custom endpoint, or only the main completion call?
- What is logged on the vendor side after the switch, and for how long?
- Which subprocessors remain in the chain once inference moves to you?
The first two questions resolve most of the risk. A scribe vendor that routes completions to your gateway but sends audio to its own transcription service has moved perhaps half of the exposure, and you need to know which half before you tell your board the problem is solved.
Step 7: Close the bypass
A gateway nobody is required to use is a convenience layer. The final step, and the one most programs defer indefinitely, is making the gateway the only reachable path:
- Block egress to model provider domains at the firewall and the cloud egress policy, allowing only the gateway’s own service identity.
- Remove direct provider credentials from application secrets stores and rotate anything that leaked. Issue per application gateway keys instead, scoped to a data class and a budget.
- Deny the provider SDK base URL override in code review, or better, ship an internal client library whose default base URL is the gateway and whose configuration cannot be overridden in production builds.
- Alert on any egress attempt that bypasses the gateway, and treat it as a finding rather than a bug.
Expect friction here. The honest framing for your engineering organization is that the gateway removes a decision they never wanted to own: which model is legally allowed to see this data. In exchange for the chokepoint they get centralized retries, failover, caching, cost attribution, and the ability to adopt a new model everywhere by changing one policy file.
Build, buy, or assemble
Nobody should write a model gateway from scratch in 2026. The realistic options are an open source gateway you host, a commercial gateway deployed in your VPC, or your cloud provider’s native stack extended with your own policy layer.
| Approach | What you get | What you still build |
|---|---|---|
| Open source gateway LiteLLM and similar, self hosted |
Unified API across a very wide provider set, key management, budgets, basic guardrail hooks, full control of deployment | Clinical grade PHI detection, the data class policy engine, signed audit chain, break glass workflow, and the operational burden of running it |
| Commercial gateway, in VPC Portkey, Kong AI Gateway, Bifrost, F5 and others |
PII and PHI redaction modes, role based access, audit logging, SIEM export, MCP tool governance, vendor support for the review | Detector tuning on your own clinical corpus, data class taxonomy, integration with your identity provider and privacy workflows |
| Cloud native plus policy layer Bedrock, Azure AI Foundry, Apigee |
Compliance posture inherited from a provider you already have a BAA with, native logging into CloudWatch or Azure Monitor | Cross provider routing, the redaction layer, and a policy model that is portable if you ever add a second substrate |
Our recommendation for most health systems: take a gateway off the shelf, deploy it in your own VPC, and spend your engineering effort on the three things no vendor can give you. Those are the data class taxonomy mapped to your actual workflows, the detector tuned on your actual clinical text, and the evidence pipeline shaped to the questions your actual compliance office asks. That is roughly a six to ten week build with a small team, not a year.
A 90 day rollout
The sequencing below is designed so that each phase ends with an artifact somebody outside engineering can read. That is what keeps the program funded past the first budget review.
Figure 3. Thirteen weeks from inventory to enforced path. The phase boundaries are deliberately set by artifacts rather than by code milestones.
Two scheduling notes from projects that went well. First, run the tabletop exercise in week nine, not week thirteen. Walking your privacy officer through a simulated disclosure while there is still time to change the logging schema is worth more than a polished readout at the end. Second, pick the first enforced workload for its clarity, not its size. A single internal copilot with one owner and a well understood data class beats an enterprise wide rollout that nobody can roll back.
Five mistakes we keep seeing
- Treating the gateway as a compliance product. It is an engineering control that produces compliance evidence. If the program is owned by the privacy office with engineering consulting, it stalls. Reverse that.
- Logging everything, then discovering the log is a PHI repository. Decide the content retention policy before the first line of code. Retrofitting redaction into an existing observability pipeline is painful and usually incomplete.
- Shipping redaction in blocking mode on day one. False positives on clinical text destroy user trust in the first week, and you do not get a second launch. Shadow mode first, always.
- Pinning to model aliases instead of resolved identifiers. An alias that silently rolls forward to a new model version moves your workload without any change you approved.
- Declaring victory before the bypass is closed. Until direct egress is blocked, adoption of the gateway is voluntary, and voluntary controls are not controls.
Frequently asked questions
What is a BYOLLM gateway?
A BYOLLM gateway is a single endpoint, hosted inside your own cloud environment, that every application and AI vendor calls instead of calling a model provider directly. It authenticates the caller, classifies the data in the request, redacts protected health information when policy requires it, routes the request only to models your organization has approved and has a business associate agreement for, and writes an audit record of what happened. BYOLLM stands for bring your own large language model.
How is BYOLLM different from BYOK?
BYOK, bring your own key, means you supply an API key to a vendor’s product and the vendor keeps deciding which model to call and how. BYOLLM means you supply the endpoint or the model itself, so the application never makes the model choice. For healthcare, the difference is decisive: BYOK moves the invoice to you, while BYOLLM plus a gateway moves the control to you, including region pinning, redaction, and the audit trail.
Does a BYOLLM gateway make our AI deployment HIPAA compliant?
No. Compliance is a property of your whole deployment: executed business associate agreements, administrative and physical safeguards, workforce training, access management, and incident response. What a gateway does is make the technical safeguards enforceable on every request rather than voluntary per application, and make the evidence producible when somebody asks. It is a necessary component, not a certificate.
Which model providers will sign a HIPAA business associate agreement?
As of late 2026, Anthropic, OpenAI, Microsoft Azure OpenAI, Google Cloud Vertex AI and AWS Bedrock all offer BAA coverage, but each covers specific products and configurations rather than everything the company sells. Bedrock is listed as HIPAA eligible under the AWS BAA when used in HIPAA eligible regions with an appropriately configured account. Azure OpenAI is covered under the Microsoft agreement. Vertex AI is covered under the Google Cloud BAA. Verify the scope per service, per endpoint, and per region, and re-verify when you adopt a new model.
Should we redact PHI before sending it to a model we have a BAA with?
Not necessarily. If the model is covered by your BAA and the workload genuinely needs identified data, such as drafting a patient message, redaction degrades quality for no compliance gain. The minimum necessary principle still applies, so send only the fields the task requires. Redaction becomes mandatory on the path to any model outside that coverage, which is exactly the boundary the gateway enforces.
How long does it take to build one, and who do you need?
With a gateway taken off the shelf and deployed in your own VPC, plan on roughly six to ten weeks of build and a 90 day arc to an enforced path on a first workload. The core team is small: two platform engineers, one security engineer, a clinical informaticist to validate the detector on real text, and a named owner in the privacy office who can approve policy changes without a committee cycle.
What does it cost to run?
The gateway itself is usually a rounding error next to inference spend, typically a couple of modest container instances plus the detector, which is the only latency sensitive component worth sizing carefully. The savings usually show up in two places: consolidated routing lets you move routine workloads to cheaper models without touching application code, and per application cost attribution ends the quarterly argument about whose AI spend is growing.
