Connect your own and local LLMs
Connect local or self-hosted language models through the model panel: vLLM, NVIDIA NIM, Kimi K2, or any OpenAI-compatible endpoint. Governance (data classes, residency) stays enforced.
What it is for
Section titled “What it is for”Alongside the existing Bedrock models, the model panel lets you register your own endpoints: a cluster-hosted vLLM, an NVIDIA NIM microservice, a hosted Kimi endpoint or any other OpenAI-compatible service. The prerequisite is a «/v1»-compatible API.
Creating an entry tests the connection; if the test fails, no entry is created. The API key is a write-only field. It is stored encrypted in LiteLLM and never read back.
Example configurations
Section titled “Example configurations”vLLM (in cluster)
endpoint_url: http://vllm.aif-platform.svc.cluster.local:8000/v1
Self-hosted model (e.g. Kimi K2) behind vLLM, reachable in the cluster. A placeholder is enough for «api_key» if vLLM runs without authentication.
NVIDIA NIM (in cluster)
endpoint_url: http://nim.aif-platform.svc.cluster.local:8000/v1
NIM microservice (e.g. Nemotron), OpenAI-compatible, reachable in the cluster. Endpoint analogous to vLLM; «api_key» depending on the NIM configuration.
Kimi K2 (external, hosted)
endpoint_url: https://api.moonshot.ai/v1
Hosted Kimi endpoint (Moonshot). For «api_key», use the key of the Moonshot account. Because the endpoint is external, sovereign data classes are blocked without an override (see «Data classes and residency gate»).
Data classes and residency gate
Section titled “Data classes and residency gate”Every model is assigned data classes that determine which content may run through it:
- C0 (experimental): trials without protected data only; the lowest tier.
- C1 (internal): general content that is not especially sensitive.
- C2 (confidential): sensitive, restricted audience.
- C3 (strictly confidential): sovereign, highest protection level.
The residency gate links class and location: an external endpoint must not be assigned a sovereign class (C3 by default), except with an audited override and a justification. This hard boundary lives in the backend; the interface requires a justification for that combination.
For external models, the gate blocks only the sovereign classes. C0, C1, and C2 can use external models, including models that process data outside the EU and Switzerland. Wherever an external model answers, a notice names the provider and the place of processing.
Connect a model
Section titled “Connect a model”Creating and managing models runs through the platform panel (platform admins only).
Connect a model: Platform → Models