Building a Sovereign AI POC: Local LLMs and RAG

The first sovereign AI proof of concept I built for a regulated workflow started with a hard constraint: no document, embedding, or model weight could leave the customer's network. No OpenAI, no Anthropic, no managed vector database with a foreign control plane. Just a rack, a GPU, and a promise that a lawyer could read the network diagram without flinching.

That constraint changes almost every default in a modern RAG stack. Below is what actually worked, what I threw away, and the numbers I would use to defend each choice in front of a security review.

What "sovereign AI" actually means in a POC

A sovereign AI system keeps model weights, inference, embeddings, retrieval indexes, logs, and prompts inside an organization's legal and network boundary. In practice, for a POC, that means self-hosted open-weight models, a local vector store, and an audit trail the compliance team can inspect without a vendor NDA.

The word gets abused. I have seen "sovereign" used to describe SaaS with EU data residency, which is not the same thing. Data residency answers where the bytes sit. Sovereignty answers who has the technical and legal ability to read them. If the control plane, KMS, or model provider can be compelled by a foreign court, you have residency, not sovereignty.

For a POC in a regulated industry (finance, health, defense, legal), I treat the following as non-negotiable:

• Model weights on infrastructure the customer controls (bare metal, private cloud, or air-gapped).
• Embeddings and vector index inside the same trust boundary as the source documents.
• No telemetry leaving the boundary. No "we send anonymized prompts for improvement".
• A written data flow diagram. Every arrow ends inside the boundary.

Everything else (model choice, chunking, reranker) is engineering. These four are policy.

Choosing the local model: what to actually run in 2026

For a POC in 2026, the sensible default is a Llama 3.3 or Qwen 2.5 class model in the 32B to 70B range, quantized to Q4 or Q5 for GPU efficiency, served through vLLM or llama.cpp behind an OpenAI-compatible endpoint. That gives you API portability if the customer later moves to a hosted frontier model, and enough quality that the RAG layer, not the model, is the bottleneck.

The trade-off I keep hitting:

For most sovereign POCs, a 32B model with a good retrieval layer beats a 70B model with mediocre retrieval. I have watched teams burn a quarter chasing a bigger model when their real problem was that chunk 4 out of 5 was noise.

A few practical notes from actually running these:

• Serve with vLLM if you have a GPU with paged attention support. Throughput is 3 to 5x llama.cpp on the same hardware once concurrency goes above one user.
• Keep an OpenAI-compatible interface. Every client library, every agent framework, every eval tool speaks it. Do not invent a bespoke protocol.
• Fix your model version. "Latest" is a footgun in regulated environments. Pin the exact weights hash and record it with every generation for audit.

For the smallest embedded workflows (a laptop demo for a stakeholder), Ollama with a 7B or 8B model is fine. Do not ship it to production. It is a demo tool.

The RAG pipeline: where the actual quality lives

The retrieval layer is where sovereign POCs succeed or fail. The model is a commodity now. The pipeline is not.

Here is the shape I use, refined across several production RAG builds including the local-LLM RAG pipelines I run inside BizFlowAI ContentStudio:

1. Ingestion: parse documents into structured blocks (title, section, table, footnote). PDFs are 60% of the work. Use a real parser (Unstructured, LlamaParse self-hosted, or a custom one for a known format), not a naive text extractor.
2. Chunking: semantic chunks of 400 to 800 tokens, with 10 to 15% overlap, respecting section boundaries. Never chunk mid-table.
3. Embedding: a local model like bge-m3 or nomic-embed-text-v2 for multilingual, bge-large-en-v1.5 for English-only. Both run on CPU acceptably for a POC corpus under 500k chunks.
4. Storage: Postgres with pgvector. One container, one backup story, one thing to explain to the DBA. For corpora above a few million chunks, Qdrant self-hosted is my next stop.
5. Retrieval: hybrid search. Dense (pgvector) plus BM25/FTS, fused with Reciprocal Rank Fusion. I wrote about this in detail in an earlier post; RRF consistently beats either channel alone on messy enterprise text.
6. Reranking: a cross-encoder like bge-reranker-v2-m3 on the top 30 candidates, returning the top 5. This one step usually moves answer quality more than any prompt tweak.
7. Generation: the local LLM with a strict prompt that forbids answering outside the retrieved context and requires citations by chunk ID.

The reranker is the step teams skip. Do not skip it. On one internal corpus of about 180k chunks, adding a reranker moved top-1 retrieval accuracy from 61% to 84% on a hand-labeled eval set of 200 questions. The model did not change. The prompt did not change. Just the reranker.

A minimal retrieval function

def retrieve(query: str, k: int = 5) -> list[Chunk]:
    dense = pgvector_search(query_embedding(query), limit=50)
    sparse = fts_search(query, limit=50)
    fused = rrf_fuse(dense, sparse, k=60)  # RRF constant
    reranked = cross_encoder_rerank(query, fused[:30])
    return reranked[:k]

Four steps. Every one of them earns its place in an ablation.

Evaluating retrieval quality without cheating

The single most useful thing I do on a sovereign POC is build a labeled eval set in week one, before writing much pipeline code. About 100 to 200 questions written by a real subject-matter expert, each tagged with the source chunks that should answer it.

Then I track three numbers weekly:

• Retrieval recall@5: does the correct chunk appear in the top 5? Target 90%+.
• Answer faithfulness: does the generated answer cite only retrieved chunks and not hallucinate? I use a stronger local model as judge, spot-checked by hand.
• Refusal rate on out-of-corpus questions: when I ask something the corpus cannot answer, does it correctly say "I don't know"? Target 95%+.

If any of these is under target, changing the LLM will not save you. Fix retrieval, fix chunking, fix the prompt guardrails. In that order.

The refusal metric is the one that gets a POC through legal review. A system that confidently makes things up on questions outside its documents is not a system a regulated team will approve, no matter how good it looks in a demo.

Compliance-friendly architecture: what to draw on the whiteboard

For a sovereign AI POC in a regulated shop, the architecture diagram is a compliance artifact. Draw it before you write code.

The version I keep coming back to:

[Source systems]
      |
      v
[Ingestion worker] --> [Object storage: raw docs, encrypted at rest]
      |
      v
[Parser + chunker] --> [Postgres: chunks + metadata + pgvector index]
      |
      v
[Embedding worker (local model)]
      |
      v
[Retrieval API] <---> [Reranker service] <---> [LLM inference (vLLM)]
      |
      v
[Application / chat UI]
      |
      v
[Audit log: prompt, retrieved chunks, response, model hash, user, timestamp]

Everything inside one VPC or one physical rack. No external calls. The audit log is not optional. Every generation gets logged with the exact prompt, the retrieved chunk IDs, the model version hash, the user identity, and the response. That log is what turns "cool AI demo" into "auditable system a regulator can inspect".

A few architecture rules I enforce:

• Separate the retrieval API from the LLM service. Different scaling profiles, different failure modes, and it lets you swap either one without touching the other.
• Make the LLM stateless. All context, including conversation history, comes in the request. This is what makes horizontal scaling and audit sane.
• PII handling at ingestion, not at query time. If you have PII you do not want in the index, redact or tokenize during parsing, not by prompting the LLM to be careful.
• Backups include the vector index. Rebuilding embeddings for a large corpus is slow and non-trivially expensive. Back up the pgvector table.

The real trade-offs nobody tells you about

Sovereign AI is a set of choices, and every choice costs you something. Here are the ones I make on purpose, and the reasons.

Quality vs sovereignty. A local 70B model in 2026 sits somewhere behind the frontier hosted models on hard reasoning. For RAG-style question answering over well-structured documents, the gap is small. For open-ended reasoning, agentic planning, or complex code generation, the gap is real. If the use case genuinely needs frontier reasoning, be honest with the customer: either the scope changes, or the sovereignty requirement bends (e.g., a sovereign frontier model provider in the same jurisdiction).

Latency vs throughput. A local model on a single GPU serving one user at a time is fast. That same model serving 50 concurrent users, under vLLM with batching, has higher throughput but per-request latency doubles or triples. Plan for the concurrency you actually need. Most internal enterprise tools have peak concurrency in the single digits, not the hundreds.

Cost. A single L40S or A6000-class GPU server, on-prem or in a sovereign cloud, runs roughly $1,500 to $3,000 per month all-in for a 32B-class deployment. That is more than a hosted API for low volume, and dramatically less for high volume. The crossover is usually around 5 to 15 million tokens per day depending on the model.

Operational load. You now own model updates, GPU driver stability, quantization regressions, and the occasional cursed CUDA error. Budget a fractional platform engineer, or plan for a managed self-hosted setup where a partner handles the infrastructure layer.

What I'd do for a sovereign AI POC in 2026

If I were starting a sovereign AI POC on Monday, this is the sequence:

1. Week 1: define the use case narrowly (one workflow, one document type, one user role). Build the labeled eval set. Draw the compliance architecture diagram and get sign-off.
2. Week 2: stand up Postgres + pgvector, ingest the corpus, embed with bge-m3, get hybrid search working. Measure recall@5.
3. Week 3: add the reranker, wire up vLLM with a 32B local model, build the generation prompt with strict citation and refusal rules. Measure the three eval metrics.
4. Week 4: build the audit logging, the minimal UI, and run 10 real users through it. Collect their failure cases and add them to the eval set.

Ship in four weeks. Then decide whether to expand to more document types, more users, or a bigger model, based on what the eval set and the users are actually telling you.

Do not start with a framework choice. Do not start with a vector database bake-off. Start with the eval set and the compliance diagram. Everything else falls out of those two.

---

If you are standing up a sovereign AI proof of concept and want a second pair of eyes on the architecture, or you want someone to build it with you, you can reach me at lazar-milicevic.com/#contact. More field notes from production RAG and agent systems are on the blog.