Index in place: an AI architecture that leaves your documents where they are

Most enterprise AI programmes reach the same decision within weeks. The assistant the organisation has licensed works well on the open web and on whatever a user pastes in. It knows nothing about the contracts, policies, case files and reports the organisation actually runs on. Something has to connect the two, and the way that connection is built decides whether the programme gets past its security review.

This paper describes the architecture K-Lake is built on. K-Lake reads documents where they already sit, builds its index inside the customer's own environment, and serves permission-checked passages to the assistant at the moment a question is asked. We set out why we think that design is the right default for a regulated organisation, and what it takes to run.

The hidden commitment in a second copy

The usual way to give an assistant access to company documents is to ingest them. Files are copied into a platform, cut into fragments, embedded and indexed, and the assistant is pointed at the result. It works, and it is quick to demonstrate.

What it creates is a second store of the organisation's most sensitive material, often outside the organisation's own infrastructure. That store needs its own security controls and its own retention rules. It needs a data protection assessment, because it introduces a new processor. It also needs its own permission model, because the permissions on the original files do not travel with the copy. Someone has to rebuild them, and someone has to keep them in step as people join, move and leave.

In our experience the permission problem is the one that ends projects. A copy whose permissions have drifted from the source is a data incident that has not been noticed yet.

Reading in place

K-Lake takes the opposite route. It connects to storage the organisation already runs and reads it without moving, renaming or rewriting anything. Supported sources include Windows file shares over SMB, NFS exports, Amazon S3 and S3-compatible stores, Azure Blob Storage, and SharePoint and OneDrive through Microsoft Graph. Credentials for each source are encrypted before they are stored.

A crawl records each file's path, size, timestamps and content hash, and can capture the permissions the file carries: NTFS and NFSv4 access control lists, POSIX ownership, and object tags. Crawls run on demand, on a schedule or through the API. Nothing is written back to the source.

Extraction on engines you host

To answer questions, the content of each file has to be readable as text. K-Lake extracts text and tables from PDFs and Office documents, runs OCR on scanned pages and images, and transcribes audio and video with timecodes. The extraction engines run inside the customer's environment. An optional GPU engine handles difficult scans, and a heavier engine can retry files that the default engine could not read cleanly.

Extraction is where many architectures quietly send data out, because the convenient engine is a cloud service. In K-Lake, content goes to an external extraction provider only if an administrator configures one, and every such transfer is recorded with the file, the provider and the number of bytes sent.

One index, reached several ways

The result is a single index held inside the customer's cluster. It supports keyword search and meaning-based search together, so a question finds the passage that answers it as well as the passages that share its words. Results open the original file at the right page, or the right moment in a recording, and permissions are checked again when the file is opened.

People and systems reach the index through a web console, a command-line tool, a REST API and a Model Context Protocol (MCP) server. Every route applies the same permission checks, because those checks sit in the database rather than in each interface.

What the assistant actually receives

When an assistant asks K-Lake a question over MCP, it acts as the signed-in user. K-Lake returns the passages relevant to the question, already trimmed to the files that user may open, each with a reference to its source. The assistant never receives the original files.

If the assistant is a cloud service such as Copilot, Claude, ChatGPT or Gemini, those passages are the only content that crosses the boundary, under the customer's own agreement with that vendor. If the customer runs a model inside its own network, nothing crosses at all.

Uses beyond the assistant

An index of the whole estate, built with permissions attached, turns out to be useful before any assistant is connected. Discover shows what the organisation holds, how old it is, who owns it and how much of it is open to everyone. The knowledge graph records the people, organisations and agreements that documents name, and the relationships between them. Smart Actions can redact identifiers from what search and AI return, or notify other systems when files are processed. For a governance team these are reasons to deploy in their own right.

What it takes to run

A proof of concept runs on a single virtual machine with Docker or Podman, installed with one command. We recommend 8 vCPU and 16 GB of memory for evaluation. Production deployments install into the customer's Kubernetes cluster with Helm, use a managed database or the customer's own, and scale crawl and extraction capacity separately as backlogs rise and fall. The proof of concept and production run the same software, so a successful evaluation does not need rebuilding.

Questions to put to any architecture

Whichever product an organisation chooses, five questions separate designs that pass a regulated security review from designs that stall in one.

1. How many copies of our documents will exist once this is running, and who holds each one?
2. Are the permissions on our files enforced as they stand, or rebuilt in a new system?
3. Where does text extraction happen, and what leaves our environment while it runs?
4. What exactly does the model receive when someone asks a question?
5. Can we prove, after the fact, who asked what and which files the answer drew on?

K-Lake was designed so that each answer is short and checkable. A proof of concept on your own infrastructure is the quickest way to test that.