Getting started

Introduction

What K-Lake is, and the handful of concepts everything else builds on.

K-Lake is a Kubernetes-native indexer. It catalogues the files in your object stores and filesystems so they can be searched, audited and reasoned about as one inventory — and so an AI assistant can answer from them with a citation for every fact.

It runs as a small set of services in your cluster. It pulls metadata from the sources you configure, stores that metadata in a managed database inside your boundary, and presents it through a web console, a command-line tool and a REST API. Your documents stay where they are. Nothing is migrated, copied or modified.

What K-Lake indexes

A source is one location K-Lake is told to track:

Each source carries its own credentials, encrypted at rest, its own enable/disable state and its own metadata-enrichment policy.

Core concepts

Tenant. Every source, user and indexed file belongs to a tenant. Tenants are fully isolated — a user in one tenant cannot see another tenant's sources or files.

Crawl. A crawl walks a source from a root prefix and records what it finds. Crawls run on demand or are triggered through the API.

File record. For each file discovered, K-Lake stores the path, size, timestamps, content hash where available, and optional protocol-specific metadata — S3 tags, Azure blob tags and content-type, NTFS ACLs, NFSv4 ACLs, extended attributes.

Metadata caps. Each source declares which enrichments it wants. Cheap fields are recorded inline during the crawl; heavier fields are enriched in the background. Caps can be changed per source at any time.

Content extraction. Beyond metadata, K-Lake reads each file's content so it becomes searchable by keyword and by meaning: text and structure from PDFs and Office documents, including tables and headings; OCR from images and scanned pages; and transcription of audio and video — the words spoken and the text shown on screen, with timecodes, so a search hit deep-links to the exact moment. All extraction is self-hosted. No document content leaves your environment.

Token. Calls to the API and CLI authenticate with a personal access token (PAT) or an OIDC bearer token. PATs are minted from the console or API and belong to the user who created them.

What you'll see running

• API — serves the console and answers REST requests
• Web console — the operator interface
• Workers — stateless pods that crawl sources and write metadata
• Database — the metadata store and the work queue

Sources, users and tokens are managed through any of the three interfaces — console, CLI or API. All three act on the same data.

Quick start

From a running deployment to your first verifiable answer.

This assumes an administrator has deployed K-Lake and given you two things: the URL of your instance, for example https://zegress.example.com, and a personal access token. If you are the administrator and still need to deploy, use the production deployment guide, or stand up an evaluation stack on a single VM with the proof of concept below.

You'll use one or more of three interchangeable interfaces: the web console, the zegress-control CLI, and the REST API. They act on the same data, so pick whichever suits. Every action is scoped to your tenant and audited.

1. Sign in

Open your K-Lake URL in a browser. Sign in with your PAT, or use the SSO button if your tenant is configured for single sign-on. You land on the dashboard, which shows queue depth and a per-source summary.

To use the CLI or API, set these once in your shell:

export ZEGRESS_URL=https://zegress.example.com
export ZEGRESS_TOKEN=zgpat_<your-token>

2. Add a source

A source is one location K-Lake tracks — an S3 bucket, an Azure Blob container, an SMB/CIFS share, an NFS export, or a SharePoint or OneDrive drive. The quickest start is a small bucket or container.

Guided. zegress-control --api-url "$ZEGRESS_URL" --api-token "$ZEGRESS_TOKEN" init walks you through this whole section — connect a source, enable extraction, start the first crawl — in one interactive flow. The manual steps below are the same thing, broken out.

Console. Go to Sources → Add source, choose s3, give it a stable ID such as s3://docs-bucket, enter the bucket, region and credentials, then Save.

CLI — S3:

printf '%s' "$AWS_SECRET_ACCESS_KEY" | zegress-control \
  --api-url "$ZEGRESS_URL" --api-token "$ZEGRESS_TOKEN" \
  source add-s3 \
  --source-id 's3://docs-bucket' \
  --bucket docs-bucket --region us-east-1 \
  --access-key-id AKIA... --secret-access-key-stdin

CLI — SMB/CIFS. Two backends: smb runs in userspace and needs no special privileges; smbfs uses a kernel CIFS mount and adds NTFS ACLs, SMB3 multichannel and backup-operator intent.

# userspace backend
printf '%s' "$SMB_PASSWORD" | zegress-control \
  --api-url "$ZEGRESS_URL" --api-token "$ZEGRESS_TOKEN" \
  source add-smb \
  --source-id 'smb://fileserver/share' \
  --server fileserver --share share --username svc-zegress

# kernel-mount backend (NTFS ACLs, etc.)
printf '%s' "$SMB_PASSWORD" | zegress-control \
  --api-url "$ZEGRESS_URL" --api-token "$ZEGRESS_TOKEN" \
  source add-smbfs \
  --source-id 'smbfs://fileserver/share' \
  --server fileserver --share share --username svc-zegress --backup-intent

Secrets and passwords are read from stdin so they never reach your shell history, and credentials are encrypted at rest. See Sources for NFS and for narrowing a source to a subtree.

3. Tune extraction — optional

Extraction reads each file's content so it becomes searchable. It's already on from step 2, so you can skip straight to the crawl. Use this step only to restrict extraction by file type, or to enable it if you added the source with --no-extract or through the raw API.

Console. On the source, open the Extraction tab, toggle it on and optionally set an extension allowlist such as pdf, docx.

zegress-control --api-url "$ZEGRESS_URL" --api-token "$ZEGRESS_TOKEN" \
  source extract enable --source-id 's3://docs-bucket' --extensions pdf,docx

4. Crawl the source

A crawl walks the source, records every file it finds, and — with extraction enabled — queues each eligible file for content extraction.

Console. Open the source and select Crawl.

zegress-control --api-url "$ZEGRESS_URL" --api-token "$ZEGRESS_TOKEN" \
  crawl --source-id 's3://docs-bucket'

curl -X POST -H "Authorization: Bearer $ZEGRESS_TOKEN" \
  "$ZEGRESS_URL/api/sources/s3%3A%2F%2Fdocs-bucket/crawl"

The source page shows live file and byte counts as the crawl runs, and extracted content becomes searchable as each file finishes. Source IDs are URL-encoded in API paths: s3://docs-bucket becomes s3%3A%2F%2Fdocs-bucket.

zegress-control --api-url "$ZEGRESS_URL" --api-token "$ZEGRESS_TOKEN" \
  crawl --source-id 's3://docs-bucket' --force-reextract

5. Search your content

Console. Open Content search, pick the source and enter a query. Each result shows the file, an in-document locator — page or timestamp — and a highlighted snippet.

curl -H "Authorization: Bearer $ZEGRESS_TOKEN" \
  "$ZEGRESS_URL/api/sources/s3%3A%2F%2Fdocs-bucket/content/search?q=invoice"

Query syntax is familiar: bare words are AND, OR gives alternatives, "quoted phrases" match exactly, and -exclude removes terms. Lead with the most distinctive term.

6. Open the original to verify

Every result traces back to its source. In the console, each search hit and each file-detail page carries an Open original link and a Download option; the API returns the same as a signed url on each hit. Opening it re-fetches the original file on demand, so you can confirm exactly where an answer came from.

Access is enforced at every step: the link is short-lived, re-checks your permissions at the moment you open it, and is audited. You only ever see files you are authorised to see.

7. Connect an AI assistant — optional

K-Lake can serve your content to an AI assistant over the Model Context Protocol, so the assistant answers only from your knowledge base and cites a source for every fact — under the same per-user security and auditing.

Ask your administrator to enable MCP on your instance, then point your client at it. See MCP clients for client-by-client setup and MCP skills for the available tools.

Proof of concept

The whole stack on one VM, in a single command.

Stand up the full K-Lake stack — database, work queue, workers, API, web console and a CPU content extractor — on one Docker or Podman VM, with a demo tenant created and its PAT printed, in a single command. This is the fastest path to an evaluable system. When you're ready for a real cluster, move to the production deployment guide.

export ZEGRESS_LICENCE='ZGLIC1.<the blob Zegress provided>'

Run the single-command evaluation installer included in your evaluation bundle. It provisions a self-contained single-node Kubernetes environment on the VM, applies the evaluation configuration and onboards a demo tenant. When it finishes it prints the console URL, the cluster-admin token and the demo tenant's PAT, and maps the VM's ports 80 and 443 onto the cluster so the console is served at http://<vm-ip>/ with no further setup.

Sizing the VM

The box runs several memory-hungry tenants at once — the database, the content extractor and the Kubernetes control plane.

Four vCPU is not viable. The evaluation stack reserves just over 5 vCPU before anything starts, so on a 4-vCPU box the scheduler never places the last pods and they sit Pending — which looks like a hang rather than a sizing problem. Six fits the stack at rest; eight leaves room for a rolling update to schedule a replacement pod before retiring the old one, which is why eight is the recommendation.

Do not go below 16 GB of RAM. 8 GB looks plausible on paper and gets squeezed the moment the extractor parses a real PDF while the database is ingesting. At 16 GB memory is the tighter budget of the two — the stack reserves close to 13 GB, so it runs out of RAM before it runs out of CPU.

Three things bite specifically on a single VM:

• Disk, not RAM, is the usual first failure. The worker and content-extractor images are multi-gigabyte, and image churn on a small root disk can evict pods. Give it 60–80 GB on SSD.
• The space must be free where container storage lives, not merely on the disk. Stock RHEL and LVM images cap /var at 10 GB and leave the rest of the volume group unallocated, so a "100 GB" VM still fails partway through the image pulls with no space left on device. Check df -h /var before you start and extend the logical volume if it's short.
• The database's shared memory. The default container shared-memory size is too small for evaluation and shows up as crawl stalls rather than an obvious out-of-memory error. The evaluation configuration sizes it appropriately; raise it further if you see stalls under a heavier corpus.

This is evaluation-scale guidance. Large-corpus ingest above ten million rows is a different and much larger sizing conversation, and a GPU extractor changes the picture entirely.

How the evaluation configuration differs from production

The evaluation configuration is a self-contained single-node install, separate from the production Helm deployment. It re-skins the same product for one small node and shares no state with a production cluster.

• One replica of every component, with no autoscaling.
• Database sized for evaluation rather than production throughput.
• Worker at reduced concurrency on the direct write path — the high-throughput path is overkill at this scale.
• Extractor at single-file concurrency with a per-file size ceiling, to bound memory.
• Survives reboots. The stack restarts automatically, so closing a laptop lid or rebooting the VM brings K-Lake back up on its own. No need to re-run the installer.
• MCP is enabled by default and reachable through the console's ingress, so you can connect an AI client immediately. Production leaves MCP opt-in.

Options

The installer accepts flags to bring up infrastructure only and skip tenant onboarding, name the demo tenant (it defaults to demo), skip the content extractor, or tear the whole environment down. Run it with --help for exact flag names.

Private images. During the evaluation period the container images may be private. If so, the installer accepts registry credentials — server, username and read token — and creates an image pull secret for you. Once images are public for open evaluation, no token is needed and they pull anonymously.

After it's up

The installer prints a console URL, a cluster-admin token and the demo tenant's PAT. Point the CLI at the instance to check health and onboard your first source:

export ZEGRESS_URL=http://<vm-ip>
export ZEGRESS_TOKEN=<ADMIN_TOKEN>   # cluster-admin token printed by the installer

# Health check
zegress-control --api-url "$ZEGRESS_URL" --api-token "$ZEGRESS_TOKEN" doctor

# First source and crawl, as the demo tenant (use the demo PAT it printed)
zegress-control --api-url "$ZEGRESS_URL" --api-token "<DEMO_PAT>" init

Then open http://<vm-ip>/ and follow the quick start to get a grounded, verifiable answer.

Troubleshooting

Sizing guide

Starting points for a real deployment — then tune from the metrics.

K-Lake scales horizontally. Workers are stateless: crawling, content extraction and Smart Actions pipelines all run on pools of interchangeable workers that coordinate through a shared work queue and never duplicate effort. Add capacity and throughput goes up. The database is the floor on how fast metadata can be persisted, so size it for your peak ingest rate rather than your steady state.

Automatic scaling

Worker pools scale automatically with the workload. When a backlog builds — a large first crawl, a bulk re-extraction, a burst of pipeline work — capacity is added; when the backlog drains, it's released. Scaling is proportional: the busier the queue, the more workers come online, up to a ceiling you set, and back down to a baseline when things are quiet.

This uses standard Kubernetes autoscaling, so it fits a cluster you already operate, driven by the same queue-depth signals you can see in Telemetry. The figures below are per-pod sizing and the floor and ceiling for each pool; autoscaling moves the replica count between those bounds for you. For most deployments you set sensible limits once and let K-Lake track demand.

The crawl, extraction and Smart Actions pools scale independently, so a heavy extraction backlog doesn't starve crawling, or the other way round.

Choosing a deployment size

K-Lake ships four deployment sizes — Small, Medium, Large and Enterprise — that map one-to-one to the capacity tiers you can license. Each preset provisions sensible baselines and scaling ceilings for every component, so the fast path is simply to deploy the size that matches the plan you bought.

Values shown as baseline → ceiling mean the baseline is always running and the component scales up to the ceiling under load. The extractor fleet's Small baseline is zero — it scales to nothing when there's nothing to extract, so an idle deployment carries no extractor cost.

Database tuning per size. Each size also tunes the database internals — shared buffers, connection limit, work memory, WAL size. Small starts at 1 GB shared buffers and 200 connections; Medium at 8 GB and 600; Large and Enterprise at 16 GB and 800. The preset sets these for you; override them only with a specific reason. Enterprise deployments frequently point at an external, highly-available database instead of the bundled one.

Translating a size to infrastructure

Kubernetes schedules pods by their CPU and memory requests, so requests — not limits — determine how many fit on your nodes. Two groups matter:

• Fixed baseline, always running: the database, the cache and work queue, the API and console replicas, the connection pooler, the Smart Actions pool and the metadata workers. Size your cluster to hold these first.
• Elastic pools, scaling with load: crawl workers at roughly 1.25 vCPU and 0.75 Gi requested each, and the extractor fleet. These grow toward the size's ceiling when there's a backlog and release when it drains.

A rule of thumb: usable vCPU per node is about node vCPU minus one, and crawl-worker pods per node is about usable vCPU divided by 1.25.

Example — three 8-vCPU, 32 GB nodes at Medium. The fixed baseline (database around 4 vCPU, cache 1, two API, two console, two pooler, plus the metadata and Smart Actions pools) requests roughly 6–7 vCPU and 26 GB before any crawling. The remaining ~17 vCPU comfortably runs ten or more crawl workers plus a few extractor pods, inside Medium's ceilings. Fits Medium with headroom.

Example — a single 16-vCPU, 64 GB node at Small. The database alone requests 2 vCPU and 4 GB; baseline services add about 1.5 vCPU. That leaves roughly 12 vCPU for Small's ceilings with room to spare. This is the natural first real node when graduating from a proof of concept. Fits Small.

For Large and Enterprise, plan on multiple nodes — the crawl and extractor pools alone can reach dozens of pods — and at Enterprise scale, an external highly-available database rather than a single bundled instance.

Component defaults

The shipped defaults set conservative requests with headroom in the limits.

Worker

Each worker handles many in-flight crawl tasks concurrently. Increase replicas to increase throughput; workers coordinate through the work queue and will not duplicate work.

Extractor

Content extraction runs as its own scale-out fleet, separate from the workers. It defaults to the highest-throughput engine and scales on queue depth — add replicas to extract more files in parallel, worth roughly a thirtyfold throughput uplift over the previous default on suitable hardware. The high-performance engine is x86-64 only today; ARM hosts fall back to the portable engine, with native ARM performance planned. A GPU-accelerated, high-accuracy vision extractor is available for scanned or image-heavy documents — provision a small GPU pool if you enable it.

If you index audio or video, transcription runs as a separate, optional media-extractor fleet. Budget for it independently, and only when A/V sources are in scope:

• Heavier per pod, one file at a time. Each pod holds a resident transcription model of roughly 1–3 GB and processes one media file at a time. Parallelism comes from replicas, not concurrency per pod — size pods for that memory floor and scale the fleet out.
• Compute-bound; GPU for throughput. Transcription is on the order of minutes of compute per file on CPU. That's fine for modest volumes, but for large or continuously-growing A/V corpora, provision a GPU pool for a substantial throughput gain.
• Video costs more than audio. Video is transcribed and frame-sampled for on-screen text, reusing your OCR and vision backend, so a video minute is more work than an audio minute. Factor that into the fleet ceiling if your corpus is video-heavy.
• Idle-cheap. Like the other extractors it scales on queue depth and back down when there's no media to process, adding no steady-state cost to deployments that only occasionally see A/V.

Embedding service

Hybrid search is powered by a dedicated embedding service that scales independently of both the workers and the extractor. It's only needed when hybrid search is enabled — size it to the ingest rate of your hybrid-enabled sources. Lexical-only deployments don't need it at all.

API, console and database

The API is thin — it reads and writes the database on behalf of the console and CLI. Two replicas is a sensible default for availability; scale up only under heavy programmatic API traffic. The console is static assets served by a lightweight web server, so one or two replicas is enough.

Everything persists in a managed database. Start at 2 CPU cores and scale up under heavy ingest; allow at least 8 Gi of memory for ingests above a few million files; and provision disk for the eventual file count, figuring tens of bytes per file record plus indexes. Put a connection pooler in front of the database if you run more than a handful of workers.

When to scale up

Worker scaling is automatic, so in the common case you don't intervene. Reach for this table when autoscaling has hit the ceiling you set, or for the parts that don't autoscale.

Source-level tuning

Two source-level toggles affect throughput. Both are exposed in the console, the CLI (source bulk-ingest, source meta-caps) and the API.

• Bulk ingest — on by default. Optimises the write path for first-time crawls and large catch-up runs. Leave it on unless you're doing many small incremental updates and have benchmarked the alternative.
• Metadata caps — controls which optional enrichments are gathered: S3 tags, Azure tags and content-type, NTFS and NFSv4 ACLs, extended attributes. More enrichment costs more crawl time and storage. Start narrow and widen as needed.

Estimating headroom

A worker pod can sustain steady-state ingest from a single source at network-bound rates for most object stores and NAS protocols. Real throughput depends heavily on object size distribution — many small files is harder than fewer large ones — on source latency, since same-region S3 is much faster than a remote SMB share, and on whether metadata enrichment is enabled.

Plan capacity using a representative source: run a crawl on it, watch the metrics, and extrapolate.

Where to go next

• Sources — protocol-specific configuration and lifecycle
• Security trimming — per-file access control at query time
• MCP server — grounded AI retrieval over your content
• Telemetry — metrics, logs and alerting, all inside your boundary