Operations

Install

A hev layer install has two stages. Terraform provisions the required AWS resources: IAM, S3, ECR, networking, cost-read roles, and, for the recommended path, a fresh EKS cluster. Helm installs the gateway, operator, and document cache into that cluster and wires them to the AWS resources Terraform produced.

One command runs both stages: layer install. It is the only install entrypoint.

You can skip Terraform if you already have the AWS resources hev layer needs. At minimum, provide an S3 bucket and gateway IRSA role for snapshots and history. For the full feature set, also provide gateway cost-read IAM, image registry locations, and cluster-level components equivalent to the Terraform outputs.

layer install

Prerequisites: aws (with credentials configured), terraform, helm, and kubectl on your PATH, plus your upstream Turbopuffer API key. Clone the hev/layer repository — it carries the Terraform configuration and the Helm chart the command drives — and run layer install from the checkout:

git clone https://github.com/hev/layer
cd layer
go build -o layer ./apps/layer-cli
export TURBOPUFFER_API_KEY="tpuf_..."
./layer install

AWS credentials come from the ambient environment by default; pass --aws-profile (or set AWS_PROFILE) to select a named credentials profile. Missing required inputs prompt interactively on a TTY. The command prints the resolved plan — profile, region, cluster name, system node shape — and asks for confirmation before provisioning anything. Then it runs both stages in one shot: Terraform provisions the opinionated AWS footprint — VPC, EKS, IAM/IRSA, S3, ECR — and Helm installs the release wired to those outputs, waiting for the gateway rollout before printing next steps.

Flag	Environment	Default
`--profile`	`LAYER_INSTALL_PROFILE`	`demo`
`--aws-profile`	`AWS_PROFILE`	ambient credentials
`--region`	`AWS_REGION`	`us-east-1`
`--cluster-name`	`CLUSTER_NAME`	`layer`
`--namespace`	`NAMESPACE`	`layer`
`--helm-release`	`HELM_RELEASE`	`layer`
`--node-type`	`SYSTEM_NODE_INSTANCE_TYPE`	profile default (`m8g.large`)
`--turbopuffer-api-key`	`TURBOPUFFER_API_KEY`	none (required)
`--version`	`LAYER_VERSION`	`latest`
`--license-token`	`LICENSE_TOKEN`	none
`--dashboard-user`	`DASHBOARD_USER`	`admin`
`--dashboard-password`	`DASHBOARD_PASSWORD`	generated and printed
`--source`	`LAYER_SRC`	walk up from the working directory
`--skip-terraform`	none	off
`--yes`	none	off
`--dry-run`	none	off

Non-interactive installs (CI, scripts) pass every value as a flag or environment variable and must add --yes; without a terminal the command refuses to provision unconfirmed. --dry-run prints the resolved plan and the exact commands it would run, then exits. --skip-terraform reuses existing Terraform outputs and reruns only the cluster components and the Helm release; on that path, the saved cluster name and namespace are authoritative.

Profiles

--profile selects the install footprint. demo is the default.

demo is the lean evaluation footprint. The always-on system node is one arm64 m8g.large with an 80 GiB gp3 data volume. The document cache runs on that system pool, backed by a gp3 PVC (80 Gi) instead of NVMe instance store; no dedicated cache node pool is created. Every Pro feature — gateway, operator, dashboard, extractions, snapshots, keys — runs the same as on the larger footprint. Always-on cost is roughly $40–70/mo on current us-east-1 on-demand pricing.

indexing is the design-partner footprint for installs with heavy indexing duty cycles. It adds a dedicated document-cache Karpenter pool on arm64 i4g.large NVMe instance-store nodes, so the cache gets local flash instead of a gp3 volume. Pass --node-type to override the system node instance type on either profile.

The cache is ephemeral on both profiles (resetOnStart: true); losing the volume or the node is a non-event.

install status

layer install status reports the installed release and workload health: Helm release status, pod readiness, and document-cache node age, type, and utilization — so a forgotten instance-store node shows up instead of billing quietly.

./layer install status

It takes --aws-profile, --region, --cluster-name, --namespace, and --helm-release, with the same environment variables and defaults as layer install.

install uninstall

layer install uninstall removes the Helm release and cluster components and destroys the Terraform footprint. It confirms before destroying (pass --yes to skip) and is idempotent. With --skip-terraform it removes only the in-cluster install and retains the AWS footprint.

./layer install uninstall

The rest of this page is the reference for what the install sets up and the bring-your-own-cluster alternative.

Install shape

An install is one Helm release per environment with one S3 bucket for snapshot and history data. The chart renders a default VectorStore from the credential you provide; an install can define additional VectorStore resources, each with its own upstream credential and inbound auth policy, and route namespaces between them with Index.spec.backend.storeRef. Scoped gateway-only bearer keys are available through the keys inbound auth mode described below.

Terraform

The Terraform configuration in infra/terraform/ provisions the AWS resources that the gateway and operator need. It is opinionated about the resources hev layer needs to behave correctly and conservative about resources around it. Route53 hosted zones and ACM certificates are opt-in; most installs bring existing DNS and TLS.

What it sets up

Resource	Purpose
S3 bucket	Durable storage for namespace snapshots, search history, and clickstream events.
IAM roles + IRSA policies	Gateway S3 and Cost Explorer access, plus worker/operator AWS access.
ECR repositories	Registry space for customer-built function and pipeline worker images. Layer-owned gateway, operator, and dashboard images are pulled from Docker Hub.
EKS + VPC + node pools	Recommended fresh-cluster runtime.
Route53 + ACM	Optional DNS zones, records, and TLS certificates when `manage_public_dns=true`.

Cluster: recommended

Installs should use a fresh EKS cluster unless there is a specific reason to bind hev layer to an existing one. The cluster path provisions:

a VPC with the subnets and endpoints hev layer expects
an EKS control plane and one always-on system node group, defaulting to an arm64 m8g.large with an 80 GiB gp3 data volume
public worker subnets by default, with no NAT Gateway in the fresh cluster path
Karpenter for scale-from-zero worker-cpu and worker-gpu indexing capacity
the AWS EBS CSI add-on and a gp3 StorageClass for the demo-profile document-cache PVC
the AWS Load Balancer Controller for ingress
EFS for shared persistent volumes

If you already operate an EKS cluster, you can disable the cluster modules and point hev layer at the existing cluster. You are still responsible for the functional prerequisites: an S3 bucket for snapshots/history, gateway IRSA that can read/write that bucket and call AWS Cost Explorer for tag-scoped cost reads, Docker Hub pull access for Layer-owned images, registry access for worker images, Karpenter or equivalent node autoscaling for workers, and the AWS Load Balancer Controller if you use public ingress.

Cost notes

The Terraform is designed to deploy a cost-efficient AWS footprint with autoscaling for on-demand indexing work. At rest, the fixed costs are EKS, one small ARM system node, the shared ALB, and small storage lines — roughly $40–70/mo on current us-east-1 on-demand pricing for the demo profile, before variable traffic, object storage, and upstream vector-store usage. The indexing profile adds the dedicated document-cache node while it runs. Indexing bursts scale CPU or GPU worker nodes up through Karpenter and back down when queues drain. If you switch workers to private subnets, enabling NAT adds a standing hourly and egress cost.

Heavier search use cases may need more read-side infrastructure: additional gateway replicas, larger always-on nodes, or a dedicated document-cache pool for steady cache pressure. Contact hev layer for help sizing read-heavy deployments.

Outputs

Terraform emits the values the Helm chart needs to install: the S3 bucket name, gateway IRSA role ARN, and cluster metadata. Runtime images are pulled from Docker Hub for the Layer-owned gateway, operator, and dashboard containers. Pass these into the Helm values file described below. The Terraform provider tags managed resources with Project=hevlayer; activate that tag as a cost-allocation tag in AWS Billing so the gateway can scope Cost Explorer reads to the Layer stack.

Helm

The Helm chart at infra/helm/layer/ installs the gateway, operator, and document cache into a cluster that already has the AWS resources from Terraform or equivalent resources you manage. layer install applies the chart with the profile overlay (values-demo.yaml or values-indexing.yaml) plus a generated values file; a bring-your-own-cluster install runs Helm directly with the same values.

Local gateway development

docker compose starts the gateway’s local dependencies, not a replacement control plane. The gateway still resolves VectorStore and Index resources from Kubernetes at startup, so a compose-based gateway run needs a current kube context with Layer CRDs installed and the matching VectorStore, Index, and Secret objects applied. For a local search backend running on the compose network, use a local-only VectorStore.endpoint.url that the gateway container can reach; cluster Service DNS names only work inside Kubernetes.

Required values

Most of the chart is opinionated defaults. In a typical install the credential you bring from outside the cluster becomes the default VectorStore credential.

Value	Required	Notes
`vectorStore.credential.apiKey`	yes	Upstream store credential. With the default `deriveFromStore` auth mode, clients also send this as the gateway bearer key.
`vectorStore.endpoint.url`	yes	Upstream store API base URL. Defaults to turbopuffer’s AWS us-east-1 endpoint.
`vectorStore.endpoint.region`	yes	Region label for the rendered `VectorStore`.
`vectorStore.inboundAuth.mode`	no	`deriveFromStore`, `keys`, or `open`. Defaults to `deriveFromStore`.
`vectorStore.inboundAuth.keys`	for `keys` mode	Gateway-only bearer keys with `read`, `write`, and `admin` scopes.
`search.enabled`	no	Installs the Layer-operated `search` backend in-cluster and, when `vectorStore.kind=search` and `vectorStore.endpoint.url` is blank, points the default `VectorStore` at that Service.
`search.image`	for `search.enabled`	Container image for the `search` backend, distributed separately from the Layer repo images. The chart rejects `ghcr.io/hev/*`.
`gateway.image`	yes	Gateway image URL. Customer installs pull the pro image from Docker Hub, `hevlayer/layer-gateway-pro:<version>`.
`operator.image`	for `operator.enabled`	Operator image URL. Customer installs pull `hevlayer/layer-operator:<version>` from Docker Hub.
`dashboard.image`	for `dashboard.enabled`	Dashboard image URL. Customer installs pull `hevlayer/layer-dashboard:<version>` from Docker Hub.
`license.token`	trial/commercial installs	Signed hev layer license key from the trial or commercial license email. The chart writes it to a Kubernetes Secret and surfaces it to the gateway.
`license.existingSecret` / `license.secretKey`	optional	Existing Secret name and key containing the license key, for clusters where secret material is managed outside Helm.
`s3.bucket`	yes	S3 bucket Terraform created for snapshots and history.
`serviceAccount.roleArn`	yes	IRSA role ARN that grants the gateway access to the S3 bucket and Cost Explorer.
`dashboard.serviceAccount.name`	yes	Dashboard ServiceAccount name from Terraform output `layer_dashboard_service_account_name`.
`dashboard.serviceAccount.roleArn`	yes	Dashboard IRSA role ARN from Terraform output `layer_dashboard_role_arn`; this renders the EKS role annotation on the dashboard ServiceAccount at first boot.
`documentCache.storage.pvc.enabled`	no	Back the document cache with a PVC (`size`, `storageClass`) instead of node-local instance store. The demo profile sets an 80 Gi `gp3` PVC.
`documentCache.nodeRole`	no	`system` schedules the cache on the always-on system pool (demo profile); `document-cache` targets a dedicated pool (indexing profile).
`documentCache.karpenter.enabled`	no	Provision a dedicated document-cache Karpenter pool. The indexing profile enables it with `instanceType: i4g.large`.
`gateway.indexNamespace`	no	Namespace containing `Index` CRs. Blank follows `operator.discovery.indexNamespace`, then the Helm release namespace.
`gateway.indexConfig.enabled`	no	Enables gateway reads of `Index` CR routing and policy such as `spec.backend.storeRef`, `spec.snapshot.facetFields`, and `spec.scan.threads`.
`gateway.indexGc.enabled`	no	Enables namespace hard-delete cleanup of operator-discovered `Index` CRs.
`gateway.consistency.stablePollIntervalMs`	no	Slow polling cadence for namespaces last observed stable. Defaults to `60000`; cold and updating namespaces keep the fast gateway default.
`gateway.cost.tagKey` / `gateway.cost.tagValue`	no	Cost-allocation tag filter for AWS Cost Explorer. Defaults to `Project=hevlayer`.
`ingress.host`	optional	Set when you want a public ingress; use your DNS/TLS or enable Terraform-managed Route53/ACM.

Image Coordinates

The customer pull path for the Layer-owned runtime images is Docker Hub:

gateway:
  image: hevlayer/layer-gateway-pro:<version>

operator:
  enabled: true
  image: hevlayer/layer-operator:<version>

dashboard:
  enabled: true
  image: hevlayer/layer-dashboard:<version>

These pro images are public to pull, but licensed surfaces are only useful with a valid hevlayer license key installed through license.token or license.existingSecret. Without a valid key, the gateway, operator, and dashboard project the install to the license floor described in Licensing.

The in-cluster search backend image is distributed separately from the Layer repo images. Demo worker images are not part of the first Docker Hub release lane; build customer Function and Pipeline images into the registry your cluster already pulls from.

Layer-Operated Search

Set search.enabled=true to run the search backend beside the gateway. The chart uses the existing service account/IRSA, stores backend data under s3://<s3.bucket>/search by default, mounts a node-local object cache, adds Prometheus scrape annotations for the bundled vmsingle, and restricts backend ingress to Layer’s own components (gateway, operator, and metrics scraper).

The in-chart backend takes no credential of its own and is reachable only through that internal connection. For a default kind: search VectorStore, set vectorStore.inboundAuth.mode to keys or open; deriveFromStore is only valid when the upstream store has its own credential.

Gateway auth modes

The default deriveFromStore mode is the single-tenant BYOC path:

vectorStore:
  credential:
    apiKey: tpuf_...
  inboundAuth:
    mode: deriveFromStore

For an install that needs a gateway-only bearer, use keys mode. The chart renders apiKey values into the release Secret and references them from the VectorStore; omit apiKey when pointing at a pre-created Secret.

vectorStore:
  credential:
    apiKey: tpuf_...
  inboundAuth:
    mode: keys
    workerSecretKey: layer-inbound-worker-api-key
    keys:
      - name: worker
        scopes: [read, write, admin]
        apiKey: layer_worker_...
        secretRef:
          key: layer-inbound-worker-api-key

In keys mode, operator workers, KEDA, and the dashboard use workerSecretName / workerSecretKey as their gateway bearer. Blank workerSecretName uses the release Secret; blank workerSecretKey uses layer-inbound-worker-api-key.

Run the install

On a bring-your-own cluster, run Helm directly with a profile overlay and your values:

helm upgrade --install layer ./infra/helm/layer \
  --namespace layer --create-namespace \
  -f ./infra/helm/layer/values-demo.yaml \
  -f values.customer.yaml

The chart is not published to a public Helm repository — install from the source path or from the chart artifact provided during onboarding.

What gets installed

layer-gateway — Rust gateway for turbopuffer-compatible routes, fetch, scans, snapshots, warm jobs, and pipeline state.
layer-operator — reconciler for VectorStore, Index, InfraRules, Pipeline, and Function CRDs documented in Kubernetes.
layer-document-cache — Aerospike-backed document cache, scale-to-zero by default. On the demo profile it runs on the system pool over a gp3 PVC; on the indexing profile it gets a dedicated instance-store node pool.
Optional Karpenter NodePool / EC2NodeClass resources for worker-cpu and worker-gpu indexing capacity when workerKarpenter.enabled=true, and the dedicated document-cache pool when the indexing profile (or documentCache.karpenter.enabled=true) is in effect.
Supporting resources: service accounts, IRSA bindings, ingress, and CRDs.

Built-in compute pools

The operator supplies built-in compute pools when no cluster-scoped InfraRules/default object exists. Helm does not create that object. If a workload omits scaling.pool, the operator maps worker.computeClass: cpu or gpu to the stock cpu or gpu pool.

The default compute pools are:

Pool	Use
`cpu`	General CPU workers such as extraction, ingestion, and lightweight Functions.
`cpu-large`	CPU workers that need local ephemeral-storage headroom for per-pod source caches.
`gpu`	One-NVIDIA-GPU workers for embedding and model inference.

The built-in pools select layer.hev.dev/node-role=worker-cpu or worker-gpu, matching the chart’s workerKarpenter NodePools. Override them by applying InfraRules/default when you need to tune resource requests, limits, node selectors, tolerations, GPU SKU hints, or per-workload replica ceilings for your cluster.

See InfraRules CRD for the full field shape.