themisim.build_index

themisim.build_index(data_root, artifacts, checkpoint, *, nlist='auto', device='auto', batch_size=128, num_workers=4, train_target=500000, use_gpu_quantizer=True, streaming=True, use_compile=True, on_progress=None)

Embed every downloaded CDF, concatenate, and build the FAISS index.

Returns the path to index.faiss. All three stages resume cleanly: already-embedded shards are skipped, and concat/index always rebuild from the current shard set.

streaming (default) uses the persistent-DataLoader embed path (embed_inventory_streaming()) — workers are spawned once and stay warm, and (when use_compile and a GPU are present) the encoder is torch.compile-d — which is dramatically faster than the eager per-CDF loop on a large archive. Set streaming=False for the eager loop. This runs the embed in a single process on one device; for a multi-GPU / multi-physical-drive full-archive build, pre-embed with the coordinator (python -m themisim.embed --streaming, which fans out one worker per GPU/drive) and then call this — the embed step will find every shard already on disk and skip straight to concat + index.

Parameters:

• data_root (str | Path)

• artifacts (str | Path)

• checkpoint (str | Path)

• nlist (int | str)

• device (str)

• batch_size (int)

• num_workers (int)

• train_target (int)

• use_gpu_quantizer (bool)

• streaming (bool)

• use_compile (bool)

• on_progress (Callable[[str, object], None] | None)

Return type:

Path