Skip to main content
RAGLight 4 accepts one or more local directories through FolderSource. Ingestion is explicit: it runs when pipeline.ingest(...) is called, not during pipeline construction.

Source options

The default ignore list contains .git, .venv, venv, node_modules, __pycache__, build, and dist. Override it with a tuple of directory names:

Deterministic and resilient processing

Directory entries are processed in deterministic name order. Symbolic links and unsupported file types are skipped. A failure in one file is recorded without discarding files already indexed. IngestionResult exposes:
  • discovered_count, processed_count, and indexed_count;
  • skipped_count, skipped paths, and reasons;
  • failed_count and failures grouped by discovery, processing, or indexing stage;
  • to_dict() for logs, APIs, and job records.
Callers should treat non-empty failures as an incomplete ingestion and surface them to operators. Successfully processed files remain indexed.

Index a checked-out repository

The v4 framework does not clone remote repositories. Clone or synchronize a repository with the tooling used by your application, then pass its local checkout:
This keeps credentials, branch selection, and update policy outside the retrieval framework while preserving recursive codebase ingestion.

Processor selection

The default registry handles text, Markdown, HTML, PDFs, and common programming languages. Files with other extensions are reported as unsupported until a custom processor is registered. See Document processors.