Roadmap

What v0.1.0 ships, where it is not generic today, and the two directions in which the next iterations can extend it. Both directions are independent and can land in either order.

Where v0.1.0 stands

Roughly 60 % of the code is content-agnostic. The remaining 40 % (chunker, schema, ingest, CLIs) is Pascal/mORMot-specific by design — v0.1.0 is the “RAG on mORMot 2 source” deliverable.

TODO 1 — Multi-store backend (feat/sqlite-backend)

Status: branch open; IVectorBackend interface extracted in commit ae42cff. TPgvectorBackend ships as the reference implementation. 2 665 / 2 665 unit tests pass through the interface boundary.

What is left to deliver an SQLite target:

1. Choose the SQLite extension. Two viable options, both modern:
2. sqlite-vec (Alex Garcia, Apache-2.0). Vector-only, no FTS — combine with SQLite's built-in FTS5 for the hybrid score.
3. sqlite-vss (older sibling, wraps Faiss). Less actively maintained as of late 2025.
4. Implement TSqliteVecBackend: IVectorBackend in src/mormot.db.sqlite.vec.pas:
5. SchemaSql → CREATE VIRTUAL TABLE ... USING vec0(...) for the vector half + CREATE VIRTUAL TABLE ... USING fts5(...) for the text half.
6. IngestInsertSql → two INSERT statements (vec0 + fts5) wrapped in a transaction at the orchestrator level, OR a single materialized mormot_chunks table joined to the two virtual tables on rowid.
7. HybridSelectSql → vec_distance_cosine(...) + FTS5 bm25(...) instead of <=> + ts_rank_cd. The α blend stays identical.
8. BindEmbedding → raw float32 little-endian BLOB (no header, no cast required — sqlite-vec's wire format is the simplest possible).
9. Tests: parallel-structure regression suite gated on MORMOT2_PGVECTOR_SQLITE_DB (or similar). Reuse the existing live-test fixtures via the orchestrator, not the backend.
10. Docs: add a “Choosing a backend” page to docs/configuration.md summarizing trade-offs (PG: bigger, faster HNSW, requires server; SQLite: single-file, embedded, simpler ops).

Effort estimate: 6 – 8 hours (mechanical port; the IVectorBackend contract is fully specified).

TODO 2 — Multi-source chunker (feat/generic-chunker)

Status: not started. The current chunker is one Pascal-shaped implementation with no interface around it.

What it would take to ingest non-Pascal sources (Markdown, plain text, PDF, JSONL, …):

1. Extract IChunker in src/mormot.ai.chunk.backend.pas:

   IChunker = interface
    /// chunk a single source (file path, raw text, stream)
    function Chunk(const Source: RawUtf8): TChunkDynArray;
    /// declare which paths/extensions this chunker recognizes
    function Handles(const Path: RawUtf8): boolean;
    /// short backend name for diagnostics
    function Name: RawUtf8;
   end;
   

2. Generic chunk record to replace the Pascal-shaped TPascalChunk:

   TGenericChunk = record
    SourceId: RawUtf8;     // file path or URL or doc id
    SectionId: RawUtf8;    // page #, anchor, line range, ...
    Content: RawUtf8;
    Tokens: integer;
    Metadata: variant;     // free-form JSON for backend-specific fields
   end;
   

3. Generic schema vector_chunks(id BIGSERIAL, source_id TEXT, section_id TEXT, content TEXT, embedding VECTOR(N), ts TSVECTOR, metadata JSONB, created_at TIMESTAMPTZ). The Pascal-shaped fields (unit_name, family, symbol, kind, line_start, line_end) land in metadata for the Pascal chunker's output, where the retrieval SQL can still filter on them via metadata->>'family'.
4. Concrete implementations:
5. TPascalChunker (rename of the current chunker; populates metadata with the existing Pascal fields)
6. TMarkdownChunker (split on heading levels, respect token caps, attach the heading path to section_id)
7. TPlainTextChunker (sliding window N tokens, overlap M)
8. TJsonlChunker (one line = one chunk, metadata from the JSON fields)
9. Optional later: TPdfChunker (via mORMot's PDF reader or an external binary), TTreeSitterChunker (multi-language code).
10. Orchestrator changes:
11. Ingest(SrcRoot, Chunker) — chunker decides what to walk and how.
12. The bulk-insert path becomes generic too.
13. CLI changes:
14. 01_ingest_mormot2 becomes 01_ingest --chunker pascal --src ... with pascal as the default for back-compat.
15. 02_query_cli and 03_chat_cli no longer hard-code mORMot wording in prompts; the RAG prompt template moves to a flag / env var.

Effort estimate: ~10 hours including tests for two chunkers (Pascal + Markdown) plus the schema migration.

Other ideas (post-v0.2)

• Incremental re-embedding driven by git diff. Only re-embed files that changed since the last ingest; mark old chunks deleted in a soft-delete column.
• Sparse retrieval (BM25-only or learned-sparse like SPLADE) to complement the dense vector half.
• Reranker pass (cross-encoder via Ollama) after the top-K hybrid retrieval, for a higher-quality top-3.
• Tree-sitter chunkers for non-Pascal source code (Go, Rust, Python, TS) via the tree-sitter C library through mORMot's FFI.
• Per-tenant isolation (one schema per tenant or a tenant_id filter on every query) for multi-customer deployments.
• Vector quantization (PQ / SQ via pgvector 0.8+ or sqlite-vec) to shrink the index footprint at large corpus sizes.

How to follow along

• feat/sqlite-backend branch — the current iteration on TODO 1.
• A feat/generic-chunker branch will open when TODO 2 starts.
• Each branch keeps main green: the TPgvectorBackend and Pascal chunker remain the v0.1.0 reference until a v0.2.0 cut.