The vec_fulltext_meta table DDL was referenced by ensure_schema()
but never defined. ADD the missing CREATE_VEC_FULLTEXT_META variable
so the meta table is created alongside vec_body_meta and vec_objects_meta.
schema.py: CREATE_VEC_FULLTEXT_META was referenced at line 328 but
never defined. Adds the missing DDL constant so the v5 schema migration
does not raise NameError on fresh databases.
tests: add embedding test suite with four tests covering vec0 resume
detection, empty-table fallback, force drop, and delete-then-write
ordering verification.
- CREATE VIRTUAL TABLE IF NOT EXISTS runs unconditionally, not gated
on schema version migration. Fixes case where vec0 tables were never
created when sqlite-vec wasn't available during v5 migration.
- Add VEC_EMBEDDING_DIM and _vec_table_ddl() to support runtime
dimension detection instead of hardcoded 1536.
loadable_path() returns a path without .dll extension on Windows,
causing vec0 extension loading to fail. Use sqlite_vec.load(conn)
which handles platform-specific paths correctly.
#27: write_encoded_payload and delete_paper_vectors use sqlite-vec
- builder.py: writes to vec0 tables + companion meta tables
- _chroma.py: delete_paper_vectors deletes from vec0 + meta by paper_id
- Stores body_units_hash, object_units_hash, retrieval_policy_version in meta tables
#28: merge_retrieve and retrieve_chunks use vec0 k-NN search
- search.py: vec0 k-NN queries with companion meta JOINs
- Same dedup/per-paper-cap logic, same result shapes
#29: build_state migrated from JSON file to SQLite build_state table
#30: E2E embed+retrieve test with sqlite-vec + FixedProvider
- 3 tests: body roundtrip, source correctness, per-paper cap
#32: E2E OCR pipeline test with fixture PDFs
Additional changes:
- schema.py: bump to v6, add hash/policy columns to vec companion meta tables
- embed.py: _assert_collections_healthy uses sqlite-vec; resume hash checks read from meta tables
- status.py: chunk counts from SQL COUNT on companion tables
- Removed unused get_collection/embed_paper imports from embed.py
- All tests updated: 76 pass (was 72, +4 new E2E tests, +4 for migration)
- FixedFixedProvider to generate 1536-dim vectors (matching vec0 schema)
- Fixed health check tests for sqlite-vec path
- _split_if_oversized: recursive _halve_text until every part <= 1000 tokens
- _incremental_units_only: also checks retrieval_policy_version match
- _rebuild_paper_units: use RETRIEVAL_POLICY_VERSION constant
- Functional test plan: Layer A (artifact) + Layer B (API), all 8 review
fixes incorporated (persisted artifacts, no jump-assertion, stronger
object/backmatter/abstract tests, proper embed flow)
P0-1: FTS per-paper insert — move SELECT INTO body_units_fts inside the
per-paper loop with WHERE paper_id = ? to avoid re-inserting other papers'
rows during full rebuild. Fix sqlite3.DatabaseError on empty FTS table.
P0-2: object_units real role_index keys — read from 'captions'/'tables'
(what build_role_indexes actually outputs) with fallback to old key names.
P1-1: unit_id collision for mixed body/backmatter — include unit_kind
suffix (:backmatter_body) in unit_id. Also fix duplicate tree node_ids
(block_id reused on different pages) by appending order{emitted_order}.
P1-2: Embed resume body_units_hash — add compute_body_units_hash() to
manifest.py, write hash + retrieval_policy_version to Chroma metadata,
resume compares both before skipping.
- RenderOutput dataclass with heading_events/emitted_block_events tracking
- Stack-algorithm structure tree with own/subtree block IDs
- Recursive body_units builder with role helper + token cap splitting
- Schema v4: section_path_json, section_level, section_title, part_ordinal
- Phase 4/5 reorder in ocr_rebuild.py and ocr.py
- 12 new test cases for tree nesting, own_block_ids, rendered-order intervals
Fix 1 (units.py): object unit_id now includes the entry's block_id
instead of the node's block_span, fixing PRIMARY KEY collisions when
a section has multiple objects (e.g. two figure_captions).
Fix 2 (builder.py): retrieval units are now persisted during
build_from_index(). Scans each paper's OCR output directory for
structure-tree.json, builds body+object units and manifest, then
upserts into the DB tables and meta. Also clears body_units and
object_units at rebuild start alongside the other tables.
- _like_query: split multi-token queries into per-field AND matching
- lookup_paper: same multi-token AND for title lookup
- read-known-paper.md: restructure Step 1 by known-info type (zotero_key,
DOI, author+year, title keyword, fuzzy)
- discover-papers.md Arm 2: note on author+year search pattern
- retrieval-routing.md Arm 2: same author+year search guidance
