Embed status no longer checks for chromadb import — vec0 runs in
paperforge.db and doesn't need the ChromaDB library. The preflight
check also no longer requires chromadb since vec0 backends do not
depend on it. The chroma_backend module remains for legacy test
compatibility but no longer gates any production build/status path.
embed status now runs a zero-vector k-NN query on each vec0 table
that has companion meta rows. If the probe fails (table dropped,
extension unavailable, index corrupted), healthy=false and
corrupted=true are reported even when meta row counts > 0.
Includes integration test that drops vec_fulltext and verifies
healthy=false.
get_embed_status opens a read-only connection that cannot load the
vec extension when another process has the DB locked. ensure_vec_extension
failure is now non-fatal — the status query proceeds with vec0 meta
table reads, which work in read-only mode.
dim_detect.detect_embedding_dim() calls the embedding API once to
determine the model's output dimension. ensure_vec_tables() drops
and recreates vec0 tables when the dimension changes, so the schema
always matches the user's model (1536 for ada-002, 2560 for Qwen,
3072 for text-embedding-3-large, etc.).
#27: write_encoded_payload and delete_paper_vectors use sqlite-vec
- builder.py: writes to vec0 tables + companion meta tables
- _chroma.py: delete_paper_vectors deletes from vec0 + meta by paper_id
- Stores body_units_hash, object_units_hash, retrieval_policy_version in meta tables
#28: merge_retrieve and retrieve_chunks use vec0 k-NN search
- search.py: vec0 k-NN queries with companion meta JOINs
- Same dedup/per-paper-cap logic, same result shapes
#29: build_state migrated from JSON file to SQLite build_state table
#30: E2E embed+retrieve test with sqlite-vec + FixedProvider
- 3 tests: body roundtrip, source correctness, per-paper cap
#32: E2E OCR pipeline test with fixture PDFs
Additional changes:
- schema.py: bump to v6, add hash/policy columns to vec companion meta tables
- embed.py: _assert_collections_healthy uses sqlite-vec; resume hash checks read from meta tables
- status.py: chunk counts from SQL COUNT on companion tables
- Removed unused get_collection/embed_paper imports from embed.py
- All tests updated: 76 pass (was 72, +4 new E2E tests, +4 for migration)
- FixedFixedProvider to generate 1536-dim vectors (matching vec0 schema)
- Fixed health check tests for sqlite-vec path
test(#17): integration tests for embed pipeline against EphemeralChromaDB
- build_state: write to .tmp first, leave as backup; read falls back to
.tmp if main file corrupt. Extracted default/fallback state helpers.
- integration: 10 tests covering payload prep, encode→write→retrieve
round-trip for all 3 collections, per-paper cap, delete, and
encode_paper_job — all against real ChromaDB EphemeralClient with
mocked provider
- #13: Critical — swap delete/write order in _complete_one (write new
vectors before deleting old ones) to prevent data loss on write failure
- #12: Add logger.warning() to 3 resume-skip except blocks so silent
re-embed fallbacks are observable
- #14: Add lightweight HNSW query probe to _assert_collections_healthy
- #11: Log collection query failures in merge_retrieve instead of silent
continue
- #16: Reuse provider across payloads in encode_paper_job — one creation
per worker per paper instead of per payload
Windows: tmp.replace(path) fails when Obsidian plugin is reading
vector-build-state.json. Fall back to direct write to avoid crashing
the embed build. Stale tmp files cleaned up lazily.
The openai Python client (v2.44.0) has compatibility issues with
SiliconFlow API — calls hang indefinitely even with TCP keepalive.
Replace with plain requests, which is simpler and works reliably.
- Removed openai/httpx/socket dependencies
- Uses requests.post() directly, 60s timeout
- ~0.3s per batch vs 2.3s+ with openai client
- status.py: healthy=True init before if exists: (was UnboundLocalError
when vector DB doesn't exist)
- manifest.py: RETRIEVAL_POLICY_VERSION l4.body.v1 → l4.body.v2 to
trigger object_units rebuild with PR7 fix (unit_id fallback)
- Object_units corrected from 20→311 after rebuild (was INSERT OR
REPLACE overwrite from old empty unit_id)
- Coverage parity: 20 papers have body_units = 20 have object_units ✅
(body=811 chunks, object=311 chunks, expected difference due to
section-split vs 1:1 caption structure)
- Unit tests: 97 pass (subset)
P0-1: FTS per-paper insert — move SELECT INTO body_units_fts inside the
per-paper loop with WHERE paper_id = ? to avoid re-inserting other papers'
rows during full rebuild. Fix sqlite3.DatabaseError on empty FTS table.
P0-2: object_units real role_index keys — read from 'captions'/'tables'
(what build_role_indexes actually outputs) with fallback to old key names.
P1-1: unit_id collision for mixed body/backmatter — include unit_kind
suffix (:backmatter_body) in unit_id. Also fix duplicate tree node_ids
(block_id reused on different pages) by appending order{emitted_order}.
P1-2: Embed resume body_units_hash — add compute_body_units_hash() to
manifest.py, write hash + retrieval_policy_version to Chroma metadata,
resume compares both before skipping.
- get_collection(name=) supports paperforge_fulltext or paperforge_body
- delete_paper_vectors deletes from both collections
- embed_body_units + get_body_units_for_embedding (reads from memory DB)
- Status reports chunk_count, body_chunk_count, total_chunks
- Embed build routes to body_units path when DB has body_units
- merge_retrieve queries both collections with unit-level dedup
