2 commits

Author	SHA1	Message	Date
LLLin000	d24c78c5dc	feat(#27 #28 #29 #30 #32): migrate write/read/status/build_state from ChromaDB to sqlite-vec ... #27: write_encoded_payload and delete_paper_vectors use sqlite-vec - builder.py: writes to vec0 tables + companion meta tables - _chroma.py: delete_paper_vectors deletes from vec0 + meta by paper_id - Stores body_units_hash, object_units_hash, retrieval_policy_version in meta tables #28: merge_retrieve and retrieve_chunks use vec0 k-NN search - search.py: vec0 k-NN queries with companion meta JOINs - Same dedup/per-paper-cap logic, same result shapes #29: build_state migrated from JSON file to SQLite build_state table #30: E2E embed+retrieve test with sqlite-vec + FixedProvider - 3 tests: body roundtrip, source correctness, per-paper cap #32: E2E OCR pipeline test with fixture PDFs Additional changes: - schema.py: bump to v6, add hash/policy columns to vec companion meta tables - embed.py: _assert_collections_healthy uses sqlite-vec; resume hash checks read from meta tables - status.py: chunk counts from SQL COUNT on companion tables - Removed unused get_collection/embed_paper imports from embed.py - All tests updated: 76 pass (was 72, +4 new E2E tests, +4 for migration) - FixedFixedProvider to generate 1536-dim vectors (matching vec0 schema) - Fixed health check tests for sqlite-vec path	2026-07-09 17:52:45 +08:00
LLLin000	6ff0f6d38b	PR9B: Embed Parallel Encode ... - Four new dataclasses: EmbeddingPayload, EncodedPayload, PaperEmbeddingJob, PaperEncodedBundle - prepare_legacy/body/object_payload: extract prepare phase from legacy embed functions - encode_payload / encode_paper_job: worker-thread-safe encode functions - write_encoded_payload: ChromaDB serial write - prepare_payloads_for_entry: routing to correct prepare function - Existing embed_body_units/embed_paper/embed_object_units refactored as wrappers - Embed main loop: three-phase pipeline (PREPARE -> ENCODE -> WRITE) with ThreadPoolExecutor - Bounded batch processing (BATCH_SIZE=8, MAX_WORKERS=4) - Encode-success-gated delete: old vectors only removed after encode succeeds - 23 regression tests for new functions	2026-07-06 22:41:25 +08:00