Commit graph

5 commits

Author SHA1 Message Date
LLLin000
745c3dd1e1 fix(db): use sqlite_vec.load(conn) instead of loadable_path()
loadable_path() returns a path without .dll extension on Windows,
causing vec0 extension loading to fail. Use sqlite_vec.load(conn)
which handles platform-specific paths correctly.
2026-07-10 02:47:58 +08:00
LLLin000
d24c78c5dc feat(#27 #28 #29 #30 #32): migrate write/read/status/build_state from ChromaDB to sqlite-vec
#27: write_encoded_payload and delete_paper_vectors use sqlite-vec
  - builder.py: writes to vec0 tables + companion meta tables
  - _chroma.py: delete_paper_vectors deletes from vec0 + meta by paper_id
  - Stores body_units_hash, object_units_hash, retrieval_policy_version in meta tables

#28: merge_retrieve and retrieve_chunks use vec0 k-NN search
  - search.py: vec0 k-NN queries with companion meta JOINs
  - Same dedup/per-paper-cap logic, same result shapes

#29: build_state migrated from JSON file to SQLite build_state table

#30: E2E embed+retrieve test with sqlite-vec + FixedProvider
  - 3 tests: body roundtrip, source correctness, per-paper cap

#32: E2E OCR pipeline test with fixture PDFs

Additional changes:
- schema.py: bump to v6, add hash/policy columns to vec companion meta tables
- embed.py: _assert_collections_healthy uses sqlite-vec; resume hash checks read from meta tables
- status.py: chunk counts from SQL COUNT on companion tables
- Removed unused get_collection/embed_paper imports from embed.py
- All tests updated: 76 pass (was 72, +4 new E2E tests, +4 for migration)
- FixedFixedProvider to generate 1536-dim vectors (matching vec0 schema)
- Fixed health check tests for sqlite-vec path
2026-07-09 17:52:45 +08:00
LLLin000
49f84a1aad feat(#31): OpenAICompatibleProvider uses openai SDK with requests fallback
Rewrite embedding provider to use openai.OpenAI client instead of
raw requests.post. Old code preserved in requests_fallback.py,
selectable via VECTOR_DB_PROVIDER_TYPE env/setting.

Changes:
- paperforge/embedding/providers/openai_compatible.py: rewrite to SDK
- paperforge/embedding/providers/requests_fallback.py: NEW old impl
- paperforge/embedding/_config.py: add get_provider_type()
- tests/test_openai_compatible.py: 6 new tests (encode, timeout,
  provider_type switch, no-key error, custom base url)

feat(#33): E2E test fixtures — 3 synthetic PDFs + expected outputs

3 synthetic papers generated with PyMuPDF, each 3 pages:
- paper_a: The Effect of Machine Learning on Clinical Outcomes
- paper_b: A Randomized Trial of Remimazolam vs Propofol (with tables)
- paper_c: Deep Learning for Medical Image Segmentation (with figures)

Changes:
- tests/fixtures/papers/: 3 PDFs
- tests/fixtures/expected_outputs/: expected blocks + body units
- tests/conftest.py: e2e_fixture_dir + synthetic_paper_paths fixtures

feat(#26): sqlite-vec schema — vec0 virtual tables + build_state

Schema version bumped 4→5. Adds:
- 3 vec0 virtual tables: vec_fulltext, vec_body, vec_objects (1536-dim)
- 3 companion metadata tables: vec_*_meta (paper_id, chunk_index, text)
- build_state table (key/value/updated_at)
- ensure_vec_extension() in db.py for extension loading

Changes:
- paperforge/memory/schema.py: schema v5, new tables in ensure_schema()
- paperforge/memory/db.py: ensure_vec_extension() helper
- tests/test_vector_schema.py: 5 new tests (tables, virtual, idempotent, v5)
2026-07-09 17:15:19 +08:00
Research Assistant
2b552be9e6 feat: runtime contract hardening + skill/command truth alignment (Package A+B)
Atomic snapshots, canonical index mutation serialization, sync post-clean truth,
plugin path config-awareness, embed stop signal honesty, full snapshot bootstrap,
pf_ prefix unification, workflow command/lifecycle/path corrections,
mechanical/cognitive route separation with unknown-command guard.
2026-05-16 22:38:43 +08:00
Research Assistant
eb8941b625 feat(memory): add db.py with connection and path resolution 2026-05-12 17:35:19 +08:00