Embed status no longer checks for chromadb import — vec0 runs in
paperforge.db and doesn't need the ChromaDB library. The preflight
check also no longer requires chromadb since vec0 backends do not
depend on it. The chroma_backend module remains for legacy test
compatibility but no longer gates any production build/status path.
Adds _renderSearchState() rendering 8 distinguishable states:
idle, searching (M/@), results, empty, vectors-not-built,
backend-unavailable, timeout, model-changed.
Keyboard: / focuses search, Escape cancels, ArrowUp/Down navigates
results, Enter opens note, Ctrl+Enter forces deep mode.
Accessibility: role=alert on errors, aria-live=polite on results
and progress, role=listbox with aria-selected/aria-posinset on cards,
aria-setsize, tabindex=0 on result cards, focus management on
state transitions.
Focus management: Stop button on build start, first result on
search complete, error banner on failure.
Maps 7 retrieval error codes (VECTOR_NOT_BUILT, VECTOR_CORRUPTED,
MODEL_CHANGED, BACKEND_UNAVAILABLE, TIMEOUT, INTERNAL_ERROR, NO_PYTHON)
to distinct UI states with type, message, recoverable flag, and action.
Adds 7 new test cases for the error classification routing.
Adds all retrieval state labels, button text, warning messages, and
descriptive copy in English and Chinese under the retrieval_* prefix.
Prepares i18n foundation for search domain (8 states) and build domain
(10 states) per PRD #57 and issue #62.
getVectorRuntime now calls embed status --json synchronously with
a 2s time-based cache before falling back to the JSON snapshot.
During build, a 2s polling interval reads embed status --json for
live build state and detects remote stop.
Stop button calls embed stop --json only (no direct .kill() on the
child process), letting the CLI handle cooperative stop.
ISettingPlugin interface extended with _embedPollInterval and
_embedPolling properties.
embed stop now sets status=stopping, waits for the build PID to exit
(8s timeout), force-kills as last resort, then settles to idle.
Build loop checks the cancellation flag (read_vector_build_state)
between papers in the main loop and drains in-flight work before
cleanly exiting without marking 'completed'.
Fixes pre-existing UnboundLocalError from a local import shadowing
the global mark_vector_build_state symbol in the run() function.
embed status now runs a zero-vector k-NN query on each vec0 table
that has companion meta rows. If the probe fails (table dropped,
extension unavailable, index corrupted), healthy=false and
corrupted=true are reported even when meta row counts > 0.
Includes integration test that drops vec_fulltext and verifies
healthy=false.
The vec_fulltext_meta table DDL was referenced by ensure_schema()
but never defined. ADD the missing CREATE_VEC_FULLTEXT_META variable
so the meta table is created alongside vec_body_meta and vec_objects_meta.
- search.py: normalize FTS results to unified field set (score from rank,
add text/heading/source keys). Result key unchanged (already 'matches').
- retrieve.py: rename data.chunks to data.matches in standard and deep paths.
Map field names: paper_id->zotero_key, chunk_text->text, section_path->heading.
Expand enrichment SQL to fetch journal and domain.
--deep flag already wired in CLI parser.
- tests/cli/test_json_contracts.py: add contract tests verifying PFResult
envelope shape, data.matches key presence, unified field names,
--deep flag acceptance, and field type correctness.
- Remove paperforge/plugin/src/services/db.ts (sql.js wrapper, initDatabase/searchMetadata/SearchResultItem)
- Remove paperforge/plugin/sql-wasm.wasm (644KB binary)
- Remove sql.js and @types/sql.js from package.json; npm install to update lockfile
- dashboard.ts: remove import of db.ts, remove _sqlJsInitialized/_sqlJsFailed state,
remove the sql.js search path block entirely — all searches go straight to CLI spawn.
Clean up stale comments.
- Bundle drops from 346KB to 167KB; 6 test files / 58 tests still pass
Replace CLI spawn for metadata search with direct sql.js FTS5 queries.
Input debounced at 200ms. Falls back to CLI when sql.js unavailable.
@ Deep Search stays on CLI.
get_embed_status opens a read-only connection that cannot load the
vec extension when another process has the DB locked. ensure_vec_extension
failure is now non-fatal — the status query proceeds with vec0 meta
table reads, which work in read-only mode.
- Stale detection: instead of hard-exiting with 'Use --force', reset
build_state to idle and continue rebuilding. User no longer gets
stuck when a previous build was killed.
- _pid_alive: avoid UnicodeDecodeError on Windows tasklist output.
Don't pass text=True to subprocess.run; manually decode stdout
with errors='replace'.
dim_detect.detect_embedding_dim() calls the embedding API once to
determine the model's output dimension. ensure_vec_tables() drops
and recreates vec0 tables when the dimension changes, so the schema
always matches the user's model (1536 for ada-002, 2560 for Qwen,
3072 for text-embedding-3-large, etc.).
- CREATE VIRTUAL TABLE IF NOT EXISTS runs unconditionally, not gated
on schema version migration. Fixes case where vec0 tables were never
created when sqlite-vec wasn't available during v5 migration.
- Add VEC_EMBEDDING_DIM and _vec_table_ddl() to support runtime
dimension detection instead of hardcoded 1536.
loadable_path() returns a path without .dll extension on Windows,
causing vec0 extension loading to fail. Use sqlite_vec.load(conn)
which handles platform-specific paths correctly.
backup_render_before_rebuild() copies render/fulltext.md (and
render-map.json, heading-events.json) to versions/v{N}/ before
rebuild overwrites them. Creates/updates versions/manifest.json
with version metadata. Idempotent: skips when no render exists.
Hook placed between phase 3 and phase 4 in _rebuild_one_paper.
6 new tests.
Add structured_content_hash field to OCRMaintenanceRow for display.
- Terminal table: new 'Hash' column showing first 12 chars of hash
- JSON output: structured_content_hash included in to_dict()
- Empty hash displayed as '-'
#27: write_encoded_payload and delete_paper_vectors use sqlite-vec
- builder.py: writes to vec0 tables + companion meta tables
- _chroma.py: delete_paper_vectors deletes from vec0 + meta by paper_id
- Stores body_units_hash, object_units_hash, retrieval_policy_version in meta tables
#28: merge_retrieve and retrieve_chunks use vec0 k-NN search
- search.py: vec0 k-NN queries with companion meta JOINs
- Same dedup/per-paper-cap logic, same result shapes
#29: build_state migrated from JSON file to SQLite build_state table
#30: E2E embed+retrieve test with sqlite-vec + FixedProvider
- 3 tests: body roundtrip, source correctness, per-paper cap
#32: E2E OCR pipeline test with fixture PDFs
Additional changes:
- schema.py: bump to v6, add hash/policy columns to vec companion meta tables
- embed.py: _assert_collections_healthy uses sqlite-vec; resume hash checks read from meta tables
- status.py: chunk counts from SQL COUNT on companion tables
- Removed unused get_collection/embed_paper imports from embed.py
- All tests updated: 76 pass (was 72, +4 new E2E tests, +4 for migration)
- FixedFixedProvider to generate 1536-dim vectors (matching vec0 schema)
- Fixed health check tests for sqlite-vec path
The 256-dim zero vector probe crashes on any collection with real
1536-dim embeddings (text-embedding-3-small). get(limit=1) has no
dimension dependency and still verifies collection accessibility.
test(#17): integration tests for embed pipeline against EphemeralChromaDB
- build_state: write to .tmp first, leave as backup; read falls back to
.tmp if main file corrupt. Extracted default/fallback state helpers.
- integration: 10 tests covering payload prep, encode→write→retrieve
round-trip for all 3 collections, per-paper cap, delete, and
encode_paper_job — all against real ChromaDB EphemeralClient with
mocked provider
- #13: Critical — swap delete/write order in _complete_one (write new
vectors before deleting old ones) to prevent data loss on write failure
- #12: Add logger.warning() to 3 resume-skip except blocks so silent
re-embed fallbacks are observable
- #14: Add lightweight HNSW query probe to _assert_collections_healthy
- #11: Log collection query failures in merge_retrieve instead of silent
continue
- #16: Reuse provider across payloads in encode_paper_job — one creation
per worker per paper instead of per payload
Windows: tmp.replace(path) fails when Obsidian plugin is reading
vector-build-state.json. Fall back to direct write to avoid crashing
the embed build. Stale tmp files cleaned up lazily.
Replace two-phase (prepare all, then encode+write) with sliding-window
pipeline. Maintains bounded in-flight window (max_workers * 4 = 16),
emits EMBED_PROGRESS from the first paper completion (~2s vs ~30s).
Key changes:
- producer/consumer loop: prepare + submit one paper at a time
- wait(FIRST_COMPLETED) in main thread for encode results
- processed_count = skip + embedded, monotonic EMBED_PROGRESS
- resume skip and no-payload paths also advance processed_count
- encode failure fails closed (return 1, no silent skip)
- removed PR9B_BATCH_SIZE and _batched() (replaced by sliding window)
- 4 integration tests for progress counting, failure, and write cycle
Previously Phase 1 processed all papers silently, then Phase 2+3
emitted EMBED_PROGRESS. The plugin's progress bar showed 0/729 for
~30s during preparation, then jumped, looking disconnected.
Now Phase 1 emits EMBED_PROGRESS for each prepared paper, so the
progress bar starts updating immediately. Phase 2+3 continues with
its own counter for actual encode/write progress.
The openai Python client (v2.44.0) has compatibility issues with
SiliconFlow API — calls hang indefinitely even with TCP keepalive.
Replace with plain requests, which is simpler and works reliably.
- Removed openai/httpx/socket dependencies
- Uses requests.post() directly, 60s timeout
- ~0.3s per batch vs 2.3s+ with openai client
- status.py: healthy=True init before if exists: (was UnboundLocalError
when vector DB doesn't exist)
- manifest.py: RETRIEVAL_POLICY_VERSION l4.body.v1 → l4.body.v2 to
trigger object_units rebuild with PR7 fix (unit_id fallback)
- Object_units corrected from 20→311 after rebuild (was INSERT OR
REPLACE overwrite from old empty unit_id)
- Coverage parity: 20 papers have body_units = 20 have object_units ✅
(body=811 chunks, object=311 chunks, expected difference due to
section-split vs 1:1 caption structure)
- Unit tests: 97 pass (subset)
P0 fixes from test report review:
- object unit_id: fallback to f"{obj_type}:p{page}:{block_id}" when
figure_id/table_id missing (prevents DB overwrite on INSERT OR REPLACE)
- object caption_key: page-qualified f"p{page}:{caption_bid}" to match
page-qualified subtree_block_ids in find_owning_node
- object block_map: page-qualified keys for consistency with body_units
- structure node_id: changed from sec:{block_id} to sec:p{page}:{block_id}
for stability (reduces vector ID churn from emitted_order changes)
New tests:
- Section 4b: Object units DB persistence (unit_id unique, non-empty labels,
DB count == list count after upsert)
- Section 5b: Coverage gate (v2_tree / render_map / body_papers counts)
Layer A: 1669/1669 pass (+60 checks from 4b + 5b)
Unit tests: 153 pass, 1 skip
- _body_unit_role_kind now excludes only reference_item/reference_heading
(per policy: everything in fulltext except ref zone)
- backmatter_body unit_kind removed; all units are 'body'
- bounds_map in structure_tree uses page-qualified key (fixes collision
when same block_id appears on different pages)
- Layer A test plan: emitted_ids uses page-qualified keys + handles
block_id=0 (falsy value)
- Layer A: 1609 passed, 0 failed on real vault (22 papers)
- Unit tests: 152 passed, 1 pre-existing skip
- Section/subsection headings now appear in BOTH heading_events and
emitted_block_events (emitted_as='heading') for complete event stream
- Backmatter headings (Funding/Acknowledgments/Data Availability/Conflicts)
now generate heading_events → appear as structure tree sections with
their own body_units
- Page-qualified block IDs (p{page}:{block_id}) in tree own/subtree_block_ids
and body_units block_map: eliminates ambiguous block_id collisions when
same block_id appears on multiple pages in real OCR data
- TC-9.2 wording fix, KEYS empty guard, global unit_id uniqueness check
- _split_if_oversized: recursive _halve_text until every part <= 1000 tokens
- _incremental_units_only: also checks retrieval_policy_version match
- _rebuild_paper_units: use RETRIEVAL_POLICY_VERSION constant
- Functional test plan: Layer A (artifact) + Layer B (API), all 8 review
fixes incorporated (persisted artifacts, no jump-assertion, stronger
object/backmatter/abstract tests, proper embed flow)
P0-1: FTS per-paper insert — move SELECT INTO body_units_fts inside the
per-paper loop with WHERE paper_id = ? to avoid re-inserting other papers'
rows during full rebuild. Fix sqlite3.DatabaseError on empty FTS table.
P0-2: object_units real role_index keys — read from 'captions'/'tables'
(what build_role_indexes actually outputs) with fallback to old key names.
P1-1: unit_id collision for mixed body/backmatter — include unit_kind
suffix (:backmatter_body) in unit_id. Also fix duplicate tree node_ids
(block_id reused on different pages) by appending order{emitted_order}.
P1-2: Embed resume body_units_hash — add compute_body_units_hash() to
manifest.py, write hash + retrieval_policy_version to Chroma metadata,
resume compares both before skipping.
Real OCR output has the same block_id on multiple pages (e.g. running
headers). block_map[str(bid)] = b overwrites correct entries with
wrong-page duplicates. Fix: prefer blocks with non-empty text content
when multiple blocks share the same block_id.
