backup_render_before_rebuild() copies render/fulltext.md (and
render-map.json, heading-events.json) to versions/v{N}/ before
rebuild overwrites them. Creates/updates versions/manifest.json
with version metadata. Idempotent: skips when no render exists.
Hook placed between phase 3 and phase 4 in _rebuild_one_paper.
6 new tests.
Add structured_content_hash field to OCRMaintenanceRow for display.
- Terminal table: new 'Hash' column showing first 12 chars of hash
- JSON output: structured_content_hash included in to_dict()
- Empty hash displayed as '-'
#27: write_encoded_payload and delete_paper_vectors use sqlite-vec
- builder.py: writes to vec0 tables + companion meta tables
- _chroma.py: delete_paper_vectors deletes from vec0 + meta by paper_id
- Stores body_units_hash, object_units_hash, retrieval_policy_version in meta tables
#28: merge_retrieve and retrieve_chunks use vec0 k-NN search
- search.py: vec0 k-NN queries with companion meta JOINs
- Same dedup/per-paper-cap logic, same result shapes
#29: build_state migrated from JSON file to SQLite build_state table
#30: E2E embed+retrieve test with sqlite-vec + FixedProvider
- 3 tests: body roundtrip, source correctness, per-paper cap
#32: E2E OCR pipeline test with fixture PDFs
Additional changes:
- schema.py: bump to v6, add hash/policy columns to vec companion meta tables
- embed.py: _assert_collections_healthy uses sqlite-vec; resume hash checks read from meta tables
- status.py: chunk counts from SQL COUNT on companion tables
- Removed unused get_collection/embed_paper imports from embed.py
- All tests updated: 76 pass (was 72, +4 new E2E tests, +4 for migration)
- FixedFixedProvider to generate 1536-dim vectors (matching vec0 schema)
- Fixed health check tests for sqlite-vec path
The 256-dim zero vector probe crashes on any collection with real
1536-dim embeddings (text-embedding-3-small). get(limit=1) has no
dimension dependency and still verifies collection accessibility.
test(#17): integration tests for embed pipeline against EphemeralChromaDB
- build_state: write to .tmp first, leave as backup; read falls back to
.tmp if main file corrupt. Extracted default/fallback state helpers.
- integration: 10 tests covering payload prep, encode→write→retrieve
round-trip for all 3 collections, per-paper cap, delete, and
encode_paper_job — all against real ChromaDB EphemeralClient with
mocked provider
- #13: Critical — swap delete/write order in _complete_one (write new
vectors before deleting old ones) to prevent data loss on write failure
- #12: Add logger.warning() to 3 resume-skip except blocks so silent
re-embed fallbacks are observable
- #14: Add lightweight HNSW query probe to _assert_collections_healthy
- #11: Log collection query failures in merge_retrieve instead of silent
continue
- #16: Reuse provider across payloads in encode_paper_job — one creation
per worker per paper instead of per payload
Windows: tmp.replace(path) fails when Obsidian plugin is reading
vector-build-state.json. Fall back to direct write to avoid crashing
the embed build. Stale tmp files cleaned up lazily.
Replace two-phase (prepare all, then encode+write) with sliding-window
pipeline. Maintains bounded in-flight window (max_workers * 4 = 16),
emits EMBED_PROGRESS from the first paper completion (~2s vs ~30s).
Key changes:
- producer/consumer loop: prepare + submit one paper at a time
- wait(FIRST_COMPLETED) in main thread for encode results
- processed_count = skip + embedded, monotonic EMBED_PROGRESS
- resume skip and no-payload paths also advance processed_count
- encode failure fails closed (return 1, no silent skip)
- removed PR9B_BATCH_SIZE and _batched() (replaced by sliding window)
- 4 integration tests for progress counting, failure, and write cycle
Previously Phase 1 processed all papers silently, then Phase 2+3
emitted EMBED_PROGRESS. The plugin's progress bar showed 0/729 for
~30s during preparation, then jumped, looking disconnected.
Now Phase 1 emits EMBED_PROGRESS for each prepared paper, so the
progress bar starts updating immediately. Phase 2+3 continues with
its own counter for actual encode/write progress.
The openai Python client (v2.44.0) has compatibility issues with
SiliconFlow API — calls hang indefinitely even with TCP keepalive.
Replace with plain requests, which is simpler and works reliably.
- Removed openai/httpx/socket dependencies
- Uses requests.post() directly, 60s timeout
- ~0.3s per batch vs 2.3s+ with openai client
- status.py: healthy=True init before if exists: (was UnboundLocalError
when vector DB doesn't exist)
- manifest.py: RETRIEVAL_POLICY_VERSION l4.body.v1 → l4.body.v2 to
trigger object_units rebuild with PR7 fix (unit_id fallback)
- Object_units corrected from 20→311 after rebuild (was INSERT OR
REPLACE overwrite from old empty unit_id)
- Coverage parity: 20 papers have body_units = 20 have object_units ✅
(body=811 chunks, object=311 chunks, expected difference due to
section-split vs 1:1 caption structure)
- Unit tests: 97 pass (subset)
P0 fixes from test report review:
- object unit_id: fallback to f"{obj_type}:p{page}:{block_id}" when
figure_id/table_id missing (prevents DB overwrite on INSERT OR REPLACE)
- object caption_key: page-qualified f"p{page}:{caption_bid}" to match
page-qualified subtree_block_ids in find_owning_node
- object block_map: page-qualified keys for consistency with body_units
- structure node_id: changed from sec:{block_id} to sec:p{page}:{block_id}
for stability (reduces vector ID churn from emitted_order changes)
New tests:
- Section 4b: Object units DB persistence (unit_id unique, non-empty labels,
DB count == list count after upsert)
- Section 5b: Coverage gate (v2_tree / render_map / body_papers counts)
Layer A: 1669/1669 pass (+60 checks from 4b + 5b)
Unit tests: 153 pass, 1 skip
- _body_unit_role_kind now excludes only reference_item/reference_heading
(per policy: everything in fulltext except ref zone)
- backmatter_body unit_kind removed; all units are 'body'
- bounds_map in structure_tree uses page-qualified key (fixes collision
when same block_id appears on different pages)
- Layer A test plan: emitted_ids uses page-qualified keys + handles
block_id=0 (falsy value)
- Layer A: 1609 passed, 0 failed on real vault (22 papers)
- Unit tests: 152 passed, 1 pre-existing skip
- Section/subsection headings now appear in BOTH heading_events and
emitted_block_events (emitted_as='heading') for complete event stream
- Backmatter headings (Funding/Acknowledgments/Data Availability/Conflicts)
now generate heading_events → appear as structure tree sections with
their own body_units
- Page-qualified block IDs (p{page}:{block_id}) in tree own/subtree_block_ids
and body_units block_map: eliminates ambiguous block_id collisions when
same block_id appears on multiple pages in real OCR data
- TC-9.2 wording fix, KEYS empty guard, global unit_id uniqueness check
- _split_if_oversized: recursive _halve_text until every part <= 1000 tokens
- _incremental_units_only: also checks retrieval_policy_version match
- _rebuild_paper_units: use RETRIEVAL_POLICY_VERSION constant
- Functional test plan: Layer A (artifact) + Layer B (API), all 8 review
fixes incorporated (persisted artifacts, no jump-assertion, stronger
object/backmatter/abstract tests, proper embed flow)
P0-1: FTS per-paper insert — move SELECT INTO body_units_fts inside the
per-paper loop with WHERE paper_id = ? to avoid re-inserting other papers'
rows during full rebuild. Fix sqlite3.DatabaseError on empty FTS table.
P0-2: object_units real role_index keys — read from 'captions'/'tables'
(what build_role_indexes actually outputs) with fallback to old key names.
P1-1: unit_id collision for mixed body/backmatter — include unit_kind
suffix (:backmatter_body) in unit_id. Also fix duplicate tree node_ids
(block_id reused on different pages) by appending order{emitted_order}.
P1-2: Embed resume body_units_hash — add compute_body_units_hash() to
manifest.py, write hash + retrieval_policy_version to Chroma metadata,
resume compares both before skipping.
Real OCR output has the same block_id on multiple pages (e.g. running
headers). block_map[str(bid)] = b overwrites correct entries with
wrong-page duplicates. Fix: prefer blocks with non-empty text content
when multiple blocks share the same block_id.
- retrieve command calls merge_retrieve instead of retrieve_chunks
- Checks total_chunks (both collections) for empty DB detection
- Display uses section_path field from merge_retrieve response format
- get_collection(name=) supports paperforge_fulltext or paperforge_body
- delete_paper_vectors deletes from both collections
- embed_body_units + get_body_units_for_embedding (reads from memory DB)
- Status reports chunk_count, body_chunk_count, total_chunks
- Embed build routes to body_units path when DB has body_units
- merge_retrieve queries both collections with unit-level dedup
- RenderOutput dataclass with heading_events/emitted_block_events tracking
- Stack-algorithm structure tree with own/subtree block IDs
- Recursive body_units builder with role helper + token cap splitting
- Schema v4: section_path_json, section_level, section_title, part_ordinal
- Phase 4/5 reorder in ocr_rebuild.py and ocr.py
- 12 new test cases for tree nesting, own_block_ids, rendered-order intervals
1. sidecar match record now includes 'text': caption legend was
missing from the rendered figure_001.md because the sidecar
materialization did not copy the text field.
2. _find_table_caption_continuation now rejects actual <table> HTML
blocks as continuations. Previously the very next block after a
truncated caption (which was the table_html itself) got merged into
caption_text.
Add a narrow sidecar rescue for the 37-like pattern:
- one formal narrow figure caption on a page
- same-row right-side image
- no x-overlap required
- bounded x-gap and strong y-overlap
- only when primary same-page matching missed the caption
This preserves the existing >=2 narrow-caption sidecar behavior while
rescuing left-caption/right-image layouts that previously fell to
cross_page_reservation.
Verified on real vault:
- 37LK5T97 page 2: FIG. 1 now matches as figure_001 via sidecar
- QGCAFI3P page 4: narrow side-caption rescued
- QZXT9L27 page 11: narrow side-caption rescued
Targeted tests: 296 passed
PR-1 of the architecture review fixes.
Two entry-level filters before figure matching:
1. _is_table_owned_media(): excludes media_asset blocks with table
roles/raw_labels/hints from the figure asset pool. Previously these
entered as figure candidates and ended up as unmatched legends/orphans.
2. _looks_like_inline_figure_body_reference(): detects body text like
'Figure 11-10 shows...' (hyphenated ranges, list ranges) and flags
it as an inline reference, not a figure legend. Applied in both
ocr_figures.py and ocr_roles.py for consistent early rejection.
Verified on real vault:
SKXTCE6M: unmatched_legends 6→0, overall yellow→green
SRNJDAA2: unmatched_legends 9→0
Tests: 384 pass (+9 new), 0 fail
_phase4_render_health now returns (markdown, health_overall),
_phase5_finalize writes it to meta before write_json.
Also adds ADR with architecture review decisions:
- #2 Meta divergence — fixed
- #3 Quality signals for consumers — deferred to role-override v1
- #4 Health report richness — keep all fields
- #5 PDF lines cache — rejected
- #6 Integration tests — deferred to role-override v1
1. Stale test: test_extract_objects_renders_same_page_once_for_multiple_crops
now asserts render_pdf_page_cached is NOT called (new PageRenderContext path)
and pages/page_001.jpg is NOT created.
2. Extract _resolve_object_crop_pdf_path() helper with 4-case test.
Phase 4 now uses it instead of inline fallback logic.
Test locks in: Phase 1 resolved path beats stale meta['source_pdf'].
3. PageRenderContext: use fitz.csRGB in get_pixmap() for safe CMYK/n>3 handling.
Removed fragile pix.n mode detection.
4. Added test_extract_and_write_objects_with_use_disk_page_cache_false_and_valid_pdf
6 fixes bundled:
1. Cleanup: remove accidental empty file '6s}'
2. Rebuild determinism: add 'use_disk_page_cache' gate to _crop_asset_from_pdf.
Rebuild (use_disk_page_cache=False) never reads or writes pages/page_XXX.jpg.
All rendering goes through PageRenderContext in-memory.
Legacy callers keep backward-compat behavior (default True).
3. PageRenderContext safety:
- Only use when page_width>0 and page_height>0 and not rotation_deg
- Fix Pixmap.n mode detection (L/RGB/RGBA -> convert to RGB)
- Fix Image.Resampling name bug (Image -> PILImage)
- Rotated crops always fall back to direct PDF clip path
4. Phase 1+2a merged traversal: always extract PDF lines on every page,
regardless of whether span_metadata already exists. Fixes figure inventory
data gaps when pages have pre-existing span coverage.
5. Phase 4: use resolved source_pdf_path from Phase 1, not ocr_meta['source_pdf']
fallback (which may be stale or missing).
6. Added 5 determinism tests
Add PageRenderContext class that renders each page once into memory
(PIL Image via get_pixmap + frombytes), shared across all crop tasks
in a single rebuild session. No full-page JPG is written to disk.
- _crop_asset_from_pdf: new fast path (page_render_context kwarg)
- Three task functions pass context through
- extract_and_write_objects opens PDF once, creates context, closes after
- Cold Phase 4a: 4.0s → 1.24s (60p, 30 unique pages, 42 crops)
- Full cold rebuild: 10.9s → 7.17s
- Extract PDF lines from rawdict data that was already fetched for span backfill,
eliminating the separate page.get_text('dict') pass (~6s for 60 pages)
- Add _extract_lines_from_rawdict() helper to avoid rawdict re-parse
- Keep extract_pdf_lines_normalized() as backward-compatible API
- First-rebuild: 14.2s -> 10.9s (60p), 8.8s -> 8.0s (81p)
- Group raw_blocks by page, call get_text('rawdict') once per page instead of once per block
(1433 calls → 60 calls, 68s → 16.6s for a 60-page paper)
- Add _spans_from_rawdict() helper to filter cached rawdict by char bbox center overlap
- Keep extract_pdf_spans_for_block() as single-block compatibility API unchanged
- Add incremental backfill: skip pages where all blocks already have span_metadata
- Remove figure_title from _is_text_like_raw_block() coverage count (figure titles are
typically image-rendered, not selectable PDF text)
- Add regression test: fast path output matches single-block API
