- Removed headless_setup call from CLI dispatch entirely. --headless,
--modular, and bare all construct the same SetupPlan with identical
_cfg/zotero_path/agent_type/skip_checks. Only the deprecation notice
differs per entry point.
- Added skip_checks parameter to SetupPlan (forwarded to CLI arg).
- ConfigWriter always writes schema_version '2' unconditionally (never
preserves a v1 value from existing config).
RED commands:
python -m pytest tests/test_setup_plan_v2.py::TestConfigWriterSchemaVersion -v
python -m pytest tests/test_setup_plan_v2.py::TestCliHeadlessViaSetupPlan -v
GREEN commands (same, after fixes): 61/61 passed.
- --headless now delegates to SetupPlan for canonical config, then runs
headless_setup for full deployment; canonical ConfigWriter pass ensures
v2 vault_config format.
- --zotero-data is forwarded through SetupPlan.zotero_path for both
--modular and bare paths.
- Deprecation warnings use print(stderr) instead of filtered
DeprecationWarning so users always see them.
- ConfigWriter.PATH_KEYS extended to converge all seven legacy path keys
(system_dir, resources_dir, literature_dir, control_dir, base_dir,
skill_dir, command_dir) into vault_config, preserving non-path metadata.
- Added CLI-level tests: deprecation on stderr, config canonicalization,
--literature-dir forwarding, --zotero-data plumbing.
ConfigWriter now writes v2 canonical format (nested vault_config with
schema_version). On rerun it merges with existing config rather than
overwriting, making repeated runs idempotent.
load_vault_config reverses the v1 precedence: vault_config block now
wins over legacy top-level path keys, which are treated as a read-only
fallback with a UserWarning. This is the N+1 transition — old v1
configs remain readable with a warning; all writes produce v2 only.
Bare and --headless setup now delegate to the canonical SetupPlan engine
(the --modular path) with a DeprecationWarning.
Closes#75.
embed stop now sets status=stopping, waits for the build PID to exit
(8s timeout), force-kills as last resort, then settles to idle.
Build loop checks the cancellation flag (read_vector_build_state)
between papers in the main loop and drains in-flight work before
cleanly exiting without marking 'completed'.
Fixes pre-existing UnboundLocalError from a local import shadowing
the global mark_vector_build_state symbol in the run() function.
embed status now runs a zero-vector k-NN query on each vec0 table
that has companion meta rows. If the probe fails (table dropped,
extension unavailable, index corrupted), healthy=false and
corrupted=true are reported even when meta row counts > 0.
Includes integration test that drops vec_fulltext and verifies
healthy=false.
schema.py: CREATE_VEC_FULLTEXT_META was referenced at line 328 but
never defined. Adds the missing DDL constant so the v5 schema migration
does not raise NameError on fresh databases.
tests: add embedding test suite with four tests covering vec0 resume
detection, empty-table fallback, force drop, and delete-then-write
ordering verification.
- search.py: normalize FTS results to unified field set (score from rank,
add text/heading/source keys). Result key unchanged (already 'matches').
- retrieve.py: rename data.chunks to data.matches in standard and deep paths.
Map field names: paper_id->zotero_key, chunk_text->text, section_path->heading.
Expand enrichment SQL to fetch journal and domain.
--deep flag already wired in CLI parser.
- tests/cli/test_json_contracts.py: add contract tests verifying PFResult
envelope shape, data.matches key presence, unified field names,
--deep flag acceptance, and field type correctness.
- search.py: normalize FTS results to unified field set (score from rank,
add text/heading/source keys). Result key unchanged (already 'matches').
- retrieve.py: rename data.chunks to data.matches in standard and deep paths.
Map field names: paper_id->zotero_key, chunk_text->text, section_path->heading.
Expand enrichment SQL to fetch journal and domain.
--deep flag already wired in CLI parser.
- tests/cli/test_json_contracts.py: add contract tests verifying PFResult
envelope shape, data.matches key presence, unified field names,
--deep flag acceptance, and field type correctness.
backup_render_before_rebuild() copies render/fulltext.md (and
render-map.json, heading-events.json) to versions/v{N}/ before
rebuild overwrites them. Creates/updates versions/manifest.json
with version metadata. Idempotent: skips when no render exists.
Hook placed between phase 3 and phase 4 in _rebuild_one_paper.
6 new tests.
#27: write_encoded_payload and delete_paper_vectors use sqlite-vec
- builder.py: writes to vec0 tables + companion meta tables
- _chroma.py: delete_paper_vectors deletes from vec0 + meta by paper_id
- Stores body_units_hash, object_units_hash, retrieval_policy_version in meta tables
#28: merge_retrieve and retrieve_chunks use vec0 k-NN search
- search.py: vec0 k-NN queries with companion meta JOINs
- Same dedup/per-paper-cap logic, same result shapes
#29: build_state migrated from JSON file to SQLite build_state table
#30: E2E embed+retrieve test with sqlite-vec + FixedProvider
- 3 tests: body roundtrip, source correctness, per-paper cap
#32: E2E OCR pipeline test with fixture PDFs
Additional changes:
- schema.py: bump to v6, add hash/policy columns to vec companion meta tables
- embed.py: _assert_collections_healthy uses sqlite-vec; resume hash checks read from meta tables
- status.py: chunk counts from SQL COUNT on companion tables
- Removed unused get_collection/embed_paper imports from embed.py
- All tests updated: 76 pass (was 72, +4 new E2E tests, +4 for migration)
- FixedFixedProvider to generate 1536-dim vectors (matching vec0 schema)
- Fixed health check tests for sqlite-vec path
test(#17): integration tests for embed pipeline against EphemeralChromaDB
- build_state: write to .tmp first, leave as backup; read falls back to
.tmp if main file corrupt. Extracted default/fallback state helpers.
- integration: 10 tests covering payload prep, encode→write→retrieve
round-trip for all 3 collections, per-paper cap, delete, and
encode_paper_job — all against real ChromaDB EphemeralClient with
mocked provider
Replace two-phase (prepare all, then encode+write) with sliding-window
pipeline. Maintains bounded in-flight window (max_workers * 4 = 16),
emits EMBED_PROGRESS from the first paper completion (~2s vs ~30s).
Key changes:
- producer/consumer loop: prepare + submit one paper at a time
- wait(FIRST_COMPLETED) in main thread for encode results
- processed_count = skip + embedded, monotonic EMBED_PROGRESS
- resume skip and no-payload paths also advance processed_count
- encode failure fails closed (return 1, no silent skip)
- removed PR9B_BATCH_SIZE and _batched() (replaced by sliding window)
- 4 integration tests for progress counting, failure, and write cycle
P0 fixes from test report review:
- object unit_id: fallback to f"{obj_type}:p{page}:{block_id}" when
figure_id/table_id missing (prevents DB overwrite on INSERT OR REPLACE)
- object caption_key: page-qualified f"p{page}:{caption_bid}" to match
page-qualified subtree_block_ids in find_owning_node
- object block_map: page-qualified keys for consistency with body_units
- structure node_id: changed from sec:{block_id} to sec:p{page}:{block_id}
for stability (reduces vector ID churn from emitted_order changes)
New tests:
- Section 4b: Object units DB persistence (unit_id unique, non-empty labels,
DB count == list count after upsert)
- Section 5b: Coverage gate (v2_tree / render_map / body_papers counts)
Layer A: 1669/1669 pass (+60 checks from 4b + 5b)
Unit tests: 153 pass, 1 skip
- _body_unit_role_kind now excludes only reference_item/reference_heading
(per policy: everything in fulltext except ref zone)
- backmatter_body unit_kind removed; all units are 'body'
- bounds_map in structure_tree uses page-qualified key (fixes collision
when same block_id appears on different pages)
- Layer A test plan: emitted_ids uses page-qualified keys + handles
block_id=0 (falsy value)
- Layer A: 1609 passed, 0 failed on real vault (22 papers)
- Unit tests: 152 passed, 1 pre-existing skip
- Section/subsection headings now appear in BOTH heading_events and
emitted_block_events (emitted_as='heading') for complete event stream
- Backmatter headings (Funding/Acknowledgments/Data Availability/Conflicts)
now generate heading_events → appear as structure tree sections with
their own body_units
- Page-qualified block IDs (p{page}:{block_id}) in tree own/subtree_block_ids
and body_units block_map: eliminates ambiguous block_id collisions when
same block_id appears on multiple pages in real OCR data
- TC-9.2 wording fix, KEYS empty guard, global unit_id uniqueness check
P0-1: FTS per-paper insert — move SELECT INTO body_units_fts inside the
per-paper loop with WHERE paper_id = ? to avoid re-inserting other papers'
rows during full rebuild. Fix sqlite3.DatabaseError on empty FTS table.
P0-2: object_units real role_index keys — read from 'captions'/'tables'
(what build_role_indexes actually outputs) with fallback to old key names.
P1-1: unit_id collision for mixed body/backmatter — include unit_kind
suffix (:backmatter_body) in unit_id. Also fix duplicate tree node_ids
(block_id reused on different pages) by appending order{emitted_order}.
P1-2: Embed resume body_units_hash — add compute_body_units_hash() to
manifest.py, write hash + retrieval_policy_version to Chroma metadata,
resume compares both before skipping.
- RenderOutput dataclass with heading_events/emitted_block_events tracking
- Stack-algorithm structure tree with own/subtree block IDs
- Recursive body_units builder with role helper + token cap splitting
- Schema v4: section_path_json, section_level, section_title, part_ordinal
- Phase 4/5 reorder in ocr_rebuild.py and ocr.py
- 12 new test cases for tree nesting, own_block_ids, rendered-order intervals
Add a narrow sidecar rescue for the 37-like pattern:
- one formal narrow figure caption on a page
- same-row right-side image
- no x-overlap required
- bounded x-gap and strong y-overlap
- only when primary same-page matching missed the caption
This preserves the existing >=2 narrow-caption sidecar behavior while
rescuing left-caption/right-image layouts that previously fell to
cross_page_reservation.
Verified on real vault:
- 37LK5T97 page 2: FIG. 1 now matches as figure_001 via sidecar
- QGCAFI3P page 4: narrow side-caption rescued
- QZXT9L27 page 11: narrow side-caption rescued
Targeted tests: 296 passed
PR-1 of the architecture review fixes.
Two entry-level filters before figure matching:
1. _is_table_owned_media(): excludes media_asset blocks with table
roles/raw_labels/hints from the figure asset pool. Previously these
entered as figure candidates and ended up as unmatched legends/orphans.
2. _looks_like_inline_figure_body_reference(): detects body text like
'Figure 11-10 shows...' (hyphenated ranges, list ranges) and flags
it as an inline reference, not a figure legend. Applied in both
ocr_figures.py and ocr_roles.py for consistent early rejection.
Verified on real vault:
SKXTCE6M: unmatched_legends 6→0, overall yellow→green
SRNJDAA2: unmatched_legends 9→0
Tests: 384 pass (+9 new), 0 fail
1. Stale test: test_extract_objects_renders_same_page_once_for_multiple_crops
now asserts render_pdf_page_cached is NOT called (new PageRenderContext path)
and pages/page_001.jpg is NOT created.
2. Extract _resolve_object_crop_pdf_path() helper with 4-case test.
Phase 4 now uses it instead of inline fallback logic.
Test locks in: Phase 1 resolved path beats stale meta['source_pdf'].
3. PageRenderContext: use fitz.csRGB in get_pixmap() for safe CMYK/n>3 handling.
Removed fragile pix.n mode detection.
4. Added test_extract_and_write_objects_with_use_disk_page_cache_false_and_valid_pdf
6 fixes bundled:
1. Cleanup: remove accidental empty file '6s}'
2. Rebuild determinism: add 'use_disk_page_cache' gate to _crop_asset_from_pdf.
Rebuild (use_disk_page_cache=False) never reads or writes pages/page_XXX.jpg.
All rendering goes through PageRenderContext in-memory.
Legacy callers keep backward-compat behavior (default True).
3. PageRenderContext safety:
- Only use when page_width>0 and page_height>0 and not rotation_deg
- Fix Pixmap.n mode detection (L/RGB/RGBA -> convert to RGB)
- Fix Image.Resampling name bug (Image -> PILImage)
- Rotated crops always fall back to direct PDF clip path
4. Phase 1+2a merged traversal: always extract PDF lines on every page,
regardless of whether span_metadata already exists. Fixes figure inventory
data gaps when pages have pre-existing span coverage.
5. Phase 4: use resolved source_pdf_path from Phase 1, not ocr_meta['source_pdf']
fallback (which may be stale or missing).
6. Added 5 determinism tests
- Group raw_blocks by page, call get_text('rawdict') once per page instead of once per block
(1433 calls → 60 calls, 68s → 16.6s for a 60-page paper)
- Add _spans_from_rawdict() helper to filter cached rawdict by char bbox center overlap
- Keep extract_pdf_spans_for_block() as single-block compatibility API unchanged
- Add incremental backfill: skip pages where all blocks already have span_metadata
- Remove figure_title from _is_text_like_raw_block() coverage count (figure titles are
typically image-rendered, not selectable PDF text)
- Add regression test: fast path output matches single-block API
