1. Stale test: test_extract_objects_renders_same_page_once_for_multiple_crops
now asserts render_pdf_page_cached is NOT called (new PageRenderContext path)
and pages/page_001.jpg is NOT created.
2. Extract _resolve_object_crop_pdf_path() helper with 4-case test.
Phase 4 now uses it instead of inline fallback logic.
Test locks in: Phase 1 resolved path beats stale meta['source_pdf'].
3. PageRenderContext: use fitz.csRGB in get_pixmap() for safe CMYK/n>3 handling.
Removed fragile pix.n mode detection.
4. Added test_extract_and_write_objects_with_use_disk_page_cache_false_and_valid_pdf
6 fixes bundled:
1. Cleanup: remove accidental empty file '6s}'
2. Rebuild determinism: add 'use_disk_page_cache' gate to _crop_asset_from_pdf.
Rebuild (use_disk_page_cache=False) never reads or writes pages/page_XXX.jpg.
All rendering goes through PageRenderContext in-memory.
Legacy callers keep backward-compat behavior (default True).
3. PageRenderContext safety:
- Only use when page_width>0 and page_height>0 and not rotation_deg
- Fix Pixmap.n mode detection (L/RGB/RGBA -> convert to RGB)
- Fix Image.Resampling name bug (Image -> PILImage)
- Rotated crops always fall back to direct PDF clip path
4. Phase 1+2a merged traversal: always extract PDF lines on every page,
regardless of whether span_metadata already exists. Fixes figure inventory
data gaps when pages have pre-existing span coverage.
5. Phase 4: use resolved source_pdf_path from Phase 1, not ocr_meta['source_pdf']
fallback (which may be stale or missing).
6. Added 5 determinism tests
Add PageRenderContext class that renders each page once into memory
(PIL Image via get_pixmap + frombytes), shared across all crop tasks
in a single rebuild session. No full-page JPG is written to disk.
- _crop_asset_from_pdf: new fast path (page_render_context kwarg)
- Three task functions pass context through
- extract_and_write_objects opens PDF once, creates context, closes after
- Cold Phase 4a: 4.0s → 1.24s (60p, 30 unique pages, 42 crops)
- Full cold rebuild: 10.9s → 7.17s
- Extract PDF lines from rawdict data that was already fetched for span backfill,
eliminating the separate page.get_text('dict') pass (~6s for 60 pages)
- Add _extract_lines_from_rawdict() helper to avoid rawdict re-parse
- Keep extract_pdf_lines_normalized() as backward-compatible API
- First-rebuild: 14.2s -> 10.9s (60p), 8.8s -> 8.0s (81p)
- Group raw_blocks by page, call get_text('rawdict') once per page instead of once per block
(1433 calls → 60 calls, 68s → 16.6s for a 60-page paper)
- Add _spans_from_rawdict() helper to filter cached rawdict by char bbox center overlap
- Keep extract_pdf_spans_for_block() as single-block compatibility API unchanged
- Add incremental backfill: skip pages where all blocks already have span_metadata
- Remove figure_title from _is_text_like_raw_block() coverage count (figure titles are
typically image-rendered, not selectable PDF text)
- Add regression test: fast path output matches single-block API
Fix 1 (units.py): object unit_id now includes the entry's block_id
instead of the node's block_span, fixing PRIMARY KEY collisions when
a section has multiple objects (e.g. two figure_captions).
Fix 2 (builder.py): retrieval units are now persisted during
build_from_index(). Scans each paper's OCR output directory for
structure-tree.json, builds body+object units and manifest, then
upserts into the DB tables and meta. Also clears body_units and
object_units at rebuild start alongside the other tables.
1. show_in_base — separate hidden from maintenance vs hidden from both
2. Unified is_degraded check — done_degraded and health yellow/red
share one entry point
3. Hash error_summary in manifest instead of raw string
4. Fix Promise.withResolvers for ES compat
1. compute_use_cases output uses status/gates/reasons (not recommended/gate_results)
2. write_feedback validates every mark has result_hash and fulltext_hash
3. load_readiness_policy(policy=...) bypasses user override (reproducible)
4. confidence_and_fallbacks yellow for degraded=True or weak span coverage
Minor: append_mark doesn't mutate caller dict; has_figure_evidence includes
unmatched_legend_count and held_count.
Creates new ocr_quality.py module as pure function layer between
build_ocr_health() and evaluate_readiness(). All 5 indicators:
- rendered_text_integrity
- body_reference_structure
- figure_table_integrity
- metadata_frontmatter_quality
- confidence_and_fallbacks
10 unit tests covering shape, thresholds, applicability,
health_profile spelling, inventory precedence, and run_integrity.
The V3 pipeline has been tested against the full 555-paper vault vs legacy:
No diff: 547 (98.6%)
Diff: 5 (0.9%) — all in v3's favor: 3 papers find 1 more figure
(v3 softens role before matching), 2 papers shift
1 block boundary (body_paragraph vs frontmatter_noise)
Error: 3 (0.5%) — 2 missing raw data, 1 pre-existing v3 pipeline edge case
The 5 diffs are marginal improvements, not regressions. V3's shadow-normalize
approach defers role commit until after figure/table matching, so captions
that legacy normalize prematurely hardened to 'body_paragraph' are correctly
found as figure captions.
Changes:
- _ocr_pipeline_v3_enabled() defaults to True now; set OCR_PIPELINE_V3=0 to
revert to legacy normalize-then-match order
- Test updated to assert default=True
- Full vault corpus diff script + result report archived
figure_inner_text role means the text IS part of the figure content
(forest plot y-axis labels, nomogram variable names, panel markers).
The earlier fix listed the text as a separate below the image — that
was wrong: the text already lives inside the cropped image when the
crop bbox covers it.
Now extract_and_write_objects expands the crop bbox to include all
figure_inner_text blocks owned by the figure (via _object_owner_id,
same page). The cropped jpg contains the labels; render only emits
image + Legend.
Verified: M84CTEM9 figure_a001 (forest plot) now crops to 787x363
(previously 444x358 — missed the y-axis labels); figure_a002
(nomogram) still includes its variable labels. 105 focused tests pass.
figure_inner_text role has been recognized (side-adjacent + contained
passes) but its blocks were dropped from body AND silently dropped from
figure render output — they had nowhere to go. The owner contract
(_object_owner_id) was already stamped on the block; the renderer
just never consumed it.
Three changes:
- extract_and_write_objects now accepts structured_blocks and matches
figure_inner_text blocks to their owner figure via _object_owner_id
- render_figure_object_markdown merges the inner text as figure body
(no separate Note section — figure_inner_text IS the figure content)
- tag_figure_contained_text now stamps _object_owner_id on contained
blocks claimed via the matched-figure path, so contained text also
flows back to its owning figure (previously only side-adjacent was
bound)
Rebuild ordering fixed too: apply_object_writebacks now runs BEFORE
write_figure_inventory so claims land in the persisted inventory.
Tested: M84CTEM9 figure_a001.md now includes the forest plot y-axis
variable labels (Age/SPADI/Sex/...) as figure content; body no longer
leaks them; 105 focused tests pass.
The rebuild entry point (ocr_rebuild.py) was still using the old
scattered write_back_figure_roles / write_back_table_roles calls
and never went through the Workstream A apply_object_writebacks seam.
This meant side-adjacent figure text claims, contained text ownership,
and render-consumption skip were missing from rebuild output even
though the main ot .py path had been updated.
Tested: rebuilt M84CTEM9 fulltext.md — zero leak of Figure A1
note text into body.
- sync render/index defaults in post_match_normalize to match legacy
- add fixture-backed real-paper parity test for DWQQK2YB
- update OCR project status and active queue to 100-test state
