backup_render_before_rebuild() copies render/fulltext.md (and
render-map.json, heading-events.json) to versions/v{N}/ before
rebuild overwrites them. Creates/updates versions/manifest.json
with version metadata. Idempotent: skips when no render exists.
Hook placed between phase 3 and phase 4 in _rebuild_one_paper.
6 new tests.
- #13: Critical — swap delete/write order in _complete_one (write new
vectors before deleting old ones) to prevent data loss on write failure
- #12: Add logger.warning() to 3 resume-skip except blocks so silent
re-embed fallbacks are observable
- #14: Add lightweight HNSW query probe to _assert_collections_healthy
- #11: Log collection query failures in merge_retrieve instead of silent
continue
- #16: Reuse provider across payloads in encode_paper_job — one creation
per worker per paper instead of per payload
P0 fixes from test report review:
- object unit_id: fallback to f"{obj_type}:p{page}:{block_id}" when
figure_id/table_id missing (prevents DB overwrite on INSERT OR REPLACE)
- object caption_key: page-qualified f"p{page}:{caption_bid}" to match
page-qualified subtree_block_ids in find_owning_node
- object block_map: page-qualified keys for consistency with body_units
- structure node_id: changed from sec:{block_id} to sec:p{page}:{block_id}
for stability (reduces vector ID churn from emitted_order changes)
New tests:
- Section 4b: Object units DB persistence (unit_id unique, non-empty labels,
DB count == list count after upsert)
- Section 5b: Coverage gate (v2_tree / render_map / body_papers counts)
Layer A: 1669/1669 pass (+60 checks from 4b + 5b)
Unit tests: 153 pass, 1 skip
- _body_unit_role_kind now excludes only reference_item/reference_heading
(per policy: everything in fulltext except ref zone)
- backmatter_body unit_kind removed; all units are 'body'
- bounds_map in structure_tree uses page-qualified key (fixes collision
when same block_id appears on different pages)
- Layer A test plan: emitted_ids uses page-qualified keys + handles
block_id=0 (falsy value)
- Layer A: 1609 passed, 0 failed on real vault (22 papers)
- Unit tests: 152 passed, 1 pre-existing skip
- Section/subsection headings now appear in BOTH heading_events and
emitted_block_events (emitted_as='heading') for complete event stream
- Backmatter headings (Funding/Acknowledgments/Data Availability/Conflicts)
now generate heading_events → appear as structure tree sections with
their own body_units
- Page-qualified block IDs (p{page}:{block_id}) in tree own/subtree_block_ids
and body_units block_map: eliminates ambiguous block_id collisions when
same block_id appears on multiple pages in real OCR data
- TC-9.2 wording fix, KEYS empty guard, global unit_id uniqueness check
- _split_if_oversized: recursive _halve_text until every part <= 1000 tokens
- _incremental_units_only: also checks retrieval_policy_version match
- _rebuild_paper_units: use RETRIEVAL_POLICY_VERSION constant
- Functional test plan: Layer A (artifact) + Layer B (API), all 8 review
fixes incorporated (persisted artifacts, no jump-assertion, stronger
object/backmatter/abstract tests, proper embed flow)
- RenderOutput dataclass with heading_events/emitted_block_events tracking
- Stack-algorithm structure tree with own/subtree block IDs
- Recursive body_units builder with role helper + token cap splitting
- Schema v4: section_path_json, section_level, section_title, part_ordinal
- Phase 4/5 reorder in ocr_rebuild.py and ocr.py
- 12 new test cases for tree nesting, own_block_ids, rendered-order intervals
PR-1 of the architecture review fixes.
Two entry-level filters before figure matching:
1. _is_table_owned_media(): excludes media_asset blocks with table
roles/raw_labels/hints from the figure asset pool. Previously these
entered as figure candidates and ended up as unmatched legends/orphans.
2. _looks_like_inline_figure_body_reference(): detects body text like
'Figure 11-10 shows...' (hyphenated ranges, list ranges) and flags
it as an inline reference, not a figure legend. Applied in both
ocr_figures.py and ocr_roles.py for consistent early rejection.
Verified on real vault:
SKXTCE6M: unmatched_legends 6→0, overall yellow→green
SRNJDAA2: unmatched_legends 9→0
Tests: 384 pass (+9 new), 0 fail
_phase4_render_health now returns (markdown, health_overall),
_phase5_finalize writes it to meta before write_json.
Also adds ADR with architecture review decisions:
- #2 Meta divergence — fixed
- #3 Quality signals for consumers — deferred to role-override v1
- #4 Health report richness — keep all fields
- #5 PDF lines cache — rejected
- #6 Integration tests — deferred to role-override v1
1. compute_use_cases output uses status/gates/reasons (not recommended/gate_results)
2. write_feedback validates every mark has result_hash and fulltext_hash
3. load_readiness_policy(policy=...) bypasses user override (reproducible)
4. confidence_and_fallbacks yellow for degraded=True or weak span coverage
Minor: append_mark doesn't mutate caller dict; has_figure_evidence includes
unmatched_legend_count and held_count.
8 spec corrections from review:
1. PR 1: field-based return, not lines; insertion before title
2. PR 2: explicit body_paragraph escape in assign_block_role
3. PR 2: caption syntax supports delimiter + title patterns
4. PR 3: use existing ref_heading_block, no new scan
5. PR 3: same_page_tail_blocks + column check, test body rendering
6. PR 4: split None into full_width vs ambiguous_center
7. PR 4: column check before safe_auto_match, no sidecar exception
8. PR 2: fixture verification criteria per-fixture, not blanket
Based on layout truth audit findings (6 bug patterns), designs 4
independent PRs with precise code locations, helper contracts,
column-aware zone inference, caption formalness tightening,
and render frontmatter fallback. Includes before/after Mermaid
diagrams, test specs, and execution order.
Workstream X: 11 papers audited across 6 layout classes,
5 verified via vision subagents. 6 real bug patterns identified
with code locations and root cause analysis.
Bug patterns:
A - _is_obviously_formal_figure_caption heuristic too aggressive
(ocr_roles.py:193)
B - Cross-column figure asset mis-assignment ignores boundaries
(ocr_figures.py:704)
C - Render frontmatter skip silently drops authors/affiliations
(ocr_render.py)
D - Two-column same-page boundary body→backmatter zone pollution
(ocr_document.py)
E - Frontmatter headings misclassified (ocr_roles.py)
F - Supplementary-only PDFs not handled
Full report: docs/superpowers/analysis/2026-07-05-layout-truth-audit-findings.md
Also: codebase-memory-mcp 0.8.1 installed and repo indexed.
The V3 pipeline has been tested against the full 555-paper vault vs legacy:
No diff: 547 (98.6%)
Diff: 5 (0.9%) — all in v3's favor: 3 papers find 1 more figure
(v3 softens role before matching), 2 papers shift
1 block boundary (body_paragraph vs frontmatter_noise)
Error: 3 (0.5%) — 2 missing raw data, 1 pre-existing v3 pipeline edge case
The 5 diffs are marginal improvements, not regressions. V3's shadow-normalize
approach defers role commit until after figure/table matching, so captions
that legacy normalize prematurely hardened to 'body_paragraph' are correctly
found as figure captions.
Changes:
- _ocr_pipeline_v3_enabled() defaults to True now; set OCR_PIPELINE_V3=0 to
revert to legacy normalize-then-match order
- Test updated to assert default=True
- Full vault corpus diff script + result report archived
- Rewrite pre_match_normalize to run a shadow normalize pass that
populates role_candidate on each block while preserving public role
- Add _match_role(block) helper to ocr_figures.py (role_candidate > role > seed_role)
- Port matching-time role reads in figure builders to use _match_role
- Add _match_role(block) helper to ocr_tables.py (same priority)
- Port matching-time role reads in table builders to use _match_role
- Port role reads in FigureCorpus.from_blocks and TableCorpus.from_blocks
- Add 3 C1 tests to test_ocr_pipeline_v3.py
- All 325 OCR tests pass (6 C0+C1 + 4 rendering + 5 writeback + 310 figures)
3 distinct bugs found and fixed:
1. TABLE in figure regex (_FIGURE_NUMBER_PATTERN)
Removed TABLE/Table from figure regex — caused table captions to be
extracted as figure numbers and roadmapped into figure inventory.
Added unit tests (table_numeric_caption_is_not_a_figure_number,
table_appendix_caption_marker_has_no_figure_number).
2. int block_id type mismatch in cross-page lookup
CrossPageSettlementPass and PrimarySamePagePass compared block_id
with === but deduped_legends use int keys while ResourceRef stores
str. Fixed by casting both to str. Table continuation lookup
missed page filter, picking wrong page's same block_id. Added
page filter + int_block_id tests.
3. Table appendix support gaps
- Table prefix regex only accepted digits/roman, not 'TABLE A1'.
Added [A-Z]\d+ token and strip-leading-alpha in parse.
- _is_validation_first_table_candidate didn't cover figure_title
raw_label blocks with table_caption_like style. Added second
gate.
- _is_weak_explicit_table_caption only checked table_caption roles.
Extended to include validation-first candidates.
- Same-page tie-break didn't apply for weak-explicit captions.
Ported _bare_table_tie_break into vnext pass.
- figure_caption_candidate not excluded from note attachment.
- Continuation merge stripped leading duplicate table marker.
M84CTEM9 vault verification: 6/6 figures matched (3 main + 3
appendix + assets), 4/4 tables (Table 1,2 + Table A1,A2 + assets),
0 false positives, 0 figure asset leakage.
