Three-part fix:
1. ocr_roles.py: add inline <table> check before raw_label=table → media_asset
fallback. Blocks starting with <table> now get table_html directly regardless
of raw_label.
2. ocr_structural_gate.py: add table_html verifier — inline <table> HTML is
self-identifying, accepted without structural verification.
3. ocr_document.py: remove table_html→table_html_candidate conversion step.
This role was never handled by any downstream pipeline, causing blocks to
be downgraded to unknown_structural.
Validated on AH6Q7DLC (worst case, 30 blocks): 29/30 now correctly table_html,
1 remaining is reference_item (bibliography table, different classification).
Full corpus data pending rebuild.
585 figure/table/role tests pass.
37LK5T97 had two bugs:
1. Figure 1 sidecar demotion (ocr_document.py):
- caption in left column (246px wide), image in right column
- _is_near_figure_media uses h_overlap which fails for adjacent columns
- candidate_resolution demoted the caption as 'narrative prose'
- Fix: add _is_sidecar_candidate check — if figure_caption_candidate
has vertical overlap with a media_asset but no horizontal overlap
(adjacent columns), skip the prose-based demotion.
- New narrow helper, does not change _is_near_figure_media (avoids
regression in vision_footnote routing)
2. Rotated table caption matching (ocr_scores.py):
- Tables 1-3 had rotated captions (dir=[0,-1], vertical text)
- score_table_match only checked x_overlap + asset_below_caption
- Rotated captions are beside the table body, not above it
- Fix: when caption has rotated text and x_overlap < 0.5, use
adjacent-column gap check + y_overlap ratio instead
- Non-rotated captions unaffected (elif only fires for rotated)
Result: 37LK5T97 Figure 1 matched (figure_001, asset=block_id=9).
All 6 tables (3 rotated + 1 continuation x2 + 1 normal) matched.
428 regression tests pass.
- preserve PyMuPDF dir/wmode in span_metadata and bump span backfill version
- promote rotated vision_footnote figure descriptions into normal legend prematch
- normalize rotated caption/asset regions before score_figure_match
- carry rotation_correction_deg into matched figures and cropped renders
- use LANCZOS resampling on rotated figure crops to avoid aliasing
- keep synthetic fallback as last resort, reusing the same rotation metadata
89.6% of papers (623/695) have all refs correctly labeled as
'reference_content' by the OCR model. The pipeline was ignoring
this signal and relying solely on regex patterns + a fragile
text heuristic (\b(?:19|20)\d{2}\. + commas), causing false
positives when grant numbers like '2023.02051.BD' matched.
New architecture:
1. raw_label=reference_content → immediate True (99.3% of refs)
2. marker regex patterns → fallback
3. text heuristic → emergency fallback for ~0.7% edge cases
(refs labeled as plain 'text' by OCR, e.g. parenthesized '(2)')
4AG67PBH P21: acknowledgment text absorbed as reference_item because:
'doctoral scholarships 2023.02051.BD' matched \b(?:19|20)\d{2}\.
Fixed to \b(?:19|20)\d{2}\.(?:\s|$) — year+period must be followed by whitespace or end of string.
Left column now correct: body (Acknowledgments, Conflict, Keywords, dates) in body_zone,
then [1]-[3] in reference_zone. Right column all 33 refs unaffected.
Blocks on ref page and 1 page before no longer get frontmatter_side_zone.
Fixes 'Conflict of Interest' heading on P21 (4AG67PBH) being assigned to
frontmatter_side_zone via _is_explicit_frontmatter_support_furniture path
that bypassed _is_heading_like_block check.
Adds page gate: first_reference_page is not None and page >= first_reference_page - 1
Previous approach used hardcoded text patterns + position heuristics in
assign_block_role — fragile across journal layouts.
Current approach: after the structural gate holds a heading and the zone
is resolved, normalize any block that is:
- in frontmatter_main_zone
- role=unknown_structural (held by gate — no heading evidence)
- seed_role in heading roles (section_heading, subsection_heading, etc.)
- on page 1-2
...to frontmatter_noise. This uses only layout-derived signals (zone,
structural gate decision, seed_role), no text matching, no position/width
thresholds. Cannot accidentally eat legitimate headings because only
blocks the structural gate explicitly rejected are affected.
1. _looks_like_initial_lastname_byline: require at least one lowercase letter.
Prevents journal taglines like 'A RESEARCH VISION' from matching the
initial-lastname pattern (e.g. 'J. Smith'). Fixes 24YKLTHQ block 7.
2. _is_first_page_body_start: metadata headings no longer trigger body_start.
Page 1 section_heading for 'INFORMACIÓN DEL ARTÍCULO' (Article Information)
is frontmatter metadata, not body content. Only known body-heading terms
(introduction, methods, results, discussion, conclusion) trigger body_start.
3. build_document_abstract_span backward scan: exclude non-English keyword
headings ('palabras clave', 'mots clés', 'schlagwörter') from the abstract
body candidate set. Prevents Spanish keywords block from being collected
as abstract body.
These changes fix 24YKLTHQ: block 17 (abstract) now correctly assigned
abstract_body instead of body_paragraph, block 7 no longer mislabeled as
authors, and page 1 frontmatter zone covers 5 more blocks.
When ref_partition_active=True, pre-ref blocks (pages before the reference zone)
must not inherit the stale tail_nonref_hold_zone from infer_zones(). The ref
partition correctly separates pre_ref/reference/post_ref, but _apply_zone_labels
re-applied the original region_bus zones, putting pre-ref body content back into
the tail zone.
Fix: before _apply_zone_labels, remove pre_ref block IDs (bare + _zone_block_key
format) from region_bus.tail_nonref_hold_zone's block_ids and composite_block_ids.
_apply_content_zone_fallback then correctly assigns body_zone to these blocks.
Fixes 4KCHGV2Z page 7 (20 blocks ALL previously in tail_nonref_hold_zone →
now correctly in body_zone/display_zone), and any similar case where pre-ref
body pages are adjacent to the reference zone.
reference_zone is the only hard boundary. Pre-ref disclosure headings (CRediT,
Ethics, Declaration) are normalized as local same-column runs — they never set
global boundaries. Four contracts enforced (Contract A-D in spec).
- 13 new helpers in ocr_document.py: _has_verified_reference_zone, _partition_by_reference_zone,
_classify_same_page_block, _normalize_reference_roles_from_partition,
_normalize_pre_ref_disclosure_runs, _build_tail_boundary_from_ref_partition, etc.
- infer_zones() stores effective_end_page for trimmed reference extent
- normalize_document_structure(): ref-partitioned path replaces second _reconcile_tail_spread;
_promote_tail_body_candidates/_assign_tail_spread_ownership skipped when active
- 20 tests in test_backmatter_boundary.py
- 322/322 tests pass
On two-column tail pages with reference items but no explicit reference
heading (e.g. K7R8PEKW page 16), forced backmatter section grouping
overrides the natural column-sorted reading order.
- _reorder_tail_run: add skip_section_grouping param. When set, emit
blocks in column-sorted order with reference items grouped at end,
bypassing the backmatter/body section grouping logic entirely.
- _order_tail_blocks: detect pages with ref items + no ref heading and
pass skip_section_grouping=True.
- _normalize_backmatter_roles_after_boundary: remove forced conversion
of body_paragraph to backmatter_body inside backmatter region (body
paragraphs attach naturally via _find_owning_heading in Phase 1).
Add _merge_box_content() post-processing to normalize_document_structure:
detects structured_insert blocks whose text starts with Box N or Key
insights, then consumes following body-like blocks as box content.
Consumed blocks get role=noise + render_default=False to suppress
double-rendering. Column-crossing boxes (common in multi-column
layouts) are handled by removing the x-overlap heuristic.
In _normalize_backmatter_roles_after_boundary, respect upstream
seed_role classification for frontmatter_noise blocks. Only convert
frontmatter_noise to backmatter_body when seed_role differs (indicating
misclassification), not when it matches (indicating intentional
watermark/noise suppression).
Also remove internal reader_status labels (EXACT_MATCH, LEGEND_ONLY,
ASSET_GROUP_ONLY) from rendered fulltext output.
- ocr_render.py: extend consumed frontmatter roles skip from page 1 to pages <= 2,
preventing paper_title appearing twice on pre-proof papers
- ocr_document.py: add image/media block promotion in backmatter zone normalization
so raw image blocks become media_asset instead of backmatter_body; this fixes
build_figure_inventory producing 0 matched_figures. Also check seed_role in
_sanitize_reference_zone_boundary since it runs before role resolution
- scripts/dev/ocr_rebuild_paper.py: single-paper rebuild + block_trace regeneration
entry point (will be wired into CLI later)
- tests: boundary protection and completeness check tests
- DW trace regenerated with correct role distribution
- CAQ: Move correspondence detection before zone-based furniture check
(ocr_roles.py). Correspondence footnote now correctly gets
frontmatter_support instead of frontmatter_noise.
- DW: Accept raw_label=doc_title on page 2 as paper_title repeat
(ocr_structural_gate.py). Fixes title re-verification that was
held to unknown_structural.
- DW: Tag keywords block as structured_insert when abstract span
stop_reason=keywords (ocr_document.py). Uses abstract span boundary
signal instead of text matching.
- Zone fallback: Fix unassigned role resolution in ref_heading_pages
and last_body_heading_top pre-scans (ocr_document.py). Blocks with
role=unassigned now fall through to seed_role. Also populate
ref_heading_pages from first reference_item on pages without a
reference_heading.
- Updated expectations for DW (7 FAIL, down from 9) and CAQ (0 FAIL,
down from 3). All remaining DW FAILs are user-decided skips.
In post_reference_backmatter_zone, blocks with role=reference_item are
outside the verified reference range. The structural gate cannot verify
them and holds them to unknown_structural. Fix: the zone normalization
now converts ALL non-heading blocks (including reference_item) to
backmatter_body, which the gate accepts via _SAFE_PRESERVED_ROLES.
DW expectations: 18 FAIL -> 9 FAIL (fixed 8 biography blocks + 1 equal
contribution note). Remaining 9 are pre-existing known bugs.
Adds document-level promotion rules in normalize_document_structure() so
that backmatter_heading_candidate blocks get promoted to backmatter_heading
when followed by tail-like body text. Uses text evidence (conflict-of-interest
phrases, backmatter vocabulary) + local follower evidence (short body
paragraphs) — no paper-specific checks or literal page-number logic.
Closes task 6 of the OCR-v2 pipeline.
- Pass source_frontmatter_anchors to normalize_document_structure so gate
has source-backed IDs before running (fixes paper_title/authors always HELD)
- Add table_caption CANDIDATE handler (was falling through to unknown_structural)
- Author matching: handle & splitting, strip trailing labels, initial-based
matching for abbreviated names, subset match for truncated author lists
- Override frontmatter_noise preservation when seed_role is VERIFY_REQUIRED
- Accept authors role from source anchor regardless of seed_role
- Enrich source_metadata authors from formal-library.json fallback
- Heading merge: use vertical_gap <= 30px instead of sentence-end heuristic
All 280 tests pass. TSCKAVIS/DWQQK2YB/CAQNW9Q2 rebuilt and verified.
