FigureCandidateIndex.from_corpus was passing corpus.raw_legends to
_build_candidate_figure_groups_from_assets, which let rejected captions
inflate candidate-group legend semantics (page_legend_count, band-group
assist scoring). Pass formal_legends (accepted legends only) instead.
Adds a regression test: page with one formal legend, one rejected legend,
and one asset. After from_corpus, the candidate group reports
page_legend_count=1 even though corpus.raw_legends has length 2.
ResourceRef equality/hash must be driven only by canonical identity
fields (kind, page, block_id/group_id) so that ledger dict lookups
succeed regardless of optional metadata differences.
Change: mark figure_no and origin with compare=False in the frozen
dataclass. The __post_init__ validation/normalization is unchanged.
Test: parametrized test_resource_ref_equality_ignores_metadata covers
asset/legend/group variants with figure_no, origin, and both together.
The y-distance limit was a hardcoded 100px. Two improvements:
1. Use the OCR caption block's y-top as the natural upper bound —
the FIGURE N heading must be above the detected caption bbox.
2. Fall back to page_height * 0.08 (relative, like _cluster_page_assets)
only when the block has no bbox data.
Previous code only checked the immediate next PDF line by y-order.
In multi-column layouts, the next line might be from a different column
(e.g. right column body text between left-column heading and its caption).
Now scans up to 10 lines or 100px y-distance below the heading for a
match, then returns the first match.
Three-part fix:
1. ocr_roles.py: add inline <table> check before raw_label=table → media_asset
fallback. Blocks starting with <table> now get table_html directly regardless
of raw_label.
2. ocr_structural_gate.py: add table_html verifier — inline <table> HTML is
self-identifying, accepted without structural verification.
3. ocr_document.py: remove table_html→table_html_candidate conversion step.
This role was never handled by any downstream pipeline, causing blocks to
be downgraded to unknown_structural.
Validated on AH6Q7DLC (worst case, 30 blocks): 29/30 now correctly table_html,
1 remaining is reference_item (bibliography table, different classification).
Full corpus data pending rebuild.
585 figure/table/role tests pass.
The body_zone early-exit filter unconditionally skipped all figure_caption_candidate
blocks in body_zone. After PDF prefix recovery, recovered captions now have a
'Figure N' prefix but were still blocked by the zone filter. Added
_extract_figure_number() guard so body_zone captions with a recovered number
pass through to legend matching.
Result for 5S7UI34M: 4->9 matched figures, 33->1 unmatched assets (p1 logo).
The previous insertion point was after the zone/style filter (line 3066), but
blocks with empty zone (zone=?) get filtered out by the narrative_prose check
at line 3048 before reaching the recovery code. Moving it to line 3036 (right
after text loading and before all checks) ensures the 'Figure N' prefix is
restored before any filters examine the text.
When PaddleOCR fails to detect a standalone 'Figure N' / 'FIGURE N'
heading (rendered in a bold/small-caps font that the OCR engine doesn't
read), the caption body is captured as figure_caption_candidate but
lacks the figure number prefix. The new _recover_figure_heading_prefix()
function checks the PDF text layer (via existing page_pdf_lines_by_page
infrastructure — no extra PDF open) and prepends the heading.
Key logic:
- Scans PDF text lines on the same page for 'Figure N' headings
- Confirms by checking if the NEXT PDF line (by y-order) shares ≥15
chars of common prefix with the OCR caption candidate text
- Only runs when _extract_figure_number() returns None (no figure
number in existing caption text)
- Integrated into build_figure_inventory() before _is_formal_legend()
check, so recovered captions enter the legend matching pool
Fixes 'FIGURE 1' loss in 5S7UI34M and 'FIGURE 5' loss in HQAQBSBP.
372 figure tests pass.
P0: disambiguate table field keys from block_page_by_id in ocr_render.py
P1: add _is_tail_backmatter_continuation() guard, order backmatter correctly
P2: phase 4b tail nonref guard with tail_backmatter_blocks list
P3: page-continuation-marker false reference gate in ocr_families.py
P4: force reference heading before same-page refs, ref_item bypass usable gate
37LK5T97 Tables 1-3 span dir=[0,-1] (rotated 90° content on portrait page).
The table body AND caption are both rotated — the rendered JPEG showed
vertical text.
Fix:
- Add _table_has_rotated_content() in ocr_tables.py — checks asset
span_metadata dir to detect rotated tables, returns 270° correction
- In build_table_inventory, compute union render_bbox (caption+asset)
and store render_rotation_deg in the table entry
- In ocr_objects.py table render loop, use render_bbox + rotation_deg
when available, passing rotation_deg to _crop_asset_from_pdf
Result:
Tables 1-5 (rotated): 1908x2858 -> 2858x1908 (270° corrected, readable)
Table 6 (normal): 987x191 -> unchanged (no rotation_deg)
428 regression tests pass.
37LK5T97 had two bugs:
1. Figure 1 sidecar demotion (ocr_document.py):
- caption in left column (246px wide), image in right column
- _is_near_figure_media uses h_overlap which fails for adjacent columns
- candidate_resolution demoted the caption as 'narrative prose'
- Fix: add _is_sidecar_candidate check — if figure_caption_candidate
has vertical overlap with a media_asset but no horizontal overlap
(adjacent columns), skip the prose-based demotion.
- New narrow helper, does not change _is_near_figure_media (avoids
regression in vision_footnote routing)
2. Rotated table caption matching (ocr_scores.py):
- Tables 1-3 had rotated captions (dir=[0,-1], vertical text)
- score_table_match only checked x_overlap + asset_below_caption
- Rotated captions are beside the table body, not above it
- Fix: when caption has rotated text and x_overlap < 0.5, use
adjacent-column gap check + y_overlap ratio instead
- Non-rotated captions unaffected (elif only fires for rotated)
Result: 37LK5T97 Figure 1 matched (figure_001, asset=block_id=9).
All 6 tables (3 rotated + 1 continuation x2 + 1 normal) matched.
428 regression tests pass.
Two bugs fixed in _crop_asset_from_pdf for rotated vector figures:
1. Coordinate mismatch: bbox values are in OCR coordinates (2x PDF), but
PyMuPDF get_pixmap(clip=...) expects PDF user-space coordinates.
Added per-page scale conversion using page_width/page_height vs PDF
page rect ratio. Without this, the clip started ~52pt right of the
actual chart, cutting off the left edge.
2. Triple quality loss: removed stale _apply_rotation_if_needed inner
function (PIL save+reopen). Replaced with 4x zoom render + single-pass
PIL rotate from pix.tobytes('png'). No JPEG intermediary, one rotation
interpolation instead of two. 4.7x pixel improvement.
Resolution quality:
Before (cached page path): ~862x1309 px (1.1M px)
After (4x pixmap + PIL): 2618x1914 px (5.0M px, 4.5x)
428 regression tests pass.
adds a metadata-only recovery pass that scans matched figure assets for
internal PDF line labels ("Figure N.", "Fig. N:") when the figure lacks
a figure_number.
new components:
- extract_pdf_lines_normalized() in ocr_pdf_spans.py — reads PDF rawdict
lines page-by-page, normalizes coordinates to OCR space
- _recover_missing_figure_numbers_from_assets() — inventory pass between
synthetic fallback and dedup
- _needs_asset_internal_figure_number_recovery() — gate supporting both
synthetic_figure and figure_unknown entries
- _looks_like_internal_figure_label() — regex-based label detection
- _asset_edge_band_score() — geometric rejection for center-positioned
lines or lines covering >15% asset area
- page_pdf_lines_by_page parameter added to build_figure_inventory()
verification: U746UJ7G figure_unknown_000 -> figure_002 with recovered
label "Plot of Criteria Time". 428 regression tests pass (6 new).
- preserve PyMuPDF dir/wmode in span_metadata and bump span backfill version
- promote rotated vision_footnote figure descriptions into normal legend prematch
- normalize rotated caption/asset regions before score_figure_match
- carry rotation_correction_deg into matched figures and cropped renders
- use LANCZOS resampling on rotated figure crops to avoid aliasing
- keep synthetic fallback as last resort, reusing the same rotation metadata
- Add _apply_bbox_only_synthetic_vector_fallback helper
- Score by page/x-overlap/vertical-gap with hard gates at 300px and x_ratio<0.25
- Tie-breaking: skip when top-two scores differ by <0.15
- Duplicate figure number prevention against existing matched_figures
- Inserted after _infer_missing_main_figure_numbers in pipeline
- No PNG rendering — bbox-only, render layer shows placeholder
- Minimum score threshold 0.65 to prevent false positives
- Add _FIGURE_DESCRIPTION_OPENING_PATTERN for 'This figure...' / 'Figure N' etc.
- Insert rescue before generic footnote fallback
- Route to figure_caption (near media) or figure_caption_candidate (far)
- _TABLE_PREFIX_PATTERN in ocr_roles.py, ocr_signatures.py, ocr_tables.py
now matches Table I/II/III and Table S1/S2
- _extract_table_number and _extract_marker_signature parse Roman numbers
- figure_title generic fallback checks table prefix before figure caption
- ocr_families.py adds early table-prefix guard before reference family
- Add missing continue in ocr.py exception path (UnboundLocalError page_num)
- Accept selected_keys param in test stub_run_ocr
- Update active queue content assertion for new status text
Add _container_area_ok, _container_has_media_asset, _validated_container_regions
helpers and wire validated _container_bbox regions into tag_figure_contained_text.
Containment-only fix: no ownership mutation, no new matched_figure entries.
- Clamp backfill words to original bbox (Issue 3)
- Allow validation-first bare tables to fall through (Issue 1A)
- Materialize split table caption continuations (Issue 1B)
- Add short-form OCR health profile (Issue 4)
1. cluster_bbox: compute from consumed asset bboxes, add to matched entry
2. Visual group selection: prioritize candidate_groups (composite_parent,
distance_cluster) before falling back to loose asset collection.
Fallback only fires for tight single clusters (_is_tight_asset_cluster).
3. Page-aware key: unmatched_legends and ambiguous_figures cleanup now
uses (page, block_id) tuple, not bare block_id.
4. Reader bridge consumption: _normalize_bucket passes bridge_block_ids
through to normalized output; _materialize_reader_figure includes
bridge blocks in consumed_caption_block_ids.
5. Stale _unowned_by_page: visual group search uses used_asset_page_ids
to avoid double-consumption across multiple locators on one page.
Some papers use locator captions like 'Fig. 10 (See legend on previous
page.)' on the figure's page instead of repeating the full legend. The
full legend lives on the preceding page but may not enter legend
matching (e.g. misclassified as body_paragraph due to OCR raw_label).
This commit adds a narrow exception handler in build_figure_inventory
that runs AFTER normal same-page matching but BEFORE generic fallbacks:
1. _is_previous_page_legend_locator() detects locator captions via
pattern: see/refer + (legend|figure|caption) + (previous|preceding) page
2. During legend collection, locators are separated from ordinary
legends (stored in figure_locators, not legends list)
3. After all normal matching, the bridge component:
- Extracts figure_number from locator
- Finds full_legend on previous page (in unmatched_legends or
rejected_legends with legend_like style + display_zone)
- Finds unowned visual group on locator's page (assets above locator)
- Creates matched_figure entry with full legend as caption text
Key principles:
- Normal matching always runs first; bridge only activates for still-unmatched
- Rejected_legends scan recovers full legends misclassified as body_paragraph
(e.g. raw_label=footnote, style_family=legend_like, zone=display_zone)
- Very tight gates: full_legend must have 60+ chars, same number, prev page
- Does NOT affect other figures or change global matching rules
Fixes P2#1a. Verified on WV2FF4NV Fig 10 (3-panel composite, previous-page
locator) and WV2FF4NV Fig 6.
Three cases for blocks where OCR found a text region but extracted no text:
1. missing_text_unrecovered → ocr_raw_error (backfill ran, PDF had no text)
2. missing_text_rejected → ocr_raw_error (backfill found text but it
duplicates a neighbor block)
3. _ocr_raw_status=None + raw_label=text + empty text → ocr_text_missing
(backfill never ran, no PDF path available)
Previously case 3 fell through to unknown_structural, making it
indistinguishable from genuine pipeline classification failures.
Two-part fix:
1. ocr_bio.py: clear zone ('') when post_ref_bio_cleanup converts
block to backmatter_body, so render doesn't interleave with refs
(was reference_zone from zone inference)
2. ocr_render.py: skip_section_grouping path (no reference_heading
on page) grouped backmatter_body/backmatter_heading into non_ref,
placing them before the same page's refs. On mixed-column tail
pages this put bios between [170] and [171]. Now backmatter roles
are output after refs within each page.
Fixes 2HEUD5P9 page 26 author bios appearing before [171]-[188].
89.6% of papers (623/695) have all refs correctly labeled as
'reference_content' by the OCR model. The pipeline was ignoring
this signal and relying solely on regex patterns + a fragile
text heuristic (\b(?:19|20)\d{2}\. + commas), causing false
positives when grant numbers like '2023.02051.BD' matched.
New architecture:
1. raw_label=reference_content → immediate True (99.3% of refs)
2. marker regex patterns → fallback
3. text heuristic → emergency fallback for ~0.7% edge cases
(refs labeled as plain 'text' by OCR, e.g. parenthesized '(2)')
4AG67PBH P21: acknowledgment text absorbed as reference_item because:
'doctoral scholarships 2023.02051.BD' matched \b(?:19|20)\d{2}\.
Fixed to \b(?:19|20)\d{2}\.(?:\s|$) — year+period must be followed by whitespace or end of string.
Left column now correct: body (Acknowledgments, Conflict, Keywords, dates) in body_zone,
then [1]-[3] in reference_zone. Right column all 33 refs unaffected.
- Raise word limit 80→200 (real bio text is 90-100 words)
- Add structured_insert_candidate to Pass C role list (fixes Barbara bio miss)
- Fix page-agnostic block_id filter in ocr_figures.py (lines 4479, 4529)
- Was filtering by bare block_id, colliding same id across pages
- Changed to (page, block_id) tuples
- Add bio passes (Pass B+C) to ocr_rebuild.py pipeline
- residual_author_bio_pass (Pass B): detects portrait unmatched_assets
and unresolved_clusters with nearby bio text, reclassifies as
author_bio_asset
- Extend post_ref_bio_cleanup (Pass C) to handle figure_caption role
- Wire Pass B before Pass C in ocr.py pipeline
- Add tag_figure_contained_text protection: skip author_bio blocks
and author_bio_asset role
- 7 new P1 tests, 1018 total OCR tests pass, 0 regressions
Part of P1 bio detection (spec v3 §7-8, Tasks 4-5).
- Add author_bio_asset to render_default=False and index_default=False
skip sets in ocr_blocks.py
- Insert Pass C (post_ref_bio_cleanup + prune) after write_back_figure_roles
in ocr.py pipeline
Part of P0 bio detection (spec v3 §7, §9, Task 3).
- _bio_text_score: category-weighted 0-5 scoring with career/education/
research/institution/publication categories
- _is_portrait_like / _any_portrait_like: image profile card detection
- _has_formal_figure_number: guard for Fig./Table/Scheme prefixes
- _looks_like_reference: guard for [N] + et al/year patterns
- _nearby_blocks: spatial proximity search with cluster_bbox support
- _resolve_ref_start_page: reference zone page detection
- _add_block_keys / _any_bio_text helpers
- post_ref_bio_cleanup: detects author bio text in post-ref reference_zone
blocks and overrides role to backmatter_body
- _is_strongly_figure_matched: guards against figure-matched blocks
- _is_protected_strong_figure / _is_reversible_weak_figure_match:
strong vs weak figure settlement type classification
- prune_figure_inventory_after_bio: removes bio artifacts from
ambiguous_figures and held_figures before reader
Part of P0 bio detection (spec v3 §4-5, §8, Tasks 1+2).
Blocks on ref page and 1 page before no longer get frontmatter_side_zone.
Fixes 'Conflict of Interest' heading on P21 (4AG67PBH) being assigned to
frontmatter_side_zone via _is_explicit_frontmatter_support_furniture path
that bypassed _is_heading_like_block check.
Adds page gate: first_reference_page is not None and page >= first_reference_page - 1
Previous approach used hardcoded text patterns + position heuristics in
assign_block_role — fragile across journal layouts.
Current approach: after the structural gate holds a heading and the zone
is resolved, normalize any block that is:
- in frontmatter_main_zone
- role=unknown_structural (held by gate — no heading evidence)
- seed_role in heading roles (section_heading, subsection_heading, etc.)
- on page 1-2
...to frontmatter_noise. This uses only layout-derived signals (zone,
structural gate decision, seed_role), no text matching, no position/width
thresholds. Cannot accidentally eat legitimate headings because only
blocks the structural gate explicitly rejected are affected.
'INFORMACIÓN DEL ARTÍCULO' (Article Information) is a left-column metadata
panel label, not a section heading. Previously defaulted to section_heading
at line 1243 ('unnumbered paragraph_title on page 1 outside title zone').
Structural-guard approach: only trigger text matching when the block is
narrow (< 35% page width) AND in the left column (x_center < 50% page
width). This prevents false matches on full-width body text or right-column
headings containing similar words.
1. _looks_like_initial_lastname_byline: require at least one lowercase letter.
Prevents journal taglines like 'A RESEARCH VISION' from matching the
initial-lastname pattern (e.g. 'J. Smith'). Fixes 24YKLTHQ block 7.
2. _is_first_page_body_start: metadata headings no longer trigger body_start.
Page 1 section_heading for 'INFORMACIÓN DEL ARTÍCULO' (Article Information)
is frontmatter metadata, not body content. Only known body-heading terms
(introduction, methods, results, discussion, conclusion) trigger body_start.
3. build_document_abstract_span backward scan: exclude non-English keyword
headings ('palabras clave', 'mots clés', 'schlagwörter') from the abstract
body candidate set. Prevents Spanish keywords block from being collected
as abstract body.
These changes fix 24YKLTHQ: block 17 (abstract) now correctly assigned
abstract_body instead of body_paragraph, block 7 no longer mislabeled as
authors, and page 1 frontmatter zone covers 5 more blocks.
When ref_partition_active=True, pre-ref blocks (pages before the reference zone)
must not inherit the stale tail_nonref_hold_zone from infer_zones(). The ref
partition correctly separates pre_ref/reference/post_ref, but _apply_zone_labels
re-applied the original region_bus zones, putting pre-ref body content back into
the tail zone.
Fix: before _apply_zone_labels, remove pre_ref block IDs (bare + _zone_block_key
format) from region_bus.tail_nonref_hold_zone's block_ids and composite_block_ids.
_apply_content_zone_fallback then correctly assigns body_zone to these blocks.
Fixes 4KCHGV2Z page 7 (20 blocks ALL previously in tail_nonref_hold_zone →
now correctly in body_zone/display_zone), and any similar case where pre-ref
body pages are adjacent to the reference zone.
reference_zone is the only hard boundary. Pre-ref disclosure headings (CRediT,
Ethics, Declaration) are normalized as local same-column runs — they never set
global boundaries. Four contracts enforced (Contract A-D in spec).
- 13 new helpers in ocr_document.py: _has_verified_reference_zone, _partition_by_reference_zone,
_classify_same_page_block, _normalize_reference_roles_from_partition,
_normalize_pre_ref_disclosure_runs, _build_tail_boundary_from_ref_partition, etc.
- infer_zones() stores effective_end_page for trimmed reference extent
- normalize_document_structure(): ref-partitioned path replaces second _reconcile_tail_spread;
_promote_tail_body_candidates/_assign_tail_spread_ownership skipped when active
- 20 tests in test_backmatter_boundary.py
- 322/322 tests pass
Three issues fixed:
1. score_structured_insert: _in_visual_container weight 0.3->0.45
(evidence-driven admission is now reliable)
2. score_structured_insert: add sidebar_header_phrase check (+0.2)
for 'available with this article' etc.
3. _container_text set to block text (was True bool)
4. Normalize newlines in text before keyword matching
5. Safeguard renderer for non-string _container_text
6. Bump span_visual_container_version to force backfill re-run
