FigureCandidateIndex.from_corpus was passing corpus.raw_legends to
_build_candidate_figure_groups_from_assets, which let rejected captions
inflate candidate-group legend semantics (page_legend_count, band-group
assist scoring). Pass formal_legends (accepted legends only) instead.
Adds a regression test: page with one formal legend, one rejected legend,
and one asset. After from_corpus, the candidate group reports
page_legend_count=1 even though corpus.raw_legends has length 2.
ResourceRef equality/hash must be driven only by canonical identity
fields (kind, page, block_id/group_id) so that ledger dict lookups
succeed regardless of optional metadata differences.
Change: mark figure_no and origin with compare=False in the frozen
dataclass. The __post_init__ validation/normalization is unchanged.
Test: parametrized test_resource_ref_equality_ignores_metadata covers
asset/legend/group variants with figure_no, origin, and both together.
The y-distance limit was a hardcoded 100px. Two improvements:
1. Use the OCR caption block's y-top as the natural upper bound —
the FIGURE N heading must be above the detected caption bbox.
2. Fall back to page_height * 0.08 (relative, like _cluster_page_assets)
only when the block has no bbox data.
Previous code only checked the immediate next PDF line by y-order.
In multi-column layouts, the next line might be from a different column
(e.g. right column body text between left-column heading and its caption).
Now scans up to 10 lines or 100px y-distance below the heading for a
match, then returns the first match.
Three-part fix:
1. ocr_roles.py: add inline <table> check before raw_label=table → media_asset
fallback. Blocks starting with <table> now get table_html directly regardless
of raw_label.
2. ocr_structural_gate.py: add table_html verifier — inline <table> HTML is
self-identifying, accepted without structural verification.
3. ocr_document.py: remove table_html→table_html_candidate conversion step.
This role was never handled by any downstream pipeline, causing blocks to
be downgraded to unknown_structural.
Validated on AH6Q7DLC (worst case, 30 blocks): 29/30 now correctly table_html,
1 remaining is reference_item (bibliography table, different classification).
Full corpus data pending rebuild.
585 figure/table/role tests pass.
The body_zone early-exit filter unconditionally skipped all figure_caption_candidate
blocks in body_zone. After PDF prefix recovery, recovered captions now have a
'Figure N' prefix but were still blocked by the zone filter. Added
_extract_figure_number() guard so body_zone captions with a recovered number
pass through to legend matching.
Result for 5S7UI34M: 4->9 matched figures, 33->1 unmatched assets (p1 logo).
The previous insertion point was after the zone/style filter (line 3066), but
blocks with empty zone (zone=?) get filtered out by the narrative_prose check
at line 3048 before reaching the recovery code. Moving it to line 3036 (right
after text loading and before all checks) ensures the 'Figure N' prefix is
restored before any filters examine the text.
When PaddleOCR fails to detect a standalone 'Figure N' / 'FIGURE N'
heading (rendered in a bold/small-caps font that the OCR engine doesn't
read), the caption body is captured as figure_caption_candidate but
lacks the figure number prefix. The new _recover_figure_heading_prefix()
function checks the PDF text layer (via existing page_pdf_lines_by_page
infrastructure — no extra PDF open) and prepends the heading.
Key logic:
- Scans PDF text lines on the same page for 'Figure N' headings
- Confirms by checking if the NEXT PDF line (by y-order) shares ≥15
chars of common prefix with the OCR caption candidate text
- Only runs when _extract_figure_number() returns None (no figure
number in existing caption text)
- Integrated into build_figure_inventory() before _is_formal_legend()
check, so recovered captions enter the legend matching pool
Fixes 'FIGURE 1' loss in 5S7UI34M and 'FIGURE 5' loss in HQAQBSBP.
372 figure tests pass.
P0: disambiguate table field keys from block_page_by_id in ocr_render.py
P1: add _is_tail_backmatter_continuation() guard, order backmatter correctly
P2: phase 4b tail nonref guard with tail_backmatter_blocks list
P3: page-continuation-marker false reference gate in ocr_families.py
P4: force reference heading before same-page refs, ref_item bypass usable gate
37LK5T97 Tables 1-3 span dir=[0,-1] (rotated 90° content on portrait page).
The table body AND caption are both rotated — the rendered JPEG showed
vertical text.
Fix:
- Add _table_has_rotated_content() in ocr_tables.py — checks asset
span_metadata dir to detect rotated tables, returns 270° correction
- In build_table_inventory, compute union render_bbox (caption+asset)
and store render_rotation_deg in the table entry
- In ocr_objects.py table render loop, use render_bbox + rotation_deg
when available, passing rotation_deg to _crop_asset_from_pdf
Result:
Tables 1-5 (rotated): 1908x2858 -> 2858x1908 (270° corrected, readable)
Table 6 (normal): 987x191 -> unchanged (no rotation_deg)
428 regression tests pass.
37LK5T97 had two bugs:
1. Figure 1 sidecar demotion (ocr_document.py):
- caption in left column (246px wide), image in right column
- _is_near_figure_media uses h_overlap which fails for adjacent columns
- candidate_resolution demoted the caption as 'narrative prose'
- Fix: add _is_sidecar_candidate check — if figure_caption_candidate
has vertical overlap with a media_asset but no horizontal overlap
(adjacent columns), skip the prose-based demotion.
- New narrow helper, does not change _is_near_figure_media (avoids
regression in vision_footnote routing)
2. Rotated table caption matching (ocr_scores.py):
- Tables 1-3 had rotated captions (dir=[0,-1], vertical text)
- score_table_match only checked x_overlap + asset_below_caption
- Rotated captions are beside the table body, not above it
- Fix: when caption has rotated text and x_overlap < 0.5, use
adjacent-column gap check + y_overlap ratio instead
- Non-rotated captions unaffected (elif only fires for rotated)
Result: 37LK5T97 Figure 1 matched (figure_001, asset=block_id=9).
All 6 tables (3 rotated + 1 continuation x2 + 1 normal) matched.
428 regression tests pass.
Two bugs fixed in _crop_asset_from_pdf for rotated vector figures:
1. Coordinate mismatch: bbox values are in OCR coordinates (2x PDF), but
PyMuPDF get_pixmap(clip=...) expects PDF user-space coordinates.
Added per-page scale conversion using page_width/page_height vs PDF
page rect ratio. Without this, the clip started ~52pt right of the
actual chart, cutting off the left edge.
2. Triple quality loss: removed stale _apply_rotation_if_needed inner
function (PIL save+reopen). Replaced with 4x zoom render + single-pass
PIL rotate from pix.tobytes('png'). No JPEG intermediary, one rotation
interpolation instead of two. 4.7x pixel improvement.
Resolution quality:
Before (cached page path): ~862x1309 px (1.1M px)
After (4x pixmap + PIL): 2618x1914 px (5.0M px, 4.5x)
428 regression tests pass.
adds a metadata-only recovery pass that scans matched figure assets for
internal PDF line labels ("Figure N.", "Fig. N:") when the figure lacks
a figure_number.
new components:
- extract_pdf_lines_normalized() in ocr_pdf_spans.py — reads PDF rawdict
lines page-by-page, normalizes coordinates to OCR space
- _recover_missing_figure_numbers_from_assets() — inventory pass between
synthetic fallback and dedup
- _needs_asset_internal_figure_number_recovery() — gate supporting both
synthetic_figure and figure_unknown entries
- _looks_like_internal_figure_label() — regex-based label detection
- _asset_edge_band_score() — geometric rejection for center-positioned
lines or lines covering >15% asset area
- page_pdf_lines_by_page parameter added to build_figure_inventory()
verification: U746UJ7G figure_unknown_000 -> figure_002 with recovered
label "Plot of Criteria Time". 428 regression tests pass (6 new).
- preserve PyMuPDF dir/wmode in span_metadata and bump span backfill version
- promote rotated vision_footnote figure descriptions into normal legend prematch
- normalize rotated caption/asset regions before score_figure_match
- carry rotation_correction_deg into matched figures and cropped renders
- use LANCZOS resampling on rotated figure crops to avoid aliasing
- keep synthetic fallback as last resort, reusing the same rotation metadata
- Add _apply_bbox_only_synthetic_vector_fallback helper
- Score by page/x-overlap/vertical-gap with hard gates at 300px and x_ratio<0.25
- Tie-breaking: skip when top-two scores differ by <0.15
- Duplicate figure number prevention against existing matched_figures
- Inserted after _infer_missing_main_figure_numbers in pipeline
- No PNG rendering — bbox-only, render layer shows placeholder
- Minimum score threshold 0.65 to prevent false positives
- Add _FIGURE_DESCRIPTION_OPENING_PATTERN for 'This figure...' / 'Figure N' etc.
- Insert rescue before generic footnote fallback
- Route to figure_caption (near media) or figure_caption_candidate (far)
- _TABLE_PREFIX_PATTERN in ocr_roles.py, ocr_signatures.py, ocr_tables.py
now matches Table I/II/III and Table S1/S2
- _extract_table_number and _extract_marker_signature parse Roman numbers
- figure_title generic fallback checks table prefix before figure caption
- ocr_families.py adds early table-prefix guard before reference family
- Add missing continue in ocr.py exception path (UnboundLocalError page_num)
- Accept selected_keys param in test stub_run_ocr
- Update active queue content assertion for new status text
Add _container_area_ok, _container_has_media_asset, _validated_container_regions
helpers and wire validated _container_bbox regions into tag_figure_contained_text.
Containment-only fix: no ownership mutation, no new matched_figure entries.
- Clamp backfill words to original bbox (Issue 3)
- Allow validation-first bare tables to fall through (Issue 1A)
- Materialize split table caption continuations (Issue 1B)
- Add short-form OCR health profile (Issue 4)
test_previous_page_locator_bridge_cross_page_full_legend:
p15 full legend (in rejected_legends) + p16 locator + 3 assets.
Verifies: settlement_type, legend_page, page, text (full not locator),
bridge_block_ids, all 3 asset_block_ids, cluster_bbox present,
full legend removed from unmatched_legends, assets from unmatched_assets.
test_previous_page_locator_bridge_does_not_swallow_other_group:
p8 locator + Fig.8 3-asset group + unrelated 'other' asset.
Verifies: only the 3-asset group consumed; 'other' stays unmatched.
1. cluster_bbox: compute from consumed asset bboxes, add to matched entry
2. Visual group selection: prioritize candidate_groups (composite_parent,
distance_cluster) before falling back to loose asset collection.
Fallback only fires for tight single clusters (_is_tight_asset_cluster).
3. Page-aware key: unmatched_legends and ambiguous_figures cleanup now
uses (page, block_id) tuple, not bare block_id.
4. Reader bridge consumption: _normalize_bucket passes bridge_block_ids
through to normalized output; _materialize_reader_figure includes
bridge blocks in consumed_caption_block_ids.
5. Stale _unowned_by_page: visual group search uses used_asset_page_ids
to avoid double-consumption across multiple locators on one page.
Some papers use locator captions like 'Fig. 10 (See legend on previous
page.)' on the figure's page instead of repeating the full legend. The
full legend lives on the preceding page but may not enter legend
matching (e.g. misclassified as body_paragraph due to OCR raw_label).
This commit adds a narrow exception handler in build_figure_inventory
that runs AFTER normal same-page matching but BEFORE generic fallbacks:
1. _is_previous_page_legend_locator() detects locator captions via
pattern: see/refer + (legend|figure|caption) + (previous|preceding) page
2. During legend collection, locators are separated from ordinary
legends (stored in figure_locators, not legends list)
3. After all normal matching, the bridge component:
- Extracts figure_number from locator
- Finds full_legend on previous page (in unmatched_legends or
rejected_legends with legend_like style + display_zone)
- Finds unowned visual group on locator's page (assets above locator)
- Creates matched_figure entry with full legend as caption text
Key principles:
- Normal matching always runs first; bridge only activates for still-unmatched
- Rejected_legends scan recovers full legends misclassified as body_paragraph
(e.g. raw_label=footnote, style_family=legend_like, zone=display_zone)
- Very tight gates: full_legend must have 60+ chars, same number, prev page
- Does NOT affect other figures or change global matching rules
Fixes P2#1a. Verified on WV2FF4NV Fig 10 (3-panel composite, previous-page
locator) and WV2FF4NV Fig 6.
Three cases for blocks where OCR found a text region but extracted no text:
1. missing_text_unrecovered → ocr_raw_error (backfill ran, PDF had no text)
2. missing_text_rejected → ocr_raw_error (backfill found text but it
duplicates a neighbor block)
3. _ocr_raw_status=None + raw_label=text + empty text → ocr_text_missing
(backfill never ran, no PDF path available)
Previously case 3 fell through to unknown_structural, making it
indistinguishable from genuine pipeline classification failures.
