FigureCandidateIndex.from_corpus was passing corpus.raw_legends to
_build_candidate_figure_groups_from_assets, which let rejected captions
inflate candidate-group legend semantics (page_legend_count, band-group
assist scoring). Pass formal_legends (accepted legends only) instead.
Adds a regression test: page with one formal legend, one rejected legend,
and one asset. After from_corpus, the candidate group reports
page_legend_count=1 even though corpus.raw_legends has length 2.
ResourceRef equality/hash must be driven only by canonical identity
fields (kind, page, block_id/group_id) so that ledger dict lookups
succeed regardless of optional metadata differences.
Change: mark figure_no and origin with compare=False in the frozen
dataclass. The __post_init__ validation/normalization is unchanged.
Test: parametrized test_resource_ref_equality_ignores_metadata covers
asset/legend/group variants with figure_no, origin, and both together.
P0: disambiguate table field keys from block_page_by_id in ocr_render.py
P1: add _is_tail_backmatter_continuation() guard, order backmatter correctly
P2: phase 4b tail nonref guard with tail_backmatter_blocks list
P3: page-continuation-marker false reference gate in ocr_families.py
P4: force reference heading before same-page refs, ref_item bypass usable gate
adds a metadata-only recovery pass that scans matched figure assets for
internal PDF line labels ("Figure N.", "Fig. N:") when the figure lacks
a figure_number.
new components:
- extract_pdf_lines_normalized() in ocr_pdf_spans.py — reads PDF rawdict
lines page-by-page, normalizes coordinates to OCR space
- _recover_missing_figure_numbers_from_assets() — inventory pass between
synthetic fallback and dedup
- _needs_asset_internal_figure_number_recovery() — gate supporting both
synthetic_figure and figure_unknown entries
- _looks_like_internal_figure_label() — regex-based label detection
- _asset_edge_band_score() — geometric rejection for center-positioned
lines or lines covering >15% asset area
- page_pdf_lines_by_page parameter added to build_figure_inventory()
verification: U746UJ7G figure_unknown_000 -> figure_002 with recovered
label "Plot of Criteria Time". 428 regression tests pass (6 new).
- preserve PyMuPDF dir/wmode in span_metadata and bump span backfill version
- promote rotated vision_footnote figure descriptions into normal legend prematch
- normalize rotated caption/asset regions before score_figure_match
- carry rotation_correction_deg into matched figures and cropped renders
- use LANCZOS resampling on rotated figure crops to avoid aliasing
- keep synthetic fallback as last resort, reusing the same rotation metadata
- Add _apply_bbox_only_synthetic_vector_fallback helper
- Score by page/x-overlap/vertical-gap with hard gates at 300px and x_ratio<0.25
- Tie-breaking: skip when top-two scores differ by <0.15
- Duplicate figure number prevention against existing matched_figures
- Inserted after _infer_missing_main_figure_numbers in pipeline
- No PNG rendering — bbox-only, render layer shows placeholder
- Minimum score threshold 0.65 to prevent false positives
- Add _FIGURE_DESCRIPTION_OPENING_PATTERN for 'This figure...' / 'Figure N' etc.
- Insert rescue before generic footnote fallback
- Route to figure_caption (near media) or figure_caption_candidate (far)
- _TABLE_PREFIX_PATTERN in ocr_roles.py, ocr_signatures.py, ocr_tables.py
now matches Table I/II/III and Table S1/S2
- _extract_table_number and _extract_marker_signature parse Roman numbers
- figure_title generic fallback checks table prefix before figure caption
- ocr_families.py adds early table-prefix guard before reference family
- Add missing continue in ocr.py exception path (UnboundLocalError page_num)
- Accept selected_keys param in test stub_run_ocr
- Update active queue content assertion for new status text
Add _container_area_ok, _container_has_media_asset, _validated_container_regions
helpers and wire validated _container_bbox regions into tag_figure_contained_text.
Containment-only fix: no ownership mutation, no new matched_figure entries.
- Clamp backfill words to original bbox (Issue 3)
- Allow validation-first bare tables to fall through (Issue 1A)
- Materialize split table caption continuations (Issue 1B)
- Add short-form OCR health profile (Issue 4)
test_previous_page_locator_bridge_cross_page_full_legend:
p15 full legend (in rejected_legends) + p16 locator + 3 assets.
Verifies: settlement_type, legend_page, page, text (full not locator),
bridge_block_ids, all 3 asset_block_ids, cluster_bbox present,
full legend removed from unmatched_legends, assets from unmatched_assets.
test_previous_page_locator_bridge_does_not_swallow_other_group:
p8 locator + Fig.8 3-asset group + unrelated 'other' asset.
Verifies: only the 3-asset group consumed; 'other' stays unmatched.
4AG67PBH P21: acknowledgment text absorbed as reference_item because:
'doctoral scholarships 2023.02051.BD' matched \b(?:19|20)\d{2}\.
Fixed to \b(?:19|20)\d{2}\.(?:\s|$) — year+period must be followed by whitespace or end of string.
Left column now correct: body (Acknowledgments, Conflict, Keywords, dates) in body_zone,
then [1]-[3] in reference_zone. Right column all 33 refs unaffected.
- Raise word limit 80→200 (real bio text is 90-100 words)
- Add structured_insert_candidate to Pass C role list (fixes Barbara bio miss)
- Fix page-agnostic block_id filter in ocr_figures.py (lines 4479, 4529)
- Was filtering by bare block_id, colliding same id across pages
- Changed to (page, block_id) tuples
- Add bio passes (Pass B+C) to ocr_rebuild.py pipeline
- residual_author_bio_pass (Pass B): detects portrait unmatched_assets
and unresolved_clusters with nearby bio text, reclassifies as
author_bio_asset
- Extend post_ref_bio_cleanup (Pass C) to handle figure_caption role
- Wire Pass B before Pass C in ocr.py pipeline
- Add tag_figure_contained_text protection: skip author_bio blocks
and author_bio_asset role
- 7 new P1 tests, 1018 total OCR tests pass, 0 regressions
Part of P1 bio detection (spec v3 §7-8, Tasks 4-5).
- _bio_text_score: category-weighted 0-5 scoring with career/education/
research/institution/publication categories
- _is_portrait_like / _any_portrait_like: image profile card detection
- _has_formal_figure_number: guard for Fig./Table/Scheme prefixes
- _looks_like_reference: guard for [N] + et al/year patterns
- _nearby_blocks: spatial proximity search with cluster_bbox support
- _resolve_ref_start_page: reference zone page detection
- _add_block_keys / _any_bio_text helpers
- post_ref_bio_cleanup: detects author bio text in post-ref reference_zone
blocks and overrides role to backmatter_body
- _is_strongly_figure_matched: guards against figure-matched blocks
- _is_protected_strong_figure / _is_reversible_weak_figure_match:
strong vs weak figure settlement type classification
- prune_figure_inventory_after_bio: removes bio artifacts from
ambiguous_figures and held_figures before reader
Part of P0 bio detection (spec v3 §4-5, §8, Tasks 1+2).
reference_zone is the only hard boundary. Pre-ref disclosure headings (CRediT,
Ethics, Declaration) are normalized as local same-column runs — they never set
global boundaries. Four contracts enforced (Contract A-D in spec).
- 13 new helpers in ocr_document.py: _has_verified_reference_zone, _partition_by_reference_zone,
_classify_same_page_block, _normalize_reference_roles_from_partition,
_normalize_pre_ref_disclosure_runs, _build_tail_boundary_from_ref_partition, etc.
- infer_zones() stores effective_end_page for trimmed reference extent
- normalize_document_structure(): ref-partitioned path replaces second _reconcile_tail_spread;
_promote_tail_body_candidates/_assign_tail_spread_ownership skipped when active
- 20 tests in test_backmatter_boundary.py
- 322/322 tests pass
Add _infer_missing_main_figure_numbers() that fills figure_number=None
for matched main-sequence figures when a single leading gap (Figure 1)
can be inferred with high confidence.
- _FRONTMATTER_VISUAL_VETO + _has_frontmatter_visual_veto()
- _FIGURE_MARKER_PATTERN regex + _extract_figure_marker()
- _coerce_int_figure_number(), _resolve_legend_bbox()
- _infer_missing_main_figure_numbers() wired into build_figure_inventory()
- 9 test cases covering acceptance, veto, isolation, skip reasons
- Fix golden vault test key drift (aliases, ocr_time, etc.)
Skip only thin-border unfilled rects with near-gray color and
low brightness — these are PyMuPDF get_drawings() false positives
from clipping paths. Keep colored thin borders (legitimate boxes)
and pure black thin borders (legitimate black frames).
Added 2 tests (colored border kept, black border kept).
Verified via 50-paper audit: all 14 dark-gray rects filtered,
all colored/black rects preserved.
N6XCZD25's body paragraphs on pages 3+ were misclassified as
structured_insert because a 0.5pt unfilled page-decoration border
rectangle (covering 80% of the page) was detected as a container.
Filter: skip bordered+unfilled rectangles with stroke_width <= 1pt
that cover >50% of page area — these are text-area frames, not
callout boxes.
When two figures share the same figure_id (e.g. supplementary
"Figure S.1" and main "Figure 1" both mapped to "figure_001"),
the second occurrence is renamed with an "s" prefix:
figure_001 -> figure_s001, figure_ss001, etc.
This avoids file overwrite in ocr_objects.py and ensures both
body and supplementary figures appear at their correct positions
in the rendered fulltext.
Changes:
- _resolve_figure_id_collisions() in ocr_figures.py
- figure_id passthrough in ocr_figure_reader.py (reader + normalize)
- _reader_figure_embed_target prefers explicit figure_id
- 11 unit tests covering single/triple/multi collisions, mixed buckets,
empty IDs, realistic body+supplementary layout
- _strip_caption_number_prefix + _normalized_caption_body helpers
- When same figure number but different caption body (and not bundle-source),
retain both as distinct legends instead of deduping
- Fixes 2UIPV93M page 49 appendix figures silently removed by dedup
- Bundle-source dedup (DWQQK2YB pattern) unaffected
- same_number_distinct_legends audit surface added to inventory
- 217 tests passed, 0 failures
- Changed _reserve_cross_page_objects to return set[tuple[int, str]]
instead of set[str] so reused block_id across pages does not
overwrite reserved legends (VFS8CBW2 page 33 vs 38 both used block_id=1)
- Updated _settle_cross_page_reserved_objects to key legends_by_id
by (page, block_id)
- Fix: reserved legends no longer filtered by caption-band assist on
competing-caption pages, so deferred hypotheses still emit
- Fix: reservation test assertion updated for tuple format
- Fix: competing-caption veto test updated for band-scoped parent
acceptance
