Commit graph

63 commits

Author SHA1 Message Date
LLLin000
f95925b226 refactor(ocr): split legacy and vnext figure entrypoints 2026-07-03 12:01:11 +08:00
LLLin000
5af3a1f8ca feat: asset-internal figure number recovery for rotated vector figures
adds a metadata-only recovery pass that scans matched figure assets for
internal PDF line labels ("Figure N.", "Fig. N:") when the figure lacks
a figure_number.

new components:
- extract_pdf_lines_normalized() in ocr_pdf_spans.py — reads PDF rawdict
  lines page-by-page, normalizes coordinates to OCR space
- _recover_missing_figure_numbers_from_assets() — inventory pass between
  synthetic fallback and dedup
- _needs_asset_internal_figure_number_recovery() — gate supporting both
  synthetic_figure and figure_unknown entries
- _looks_like_internal_figure_label() — regex-based label detection
- _asset_edge_band_score() — geometric rejection for center-positioned
  lines or lines covering >15% asset area
- page_pdf_lines_by_page parameter added to build_figure_inventory()

verification: U746UJ7G figure_unknown_000 -> figure_002 with recovered
label "Plot of Criteria Time". 428 regression tests pass (6 new).
2026-07-02 03:00:33 +08:00
LLLin000
d41712d7bc fix: normalize rotated figure captions before normal matching
- preserve PyMuPDF dir/wmode in span_metadata and bump span backfill version
- promote rotated vision_footnote figure descriptions into normal legend prematch
- normalize rotated caption/asset regions before score_figure_match
- carry rotation_correction_deg into matched figures and cropped renders
- use LANCZOS resampling on rotated figure crops to avoid aliasing
- keep synthetic fallback as last resort, reusing the same rotation metadata
2026-07-02 02:28:56 +08:00
LLLin000
8678de48d0 feat: add bbox-only synthetic vector figure fallback
- Add _apply_bbox_only_synthetic_vector_fallback helper
- Score by page/x-overlap/vertical-gap with hard gates at 300px and x_ratio<0.25
- Tie-breaking: skip when top-two scores differ by <0.15
- Duplicate figure number prevention against existing matched_figures
- Inserted after _infer_missing_main_figure_numbers in pipeline
- No PNG rendering — bbox-only, render layer shows placeholder
- Minimum score threshold 0.65 to prevent false positives
2026-07-02 00:51:42 +08:00
LLLin000
a0d0be9048 fix: rescue figure-description text in vision_footnote from footnote role
- Add _FIGURE_DESCRIPTION_OPENING_PATTERN for 'This figure...' / 'Figure N' etc.
- Insert rescue before generic footnote fallback
- Route to figure_caption (near media) or figure_caption_candidate (far)
2026-07-02 00:48:43 +08:00
LLLin000
8c0fbeaae7 fix: add cross-column gate + figure-table asset arbitration
- Reject cross-column safe page-assets groups (Issue 5)
- Add conservative post-hoc figure/table arbitration (Issue 6)
- Wire arbitration before ownership_conflicts in both OCR pipelines
2026-07-01 21:51:03 +08:00
LLLin000
b32ddd3b29 test: add regression tests for previous-page legend locator bridge
test_previous_page_locator_bridge_cross_page_full_legend:
  p15 full legend (in rejected_legends) + p16 locator + 3 assets.
  Verifies: settlement_type, legend_page, page, text (full not locator),
  bridge_block_ids, all 3 asset_block_ids, cluster_bbox present,
  full legend removed from unmatched_legends, assets from unmatched_assets.

test_previous_page_locator_bridge_does_not_swallow_other_group:
  p8 locator + Fig.8 3-asset group + unrelated 'other' asset.
  Verifies: only the 3-asset group consumed; 'other' stays unmatched.
2026-06-29 19:28:38 +08:00
Research Assistant
5cc6011ff3 feat: figure-number inference for leading Figure 1 gaps
Add _infer_missing_main_figure_numbers() that fills figure_number=None
for matched main-sequence figures when a single leading gap (Figure 1)
can be inferred with high confidence.

- _FRONTMATTER_VISUAL_VETO + _has_frontmatter_visual_veto()
- _FIGURE_MARKER_PATTERN regex + _extract_figure_marker()
- _coerce_int_figure_number(), _resolve_legend_bbox()
- _infer_missing_main_figure_numbers() wired into build_figure_inventory()
- 9 test cases covering acceptance, veto, isolation, skip reasons
- Fix golden vault test key drift (aliases, ocr_time, etc.)
2026-06-26 21:13:24 +08:00
Research Assistant
faa70545eb fix: resolve figure_id collisions via name mutation
When two figures share the same figure_id (e.g. supplementary
"Figure S.1" and main "Figure 1" both mapped to "figure_001"),
the second occurrence is renamed with an "s" prefix:
  figure_001 -> figure_s001, figure_ss001, etc.

This avoids file overwrite in ocr_objects.py and ensures both
body and supplementary figures appear at their correct positions
in the rendered fulltext.

Changes:
- _resolve_figure_id_collisions() in ocr_figures.py
- figure_id passthrough in ocr_figure_reader.py (reader + normalize)
- _reader_figure_embed_target prefers explicit figure_id
- 11 unit tests covering single/triple/multi collisions, mixed buckets,
  empty IDs, realistic body+supplementary layout
2026-06-26 18:36:04 +08:00
Research Assistant
93c90f988a fix: page_numbered_legend_count uses ordered_legends (post-suppression) not deduped_legends 2026-06-26 00:24:00 +08:00
Research Assistant
5a494c0dce test: page_assets safety gate regressions (6 unit + 1 integration) 2026-06-26 00:16:33 +08:00
Research Assistant
1f7bd743ce fix: close out OCR merge-gate figure blockers 2026-06-24 01:09:38 +08:00
Research Assistant
fb19d5ba20 fix: sidecar outlier-tolerant narrow column + raw band for far-column panels
- _same_page_narrow_caption_column: cluster by x-center when global
  spread check fails, keep largest aligned column with >=2 captions.
  Singleton right-column outliers (misclassified body mentions) dropped.
- Sidecar promotion uses raw_band_assets directly (band partition is
  the ownership unit for sidecar pages, not row-coupled geometry).
- Add asset_block_ids to sidecar entry schema.
- Fixes 6FGDBFQN Fig 3: 2 assets [11,13] via sidecar (was assetless_shell).
  Fig 2: 2 assets [3,4] via sidecar (was 4 via same_page).
- 221 tests, 0 failures.
2026-06-23 19:29:38 +08:00
Research Assistant
c03b95964e fix: same-number distinct legend dedup guard
- _strip_caption_number_prefix + _normalized_caption_body helpers
- When same figure number but different caption body (and not bundle-source),
  retain both as distinct legends instead of deduping
- Fixes 2UIPV93M page 49 appendix figures silently removed by dedup
- Bundle-source dedup (DWQQK2YB pattern) unaffected
- same_number_distinct_legends audit surface added to inventory
- 217 tests passed, 0 failures
2026-06-23 16:56:52 +08:00
Research Assistant
1076cffc82 fix: converge figure ownership accounting and phantom cluster cleanup
- Add _score_dense_parent_candidate_against_local_ownership (Task 5)
- Dense parent built before legend loop to participate in arbitration (Task 7)
- Relax _dense_parent_single_group_ok to fragment_count >= 4
- Dense parents excluded from pages with competing captions
- Dense page unmatched assets -> unresolved_clusters with cluster_source
- Dedup unresolved_clusters against matched_figures after all matching
- Composite parent supports representative cases (2UIPV93M, RKSLQRIM);
  VFS8CBW2 resolved via same_page/cross_page after phantom fix
- 217 tests passed, 0 failures
2026-06-23 15:22:34 +08:00
Research Assistant
d7e7821c9c fix: resolve VFS8CBW2 page-32 Fig. 3 via page-aware reservation keys
- Changed _reserve_cross_page_objects to return set[tuple[int, str]]
  instead of set[str] so reused block_id across pages does not
  overwrite reserved legends (VFS8CBW2 page 33 vs 38 both used block_id=1)
- Updated _settle_cross_page_reserved_objects to key legends_by_id
  by (page, block_id)
- Fix: reserved legends no longer filtered by caption-band assist on
  competing-caption pages, so deferred hypotheses still emit
- Fix: reservation test assertion updated for tuple format
- Fix: competing-caption veto test updated for band-scoped parent
  acceptance
2026-06-23 14:29:05 +08:00
Research Assistant
b3ccea2de6 Dense composite parent candidate hardening — visual-only construction from atomic groups + unresolved clusters for dense fragmented pages 2026-06-23 11:50:22 +08:00
Research Assistant
1ef14b0030 Ticket B: Panel-title structural suppression — demote short unnumbered text inside visual envelopes on dense pages 2026-06-23 11:14:06 +08:00
Research Assistant
3e9454c464 Ticket A: Convergence scaffolding — ownership decision metadata, soft reservation, assetless sequence shell label 2026-06-23 11:06:57 +08:00
Research Assistant
fb28eca1fe fix: converge figure ownership arbitration and add VFS8CBW2 dense-composite diagnostics 2026-06-23 02:42:22 +08:00
Research Assistant
224882f697 fix: govern local figure pairing and ownership fallbacks 2026-06-22 22:15:39 +08:00
Research Assistant
111c8d340a refactor: remove obsolete _grow_region_from_seed, _validate_grown_region, gap helpers 2026-06-21 22:24:02 +08:00
Research Assistant
246d6d1c73 feat: add _cluster_page_assets with distance-based union-find clustering 2026-06-21 22:15:08 +08:00
Research Assistant
70debf1382 feat: add _filter_figure_assets helper for backward-compatible asset filtering 2026-06-21 22:14:22 +08:00
Research Assistant
d0c741b90a feat: add _has_text_separator and _rect_intersection_area for distance clustering 2026-06-21 22:13:37 +08:00
Research Assistant
a508f23d13 feat: integrate region-grown figure groups into inventory 2026-06-20 02:31:07 +08:00
Research Assistant
9812fd5979 fix: validate grown figure groups before strict ownership 2026-06-20 02:21:46 +08:00
Research Assistant
684cc37470 feat: add seed-based region growth for figure groups 2026-06-20 02:20:33 +08:00
Research Assistant
4afb85d72d fix: split figure namespace and gate page asset grouping
Part A: namespace separation - add _extract_figure_namespace and _format_figure_id helpers, change dedup key from int to (namespace,int) tuple so Figure 1 and Supplementary Figure 1 do not collide, update figure_id generation across all paths, add figure_namespace field to matched entries.

Part B: page assets gate - suppress page_assets candidate groups on pages with competing captions (multiple figure_caption blocks) so one group cannot swallow assets meant for multiple figures.

Tests: test_main_and_supplementary_figures_do_not_dedup_into_one_number, test_page_assets_does_not_strict_match_when_page_has_competing_captions
2026-06-20 00:13:26 +08:00
Research Assistant
70a7978de8 feat: complete OCR-v2 readiness gates + blind audit validation
- Gate 1: completeness signals for page text coverage
- Gate 2: DW figure ownership generalization with strict regression
- Gate 3: reference boundary enforcement from structure
- Gate 4: layout coverage taxonomy in audit/coverage_ledger.json
- diff_audit.py: canonical role alias mapping + auto-normalize truth_role
- ocr_structural_gate.py: table caption text-evidence fallback
- ocr-canonical-roles.md: canonical role list for audit workflow
- ocr-truth-audit.md: pre-flight checklist enforces canonical roles
- Blind audit: 5 unseen papers across domains, all PASS with coverage
2026-06-19 00:35:21 +08:00
Research Assistant
4d8c5ebb4a test: adjust DW ownership lock to match current partition logic 2026-06-18 18:18:22 +08:00
Research Assistant
6b21e67344 test: lock DW figure ownership regressions 2026-06-18 18:04:34 +08:00
Research Assistant
22e70e7aed fix: route narrow-caption same-page figure layouts 2026-06-16 20:19:21 +08:00
Research Assistant
3f879e20b2 test: add sidecar layout routing coverage 2026-06-16 20:17:35 +08:00
Research Assistant
b9c5fde6ee test: add conservative figure merge unit coverage 2026-06-16 19:14:34 +08:00
Research Assistant
9eae13a948 feat: report grouped figure match health metrics 2026-06-15 15:43:23 +08:00
Research Assistant
3cd8403b6f fix: prevent fallback from splitting grouped figure assets 2026-06-15 15:39:46 +08:00
Research Assistant
416232d638 test: add failing pair-group figure inventory case 2026-06-15 15:16:35 +08:00
Research Assistant
52b87c43ba test: complete OCR-v2 gate test coverage and harness integration 2026-06-13 17:21:29 +08:00
Research Assistant
41ebec7d89 fix: tighten OCR strict-figure safety and audit hygiene 2026-06-11 14:53:34 +08:00
Research Assistant
a0a6e7f42e feat: add strict OCR sequence match promotion 2026-06-10 20:48:57 +08:00
Research Assistant
aeb89bd626 fix: close OCR figure and legend object gaps
Add figure legend completeness check ensuring every numbered formal
legend lands in an explicit outcome bucket (matched/held/ambiguous/
unresolved_cluster/unmatched). No legend may disappear without inventory
explanation.

Changes:
- Add compute_figure_legend_completeness() to ocr_figures.py
- Integrate completeness into build_figure_inventory() return value
- Add figure_legend_completeness fields to health report (ocr_health.py)
- Gaps degrade overall health status and add degraded reasons
- Add 9 new tests for completeness edge cases
2026-06-10 13:24:08 +08:00
Research Assistant
08288ff3eb feat: switch OCR to anchor-first role authority 2026-06-09 22:10:54 +08:00
Research Assistant
63b56689c6 feat: tighten OCR figure validation matching 2026-06-09 21:13:53 +08:00
Research Assistant
9b398b8138 fix: final polish — inline figure mention detection, layout audit error gating
- Inline figure mention: _looks_like_inline_figure_mention() detects body prose
  such as 'Figure X shows/illustrates/demonstrates' and 'as shown in Figure X'
  to prevent noise in figure caption pipeline. Excludes Frontiers FIGURE N | format.
- Layout audit: added safety net so pages with layout confidence < 0.7 never
  produce ERROR-level anomalies (only INFO). Check 2 (insert_body_overlap) also
  gated by layout confidence.
- DWQQK2YB matched figure count dropped 6->3 as inline mentions correctly excluded;
  M36WA39N Frontiers captions unaffected
2026-06-08 17:49:03 +08:00
Research Assistant
2d0e5971cd fix: second polish — previous-page table scoring, figure confidence wiring, health hard-rule summary
- previous-page table: score_table_match() now adds previous_page evidence
  with caption_near_top/asset_near_previous_page_bottom, and asset_below_caption
  is gated to same-page only
- figure confidence: matched_figures.confidence now reads match_score.score
  instead of hardcoded 0.85/0.4 binary; match_score dict always present
- health: hard_rule_decision_count derived from decision log signals
  (structured_insert_decision + tail_promotion + candidate_resolution +
  candidate_forced) instead of fixed 0
2026-06-08 16:05:50 +08:00
Research Assistant
0aab860cf2 feat: OCR risk convergence — scored figure/table matching, layout confidence, insert scorer, health uncertainty metrics
- Figure matching: nearest-asset → scored multi-candidate selection with ambiguous state
- Table matching: vertical-nearest → score-based selection with prev/next page window
- Layout confidence: PageLayoutProfile gains confidence/evidence; eligible-body block filter
- Tail reorder: guarded by layout + tail confidence (low confidence preserves raw order)
- Structured insert: direct promotion → scorer + candidate + fallback state
- Health: adds 14 uncertainty metrics (ambiguous matches, unresolved clusters, forced inserts, low tail confidence)
- Audit: docs/ocr/high-risk-rule-audit.md documents 8 hard-rule decisions mapped to plans
- Spec + 6 implementation plans for agentic execution
2026-06-08 13:43:48 +08:00
Research Assistant
944ef640d6 feat: thread figure caption scores into inventory evidence 2026-06-08 12:25:20 +08:00
Research Assistant
b7ca334aca fix(ocr): add stable cluster_id to unresolved clusters in figure inventory 2026-06-08 12:11:54 +08:00
Research Assistant
9d48bf4122 test: preserve figure and tail mainline during frontmatter hardening 2026-06-07 16:23:36 +08:00