Commit graph

73 commits

Author SHA1 Message Date
LLLin000
2216ac7d87 fix(pr3): remove same-column guard from body_blocks, fix _page_width_for_zone_block arg 2026-07-04 22:36:46 +08:00
LLLin000
53a5336b6a feat(pr3): add column-aware same-page reference boundary 2026-07-04 22:36:46 +08:00
LLLin000
239d44bfe4 feat(ocr): checkpoint layout robustness and figure vnext groundwork on master 2026-07-03 17:48:50 +08:00
LLLin000
8dba963394 fix: complete reference detection fixes — 5 problem families 2026-07-03 01:46:27 +08:00
LLLin000
9aa228d60c feat: layout robustness layer — robust banding, role-aware gating, two-column continuation
Task 1: Add robust band estimator core module (ocr_banding.py)
  - LayoutBandEstimate, UsableContentDecision dataclasses
  - collect_layout_band_candidates, estimate_layout_bands
  - choose_runtime_bands, decide_usable_content

Task 2: Expose dry-run robust band diagnostics in normalize_document_structure
  - layout_band_estimate field on DocumentStructure
  - Serialized diagnostics with legacy vs robust comparison

Task 3: Enable robust runtime band selection
  - _order_tail_blocks uses robust estimate + choose_runtime_bands
  - Falls back to legacy-safe bands when robust holds

Task 4: Make layout band gating role-aware
  - decide_usable_content replaces _is_in_usable_content
  - Strong roles (reference_heading/item/body, backmatter_heading/boundary)
    bypass band gating entirely
  - backmatter_body gated at Phase 4b after heading ownership check

Task 5: Column-aware two-column continuation reference attachment
  - _block_column(), _should_attach_reference_item_to_ref_section()
  - rejected_ref_items list replaces body_pool fallback

Verification: 202/202 tests pass, 14/14 paper rebuilds OK,
95FDVE4W reference ordering confirmed
2026-07-02 19:15:44 +08:00
Research Assistant
d05623913b test: fix 25 stale test fixtures across 10 issues (616 OCR tests pass)
Pipeline fixes:
  - non_body_insert page-1 guard threshold: page_width * 0.2 -> 0.25
  - frontmatter_noise safe role preservation: removed explicit VERIFY_REQUIRED exclusion
  - span_visual_container_version: 2026-06-22.1 -> 2026-06-26.6

Test expectation updates:
  - non_body_insert: figure_caption test -> ponytail behavior
  - caffard abstract: accept Methods in main_ids as structured abstract subheading
  - legend_like role override: body_zone -> display_zone
  - backmatter heading render: **bold** -> ## heading
  - state machine: accept done_degraded as terminal status
  - body_zone anchor_family: relaxed assertion
  - truth surface docs: updated phrases for current phase
  - trace-vs-expectations: 10 assertions updated for post-P1 pipeline behavior
2026-06-28 23:19:47 +08:00
Research Assistant
d018a68fd6 fix: harden early OCR frontmatter-side zone inference 2026-06-25 15:57:53 +08:00
Research Assistant
5bbe40d08b fix: reject weak isolated layout clusters 2026-06-22 22:15:12 +08:00
Research Assistant
4cd0cf3d54 fix: heading detection, backmatter, table consumed, footnote reorder, figure crop + audit fixtures 2026-06-22 01:28:55 +08:00
Research Assistant
70a7978de8 feat: complete OCR-v2 readiness gates + blind audit validation
- Gate 1: completeness signals for page text coverage
- Gate 2: DW figure ownership generalization with strict regression
- Gate 3: reference boundary enforcement from structure
- Gate 4: layout coverage taxonomy in audit/coverage_ledger.json
- diff_audit.py: canonical role alias mapping + auto-normalize truth_role
- ocr_structural_gate.py: table caption text-evidence fallback
- ocr-canonical-roles.md: canonical role list for audit workflow
- ocr-truth-audit.md: pre-flight checklist enforces canonical roles
- Blind audit: 5 unseen papers across domains, all PASS with coverage
2026-06-19 00:35:21 +08:00
Research Assistant
ae66732a64 refactor: move OCR ordering authority upstream with reference boundary enforcement 2026-06-18 22:19:14 +08:00
Research Assistant
0beb9e823a feat: add OCR completeness coverage signals 2026-06-18 22:12:35 +08:00
Research Assistant
c717084465 fix: conservatively tighten OCR tail and post-reference cleanup 2026-06-18 02:01:19 +08:00
Research Assistant
918f50ade9 test: fix unified OCR close-out regression fixtures 2026-06-18 01:45:53 +08:00
Research Assistant
fc54e391b6 test: lock unified OCR close-out regressions 2026-06-18 01:39:13 +08:00
Research Assistant
7a038d717d fix: reduce false OCR tail and backmatter conversions 2026-06-17 23:51:29 +08:00
Research Assistant
06af315415 test: lock OCR close-out boundary regressions 2026-06-17 23:46:28 +08:00
Research Assistant
5fc9554c66 fix(ocr): protect figure/image blocks from backmatter normalization, fix duplicate title, and add unified rebuild entry point
- ocr_render.py: extend consumed frontmatter roles skip from page 1 to pages <= 2,
  preventing paper_title appearing twice on pre-proof papers
- ocr_document.py: add image/media block promotion in backmatter zone normalization
  so raw image blocks become media_asset instead of backmatter_body; this fixes
  build_figure_inventory producing 0 matched_figures. Also check seed_role in
  _sanitize_reference_zone_boundary since it runs before role resolution
- scripts/dev/ocr_rebuild_paper.py: single-paper rebuild + block_trace regeneration
  entry point (will be wired into CLI later)
- tests: boundary protection and completeness check tests
- DW trace regenerated with correct role distribution
2026-06-14 18:47:50 +08:00
Research Assistant
53c8e7ea1b fix(ocr): post-ref backmatter zone normalizes all non-heading blocks to backmatter_body
In post_reference_backmatter_zone, blocks with role=reference_item are
outside the verified reference range. The structural gate cannot verify
them and holds them to unknown_structural. Fix: the zone normalization
now converts ALL non-heading blocks (including reference_item) to
backmatter_body, which the gate accepts via _SAFE_PRESERVED_ROLES.

DW expectations: 18 FAIL -> 9 FAIL (fixed 8 biography blocks + 1 equal
contribution note). Remaining 9 are pre-existing known bugs.
2026-06-14 13:50:06 +08:00
Research Assistant
24a00d5e6f fix: normalize biography tail blocks as backmatter 2026-06-14 12:28:51 +08:00
Research Assistant
f309c5e42d test: lock gap surface for closure remediation 2026-06-14 11:51:10 +08:00
Research Assistant
322eeda2a0 test: add gate handler tests for figure_caption candidate preservation 2026-06-13 19:28:00 +08:00
Research Assistant
52b87c43ba test: complete OCR-v2 gate test coverage and harness integration 2026-06-13 17:21:29 +08:00
Research Assistant
7b8ed4be0e fix: close OCR v2 real-paper structural, object, and render gaps 2026-06-12 17:41:57 +08:00
Research Assistant
d9f76917e9 style: ruff format after OCR role gate implementation 2026-06-12 15:11:33 +08:00
Research Assistant
a65d992ab8 fix: prevent OCR role mutation after structural gate 2026-06-12 13:49:14 +08:00
Research Assistant
37106695e5 feat: install OCR verified role gate 2026-06-12 13:48:07 +08:00
Research Assistant
d1ff2179e9 feat: add OCR role gate context adapters 2026-06-12 12:19:11 +08:00
Research Assistant
4d0c99dd76 fix: close OCR real-paper anchor-first regressions 2026-06-10 18:18:48 +08:00
Research Assistant
f35470fb70 fix: make non-body families authoritative against body flow 2026-06-10 13:00:44 +08:00
Research Assistant
5758feb6c7 refactor: reorder OCR anchor-first pipeline 2026-06-10 12:35:45 +08:00
Research Assistant
e3fdc8b942 fix: close OCR reference zone authority on real papers 2026-06-10 01:50:08 +08:00
Research Assistant
80d409b644 fix: isolate OCR frontmatter side authority 2026-06-10 00:20:33 +08:00
Research Assistant
72625f76e8 fix: isolate OCR frontmatter side zones 2026-06-09 23:49:54 +08:00
Research Assistant
13df7189bb fix: preserve OCR reference and table boundaries 2026-06-09 22:29:43 +08:00
Research Assistant
08288ff3eb feat: switch OCR to anchor-first role authority 2026-06-09 22:10:54 +08:00
Research Assistant
6aed760a91 feat: add late OCR role resolution 2026-06-09 19:12:37 +08:00
Research Assistant
99e4bc349c feat: partition OCR families inside zones 2026-06-09 17:49:52 +08:00
Research Assistant
7f8a34c281 feat: add OCR region bus inference 2026-06-09 16:55:33 +08:00
Research Assistant
9dc32a5c98 feat: anchor OCR reference families 2026-06-09 16:19:53 +08:00
Research Assistant
9b398b8138 fix: final polish — inline figure mention detection, layout audit error gating
- Inline figure mention: _looks_like_inline_figure_mention() detects body prose
  such as 'Figure X shows/illustrates/demonstrates' and 'as shown in Figure X'
  to prevent noise in figure caption pipeline. Excludes Frontiers FIGURE N | format.
- Layout audit: added safety net so pages with layout confidence < 0.7 never
  produce ERROR-level anomalies (only INFO). Check 2 (insert_body_overlap) also
  gated by layout confidence.
- DWQQK2YB matched figure count dropped 6->3 as inline mentions correctly excluded;
  M36WA39N Frontiers captions unaffected
2026-06-08 17:49:03 +08:00
Research Assistant
f2eb35b96e fix: 5 root-cause fixes from real-paper rebuild — Frontiers figure seed, panel label exclusion, page 1 heading override, layout audit severity, first_author fallback
- Frontiers figure title: _FRONTIERS_FIGURE_TITLE_PATTERN seeds FIGURE N | Title
  as figure_caption even when raw_label=text (Frontiers/journals style)
- Panel label exclusion: _PANEL_LABEL_PATTERN sends A/B/C/(D)/E. to
  figure_inner_text before they enter the caption/matching pipeline
- Page 1 heading override: _explicit_scholarly_heading_role() detects Roman
  numeral (I. INTRODUCTION) and alpha (A. Materials) headings with y_top
  guard to protect frontmatter titles
- Layout audit severity: _run_layout_audit() separates anomalies into
  info/warning/error; full-width headings exempted; health only degraded
  when error_count > 0
- first_author fallback: _enrich_meta_from_paper_note() falls back to
  first_author when authors field is missing; _normalize_author_name()
  strips {superscript} markers for fuzzy matching
2026-06-08 17:10:51 +08:00
Research Assistant
0aab860cf2 feat: OCR risk convergence — scored figure/table matching, layout confidence, insert scorer, health uncertainty metrics
- Figure matching: nearest-asset → scored multi-candidate selection with ambiguous state
- Table matching: vertical-nearest → score-based selection with prev/next page window
- Layout confidence: PageLayoutProfile gains confidence/evidence; eligible-body block filter
- Tail reorder: guarded by layout + tail confidence (low confidence preserves raw order)
- Structured insert: direct promotion → scorer + candidate + fallback state
- Health: adds 14 uncertainty metrics (ambiguous matches, unresolved clusters, forced inserts, low tail confidence)
- Audit: docs/ocr/high-risk-rule-audit.md documents 8 hard-rule decisions mapped to plans
- Spec + 6 implementation plans for agentic execution
2026-06-08 13:43:48 +08:00
Research Assistant
c8b4846998 feat: add tail boundary score to DocumentStructure and health 2026-06-08 12:27:12 +08:00
Research Assistant
f9dff20cd8 feat: OCR complex layout and sidebar detection overhaul
- Add RegionPrepass data model for frontmatter/structured_insert/body classification
- Add PDF visual container detection (filled rects) for sidebar detection
- Add Box N and keyword-based sidebar anchor detection
- Add column-aware block expansion for mixed sidebar clusters
- Add _container_text from PDF text blocks for cross-column truncation
- Add degraded-mode heading level normalization with font size clustering
- Merge adjacent structured_insert blocks into single callout
- Fix metadata enrichment from Literature-hub note frontmatter for old papers
- Fix heading hierarchy using span font-size clusters when available
- Add real-paper regression harness (TSCKAVIS, CAQNW9Q2, A8E7SRVS, K7R8PEKW)
- Add body retention, heading, metadata, and callout acceptance tests
- 219 passed, 4 control guards passed
2026-06-07 21:35:57 +08:00
Research Assistant
8c49db1b11 fix: gate non-body insert by spine trust and region evidence 2026-06-07 16:23:06 +08:00
Research Assistant
fa418b218f feat: add final OCR PDF layout consistency audit 2026-06-07 16:20:04 +08:00
Research Assistant
ef73b3f0a1 feat: add explicit OCR no-span degraded mode 2026-06-07 16:13:09 +08:00
Research Assistant
a8f2a9fc50 feat: add OCR body spine quality diagnostics 2026-06-07 16:07:31 +08:00
Research Assistant
46eb6b70ee feat: derive body baselines from stable middle pages 2026-06-07 16:03:07 +08:00