Commit graph

37 commits

Author SHA1 Message Date
LLLin000
7c702b1bcf fix: exclude table-owned media and inline figure body refs from figure matching
PR-1 of the architecture review fixes.

Two entry-level filters before figure matching:

1. _is_table_owned_media(): excludes media_asset blocks with table
   roles/raw_labels/hints from the figure asset pool. Previously these
   entered as figure candidates and ended up as unmatched legends/orphans.

2. _looks_like_inline_figure_body_reference(): detects body text like
   'Figure 11-10 shows...' (hyphenated ranges, list ranges) and flags
   it as an inline reference, not a figure legend. Applied in both
   ocr_figures.py and ocr_roles.py for consistent early rejection.

Verified on real vault:
  SKXTCE6M: unmatched_legends 6→0, overall yellow→green
  SRNJDAA2: unmatched_legends 9→0

Tests: 384 pass (+9 new), 0 fail
2026-07-05 22:36:17 +08:00
LLLin000
f97365b5b4 fix(pr2): return body_paragraph for inline figure mentions regardless of zone 2026-07-04 22:28:23 +08:00
LLLin000
30cc7c3f7b feat(pr2): tighten figure caption heuristic with inline mention escape 2026-07-04 22:28:17 +08:00
LLLin000
8dba963394 fix: complete reference detection fixes — 5 problem families 2026-07-03 01:46:27 +08:00
LLLin000
a0d0be9048 fix: rescue figure-description text in vision_footnote from footnote role
- Add _FIGURE_DESCRIPTION_OPENING_PATTERN for 'This figure...' / 'Figure N' etc.
- Insert rescue before generic footnote fallback
- Route to figure_caption (near media) or figure_caption_candidate (far)
2026-07-02 00:48:43 +08:00
LLLin000
01ba102605 fix: support Roman numeral and S-prefix table captions across roles/signatures/tables
- _TABLE_PREFIX_PATTERN in ocr_roles.py, ocr_signatures.py, ocr_tables.py
  now matches Table I/II/III and Table S1/S2
- _extract_table_number and _extract_marker_signature parse Roman numbers
- figure_title generic fallback checks table prefix before figure caption
- ocr_families.py adds early table-prefix guard before reference family
2026-07-02 00:47:18 +08:00
Research Assistant
d05623913b test: fix 25 stale test fixtures across 10 issues (616 OCR tests pass)
Pipeline fixes:
  - non_body_insert page-1 guard threshold: page_width * 0.2 -> 0.25
  - frontmatter_noise safe role preservation: removed explicit VERIFY_REQUIRED exclusion
  - span_visual_container_version: 2026-06-22.1 -> 2026-06-26.6

Test expectation updates:
  - non_body_insert: figure_caption test -> ponytail behavior
  - caffard abstract: accept Methods in main_ids as structured abstract subheading
  - legend_like role override: body_zone -> display_zone
  - backmatter heading render: **bold** -> ## heading
  - state machine: accept done_degraded as terminal status
  - body_zone anchor_family: relaxed assertion
  - truth surface docs: updated phrases for current phase
  - trace-vs-expectations: 10 assertions updated for post-P1 pipeline behavior
2026-06-28 23:19:47 +08:00
Research Assistant
bcde1b0861 perf: simplify pre-ref backmatter to body classification, fix 3 residual bugs 2026-06-19 01:31:45 +08:00
Research Assistant
9074250357 test: lock DW frontmatter support regressions 2026-06-18 18:01:51 +08:00
Research Assistant
ff66752c28 fix: preserve page-1 correspondence support in OCR routing 2026-06-17 23:55:16 +08:00
Research Assistant
ff346d6149 fix: narrow sidebar heading routing 2026-06-14 11:58:52 +08:00
Research Assistant
f309c5e42d test: lock gap surface for closure remediation 2026-06-14 11:51:10 +08:00
Research Assistant
4927d35cd0 fix: keep OCR structured roles as seeds 2026-06-12 12:13:36 +08:00
Research Assistant
4d0c99dd76 fix: close OCR real-paper anchor-first regressions 2026-06-10 18:18:48 +08:00
Research Assistant
f35470fb70 fix: make non-body families authoritative against body flow 2026-06-10 13:00:44 +08:00
Research Assistant
aed5d9f3b4 fix: type OCR old-style references as reference markers and fix pre-existing test failures 2026-06-10 12:45:40 +08:00
Research Assistant
b56b7e287e refactor: split OCR seed and final roles 2026-06-10 12:06:16 +08:00
Research Assistant
08288ff3eb feat: switch OCR to anchor-first role authority 2026-06-09 22:10:54 +08:00
Research Assistant
6aed760a91 feat: add late OCR role resolution 2026-06-09 19:12:37 +08:00
Research Assistant
1ca2f7e509 fix: suppress Journal Pre-proof running header on ALL pages, not just page 1
Running headers at y<6% page height with pre-proof text are now
frontmatter_noise regardless of page number or raw_label, preventing
PaddleOCR mislabeling header as paragraph_title from inserting
page-furniture text into the body heading hierarchy
2026-06-08 19:38:31 +08:00
Research Assistant
1e310ee351 fix: pre-proof marker suppression with page-1 context guard, rescue exemption, title fallback
- _PREPROOF_MARKER suppression now guarded by page==1, y_top>8%, raw_label=paragraph_title, not_header, not_footer
- _is_preproof_marker() exported for health use
- rescue_roles_with_document_context: pre-proof frontmatter_noise blocks are exempt from body_paragraph rescue
- Page-1 title fallback: when pre-proof steals the title zone, next substantial text block becomes paper_title
- Tests: running header not suppressed, page-2 not suppressed, variants pass
2026-06-08 19:15:36 +08:00
Research Assistant
f2eb35b96e fix: 5 root-cause fixes from real-paper rebuild — Frontiers figure seed, panel label exclusion, page 1 heading override, layout audit severity, first_author fallback
- Frontiers figure title: _FRONTIERS_FIGURE_TITLE_PATTERN seeds FIGURE N | Title
  as figure_caption even when raw_label=text (Frontiers/journals style)
- Panel label exclusion: _PANEL_LABEL_PATTERN sends A/B/C/(D)/E. to
  figure_inner_text before they enter the caption/matching pipeline
- Page 1 heading override: _explicit_scholarly_heading_role() detects Roman
  numeral (I. INTRODUCTION) and alpha (A. Materials) headings with y_top
  guard to protect frontmatter titles
- Layout audit severity: _run_layout_audit() separates anomalies into
  info/warning/error; full-width headings exempted; health only degraded
  when error_count > 0
- first_author fallback: _enrich_meta_from_paper_note() falls back to
  first_author when authors field is missing; _normalize_author_name()
  strips {superscript} markers for fuzzy matching
2026-06-08 17:10:51 +08:00
Research Assistant
f9dff20cd8 feat: OCR complex layout and sidebar detection overhaul
- Add RegionPrepass data model for frontmatter/structured_insert/body classification
- Add PDF visual container detection (filled rects) for sidebar detection
- Add Box N and keyword-based sidebar anchor detection
- Add column-aware block expansion for mixed sidebar clusters
- Add _container_text from PDF text blocks for cross-column truncation
- Add degraded-mode heading level normalization with font size clustering
- Merge adjacent structured_insert blocks into single callout
- Fix metadata enrichment from Literature-hub note frontmatter for old papers
- Fix heading hierarchy using span font-size clusters when available
- Add real-paper regression harness (TSCKAVIS, CAQNW9Q2, A8E7SRVS, K7R8PEKW)
- Add body retention, heading, metadata, and callout acceptance tests
- 219 passed, 4 control guards passed
2026-06-07 21:35:57 +08:00
Research Assistant
a7d1c1176f fix: anchor OCR frontmatter to source metadata 2026-06-07 15:58:38 +08:00
Research Assistant
088fb4144d fix: prevent article-type running headers from becoming section headings 2026-06-07 12:52:02 +08:00
Research Assistant
6bdaddb366 fix: prevent author bylines from becoming section headings 2026-06-07 11:45:11 +08:00
Research Assistant
3e748dcb5d fix: gate weak boundary signals through backmatter_boundary_candidate 2026-06-07 00:59:22 +08:00
Research Assistant
e071a0b6f5 fix: demote ambiguous seed roles to candidates (figure_caption, backmatter_heading) 2026-06-06 18:40:24 +08:00
Research Assistant
67bd36c997 refactor: make document structure the single render truth
- build_structured_blocks now returns (rows, doc_structure) tuple
- ocr.py and ocr_rebuild.py pass document_structure to render_fulltext_markdown
- bare except Exception replaced with structured logging
2026-06-06 15:52:18 +08:00
Research Assistant
c974ba7480 fix: remove open-ended authors text heuristics, anchor to page-1 zone only 2026-06-06 13:28:46 +08:00
Research Assistant
6bbe00204c feat: add section-aware OCR rescue pass 2026-06-06 12:34:53 +08:00
Research Assistant
9006807127 feat: dynamic heading family discovery with profile matching, remove hardcoded font-size thresholds and absolute page gate 2026-06-06 11:21:25 +08:00
Research Assistant
d52ea37e92 feat(ocr): tail regime remediation + style-aware heading profiles + boundary detection
- Task: Bidirectional body/backmatter boundary detection (forward/backward spine + tail spread reconciliation)
- Task: Style-aware heading profiles from PDF span_metadata (extract, cluster, disambiguate)
- Task: Lock tests for tail-candidate overreach, cross-page continuation, style-aware heading detection
- 296/296 tests pass, real-paper 7C8829BD verified
2026-06-05 22:47:13 +08:00
Research Assistant
009be8c958 fix: stop paper_title pollution, text-to-heading, table HTML leak, author recovery 2026-06-05 15:02:00 +08:00
Research Assistant
f3fec67469 test: lock OCR render stabilization regressions 2026-06-05 13:31:50 +08:00
Research Assistant
86569fe01c feat: emit OCR structured block artifacts 2026-06-04 23:20:49 +08:00
Research Assistant
7406689746 feat: add OCR layered modules (roles, body spine, layout, attach) 2026-06-01 00:01:05 +08:00