Commit graph

289 commits

Author SHA1 Message Date
Research Assistant
1e310ee351 fix: pre-proof marker suppression with page-1 context guard, rescue exemption, title fallback
- _PREPROOF_MARKER suppression now guarded by page==1, y_top>8%, raw_label=paragraph_title, not_header, not_footer
- _is_preproof_marker() exported for health use
- rescue_roles_with_document_context: pre-proof frontmatter_noise blocks are exempt from body_paragraph rescue
- Page-1 title fallback: when pre-proof steals the title zone, next substantial text block becomes paper_title
- Tests: running header not suppressed, page-2 not suppressed, variants pass
2026-06-08 19:15:36 +08:00
Research Assistant
9b398b8138 fix: final polish — inline figure mention detection, layout audit error gating
- Inline figure mention: _looks_like_inline_figure_mention() detects body prose
  such as 'Figure X shows/illustrates/demonstrates' and 'as shown in Figure X'
  to prevent noise in figure caption pipeline. Excludes Frontiers FIGURE N | format.
- Layout audit: added safety net so pages with layout confidence < 0.7 never
  produce ERROR-level anomalies (only INFO). Check 2 (insert_body_overlap) also
  gated by layout confidence.
- DWQQK2YB matched figure count dropped 6->3 as inline mentions correctly excluded;
  M36WA39N Frontiers captions unaffected
2026-06-08 17:49:03 +08:00
Research Assistant
f2eb35b96e fix: 5 root-cause fixes from real-paper rebuild — Frontiers figure seed, panel label exclusion, page 1 heading override, layout audit severity, first_author fallback
- Frontiers figure title: _FRONTIERS_FIGURE_TITLE_PATTERN seeds FIGURE N | Title
  as figure_caption even when raw_label=text (Frontiers/journals style)
- Panel label exclusion: _PANEL_LABEL_PATTERN sends A/B/C/(D)/E. to
  figure_inner_text before they enter the caption/matching pipeline
- Page 1 heading override: _explicit_scholarly_heading_role() detects Roman
  numeral (I. INTRODUCTION) and alpha (A. Materials) headings with y_top
  guard to protect frontmatter titles
- Layout audit severity: _run_layout_audit() separates anomalies into
  info/warning/error; full-width headings exempted; health only degraded
  when error_count > 0
- first_author fallback: _enrich_meta_from_paper_note() falls back to
  first_author when authors field is missing; _normalize_author_name()
  strips {superscript} markers for fuzzy matching
2026-06-08 17:10:51 +08:00
Research Assistant
2d0e5971cd fix: second polish — previous-page table scoring, figure confidence wiring, health hard-rule summary
- previous-page table: score_table_match() now adds previous_page evidence
  with caption_near_top/asset_near_previous_page_bottom, and asset_below_caption
  is gated to same-page only
- figure confidence: matched_figures.confidence now reads match_score.score
  instead of hardcoded 0.85/0.4 binary; match_score dict always present
- health: hard_rule_decision_count derived from decision log signals
  (structured_insert_decision + tail_promotion + candidate_resolution +
  candidate_forced) instead of fixed 0
2026-06-08 16:05:50 +08:00
Research Assistant
0aab860cf2 feat: OCR risk convergence — scored figure/table matching, layout confidence, insert scorer, health uncertainty metrics
- Figure matching: nearest-asset → scored multi-candidate selection with ambiguous state
- Table matching: vertical-nearest → score-based selection with prev/next page window
- Layout confidence: PageLayoutProfile gains confidence/evidence; eligible-body block filter
- Tail reorder: guarded by layout + tail confidence (low confidence preserves raw order)
- Structured insert: direct promotion → scorer + candidate + fallback state
- Health: adds 14 uncertainty metrics (ambiguous matches, unresolved clusters, forced inserts, low tail confidence)
- Audit: docs/ocr/high-risk-rule-audit.md documents 8 hard-rule decisions mapped to plans
- Spec + 6 implementation plans for agentic execution
2026-06-08 13:43:48 +08:00
Research Assistant
84112c3817 feat: classify OCR errors and degraded completions 2026-06-08 12:34:44 +08:00
Research Assistant
dbab9c5ddf feat: add confidence distributions to OCR health report 2026-06-08 12:27:40 +08:00
Research Assistant
c8b4846998 feat: add tail boundary score to DocumentStructure and health 2026-06-08 12:27:12 +08:00
Research Assistant
46937f2a0a feat: thread table match scores into table inventory 2026-06-08 12:25:55 +08:00
Research Assistant
944ef640d6 feat: thread figure caption scores into inventory evidence 2026-06-08 12:25:20 +08:00
Research Assistant
b8473dac47 feat: add OCR evidence scorers (figure_caption, table_match, tail_boundary) 2026-06-08 12:24:14 +08:00
Research Assistant
38527cf5d6 feat: add OCR decision log and health summaries 2026-06-08 12:22:41 +08:00
Research Assistant
c036ef7670 fix(ocr): align object filenames with unresolved cluster IDs 2026-06-08 12:14:15 +08:00
Research Assistant
acd28d321e test: make OCR real-paper regressions reproducible 2026-06-08 12:13:11 +08:00
Research Assistant
a3c8dab389 docs: audit OCR production route and mark experimental modules 2026-06-08 12:13:07 +08:00
Research Assistant
64915aa3f2 fix(ocr): prevent empty wikilinks for unresolved clusters in renderer 2026-06-08 12:13:02 +08:00
Research Assistant
b7ca334aca fix(ocr): add stable cluster_id to unresolved clusters in figure inventory 2026-06-08 12:11:54 +08:00
Research Assistant
e39b82b99a fix: close structural convergence gaps — primary-path frontmatter anchor, health spine alignment, unresolved figure clusters, artifact cache cleanup 2026-06-08 00:24:54 +08:00
Research Assistant
463cd4d7b1 fix: references, figure matching, and OCR structural fixes
- Fix references zone detection without heading (backward reference fallback)
- Fix reference_item silently dropped in _reorder_tail_run_fifo when no heading
- Fix figure proximity matching: use nearest legend regardless of column order
- Fix figure numbering in extract_and_write_objects: use actual figure_id
- Fix body→reference_item rescue: require text starts with numeric ref pattern
- Fix tail reading order rebuilt after block normalization
- A8E7SRVS: corrected figure-legend pairing on page 5
- CAQNW9Q2: references now render without heading
- K7R8PEKW: adjusted body retention threshold
2026-06-07 22:27:58 +08:00
Research Assistant
f9dff20cd8 feat: OCR complex layout and sidebar detection overhaul
- Add RegionPrepass data model for frontmatter/structured_insert/body classification
- Add PDF visual container detection (filled rects) for sidebar detection
- Add Box N and keyword-based sidebar anchor detection
- Add column-aware block expansion for mixed sidebar clusters
- Add _container_text from PDF text blocks for cross-column truncation
- Add degraded-mode heading level normalization with font size clustering
- Merge adjacent structured_insert blocks into single callout
- Fix metadata enrichment from Literature-hub note frontmatter for old papers
- Fix heading hierarchy using span font-size clusters when available
- Add real-paper regression harness (TSCKAVIS, CAQNW9Q2, A8E7SRVS, K7R8PEKW)
- Add body retention, heading, metadata, and callout acceptance tests
- 219 passed, 4 control guards passed
2026-06-07 21:35:57 +08:00
Research Assistant
9d48bf4122 test: preserve figure and tail mainline during frontmatter hardening 2026-06-07 16:23:36 +08:00
Research Assistant
8c49db1b11 fix: gate non-body insert by spine trust and region evidence 2026-06-07 16:23:06 +08:00
Research Assistant
fa418b218f feat: add final OCR PDF layout consistency audit 2026-06-07 16:20:04 +08:00
Research Assistant
ef73b3f0a1 feat: add explicit OCR no-span degraded mode 2026-06-07 16:13:09 +08:00
Research Assistant
a46fa2a297 feat: add structured insert candidate and region analysis 2026-06-07 16:09:35 +08:00
Research Assistant
a8f2a9fc50 feat: add OCR body spine quality diagnostics 2026-06-07 16:07:31 +08:00
Research Assistant
46eb6b70ee feat: derive body baselines from stable middle pages 2026-06-07 16:03:07 +08:00
Research Assistant
a7d1c1176f fix: anchor OCR frontmatter to source metadata 2026-06-07 15:58:38 +08:00
Research Assistant
f10d244631 fix: protect page-1 paper titles from non_body_insert suppression 2026-06-07 12:53:24 +08:00
Research Assistant
ee5cd0df4e fix: render section_heading with Markdown heading prefix 2026-06-07 12:53:04 +08:00
Research Assistant
088fb4144d fix: prevent article-type running headers from becoming section headings 2026-06-07 12:52:02 +08:00
Research Assistant
644f14dad6 fix: gate non-body insert detection by spine reliability 2026-06-07 12:00:18 +08:00
Research Assistant
a723993bb9 feat: add explicit no-span degraded OCR mode 2026-06-07 11:48:58 +08:00
Research Assistant
10fe2251b5 feat: recover source PDFs for old-paper span backfill 2026-06-07 11:47:09 +08:00
Research Assistant
2404a486f0 feat: expose OCR body spine quality and anchor diagnostics 2026-06-07 11:45:44 +08:00
Research Assistant
6bdaddb366 fix: prevent author bylines from becoming section headings 2026-06-07 11:45:11 +08:00
Research Assistant
dcc07140b0 feat: anchor OCR body baselines from stable middle pages 2026-06-07 11:43:36 +08:00
Research Assistant
822895d77e fix: remove redundant figure caption prose check on resolved captions 2026-06-07 01:04:15 +08:00
Research Assistant
0fbd4e186c fix: extend render fallback normalization guards to all tail and candidate roles 2026-06-07 01:02:06 +08:00
Research Assistant
3e748dcb5d fix: gate weak boundary signals through backmatter_boundary_candidate 2026-06-07 00:59:22 +08:00
Research Assistant
32c87a903d test: add tail reading order stabilization regression test 2026-06-06 19:37:08 +08:00
Research Assistant
45768a6d5f fix: restore figure pipeline, fix Funding suppression, tighten candidate resolution 2026-06-06 19:11:51 +08:00
Research Assistant
fe1d9467ca fix: make figure inventory distrust bad legend roles with prose narrative check 2026-06-06 18:50:47 +08:00
Research Assistant
9bfd4354ca fix: add candidate resolution and activation gates to document normalization 2026-06-06 18:46:20 +08:00
Research Assistant
e071a0b6f5 fix: demote ambiguous seed roles to candidates (figure_caption, backmatter_heading) 2026-06-06 18:40:24 +08:00
Research Assistant
5c7e439696 fix: render mixed tail pages with layout-aware reference zones 2026-06-06 17:57:55 +08:00
Research Assistant
5cbf6ae066 fix: make reference rescue depend on local reference zones 2026-06-06 17:39:15 +08:00
Research Assistant
32e4a67f8d refactor: add local reference zones and layout-aware tail boundary detection 2026-06-06 17:35:08 +08:00
Research Assistant
9f0bc39692 feat: derive layout-aware tail reading segments 2026-06-06 17:28:03 +08:00
Research Assistant
2c7ab8aa62 feat: persist layout-aware OCR tail analysis artifacts 2026-06-06 17:25:47 +08:00