Commit graph

1228 commits

Author SHA1 Message Date
Research Assistant
08288ff3eb feat: switch OCR to anchor-first role authority 2026-06-09 22:10:54 +08:00
Research Assistant
5e47a70998 feat: wire OCR anchor artifacts 2026-06-09 21:56:23 +08:00
Research Assistant
e833f3e652 feat: unify OCR decision statuses 2026-06-09 21:45:51 +08:00
Research Assistant
481874f489 feat: validate OCR table matching 2026-06-09 21:36:43 +08:00
Research Assistant
63b56689c6 feat: tighten OCR figure validation matching 2026-06-09 21:13:53 +08:00
Research Assistant
6aed760a91 feat: add late OCR role resolution 2026-06-09 19:12:37 +08:00
Research Assistant
99e4bc349c feat: partition OCR families inside zones 2026-06-09 17:49:52 +08:00
Research Assistant
7f8a34c281 feat: add OCR region bus inference 2026-06-09 16:55:33 +08:00
Research Assistant
9dc32a5c98 feat: anchor OCR reference families 2026-06-09 16:19:53 +08:00
Research Assistant
d2ae65d97a feat: discover OCR body family anchor 2026-06-09 14:54:47 +08:00
Research Assistant
e320535894 feat: promote source-backed frontmatter anchors 2026-06-09 14:33:54 +08:00
Research Assistant
6263c235d0 feat: add OCR structural signatures 2026-06-09 14:00:22 +08:00
Research Assistant
eb3b73ae78 docs: prune superseded OCR worktree plans 2026-06-09 12:55:05 +08:00
Research Assistant
6d4ecc3717 fix: skip single-block unresolved clusters — lonely subpanel assets are not clusters 2026-06-08 20:01:53 +08:00
Research Assistant
6b9b3c4d32 fix: remove backmatter heading suppression, improve figure dedup strategy
- Remove _suppressed_heading=True for backmatter heading candidates treated
  as section_heading — Conflict of Interest/Acknowledgments now render normally
- Figure dedup: when both copies are on pages without assets, keep the later
  one (figure compilation sections are always at end)
- Figure matching: add column-based secondary verification for ambiguous close candidates
- Figure matching: add sequential fallback when legends and assets align
2026-06-08 20:00:50 +08:00
Research Assistant
1ca2f7e509 fix: suppress Journal Pre-proof running header on ALL pages, not just page 1
Running headers at y<6% page height with pre-proof text are now
frontmatter_noise regardless of page number or raw_label, preventing
PaddleOCR mislabeling header as paragraph_title from inserting
page-furniture text into the body heading hierarchy
2026-06-08 19:38:31 +08:00
Research Assistant
abe7c6744a fix: suppress entire pre-proof page, broaden highlights detection
- Pre-proof page: all page-1 blocks marked frontmatter_noise when pre-proof detected
- Rescue exempts pre-proof pages from body_paragraph promotion
- Highlights: text in {'highlights'} changed to text.startswith('highlights')
  so 'Highlights bullet We propose...' is no longer missed
2026-06-08 19:27:22 +08:00
Research Assistant
a2b74b183f fix: resolve_metadata skips single-author lists not confirmed by zotero source 2026-06-08 19:16:58 +08:00
Research Assistant
1e310ee351 fix: pre-proof marker suppression with page-1 context guard, rescue exemption, title fallback
- _PREPROOF_MARKER suppression now guarded by page==1, y_top>8%, raw_label=paragraph_title, not_header, not_footer
- _is_preproof_marker() exported for health use
- rescue_roles_with_document_context: pre-proof frontmatter_noise blocks are exempt from body_paragraph rescue
- Page-1 title fallback: when pre-proof steals the title zone, next substantial text block becomes paper_title
- Tests: running header not suppressed, page-2 not suppressed, variants pass
2026-06-08 19:15:36 +08:00
Research Assistant
b33ac572e4 docs: add real-paper audit report — pre-proof, author matching, inner text, ambiguous figures 2026-06-08 18:47:26 +08:00
Research Assistant
9b398b8138 fix: final polish — inline figure mention detection, layout audit error gating
- Inline figure mention: _looks_like_inline_figure_mention() detects body prose
  such as 'Figure X shows/illustrates/demonstrates' and 'as shown in Figure X'
  to prevent noise in figure caption pipeline. Excludes Frontiers FIGURE N | format.
- Layout audit: added safety net so pages with layout confidence < 0.7 never
  produce ERROR-level anomalies (only INFO). Check 2 (insert_body_overlap) also
  gated by layout confidence.
- DWQQK2YB matched figure count dropped 6->3 as inline mentions correctly excluded;
  M36WA39N Frontiers captions unaffected
2026-06-08 17:49:03 +08:00
Research Assistant
f2eb35b96e fix: 5 root-cause fixes from real-paper rebuild — Frontiers figure seed, panel label exclusion, page 1 heading override, layout audit severity, first_author fallback
- Frontiers figure title: _FRONTIERS_FIGURE_TITLE_PATTERN seeds FIGURE N | Title
  as figure_caption even when raw_label=text (Frontiers/journals style)
- Panel label exclusion: _PANEL_LABEL_PATTERN sends A/B/C/(D)/E. to
  figure_inner_text before they enter the caption/matching pipeline
- Page 1 heading override: _explicit_scholarly_heading_role() detects Roman
  numeral (I. INTRODUCTION) and alpha (A. Materials) headings with y_top
  guard to protect frontmatter titles
- Layout audit severity: _run_layout_audit() separates anomalies into
  info/warning/error; full-width headings exempted; health only degraded
  when error_count > 0
- first_author fallback: _enrich_meta_from_paper_note() falls back to
  first_author when authors field is missing; _normalize_author_name()
  strips {superscript} markers for fuzzy matching
2026-06-08 17:10:51 +08:00
Research Assistant
2d0e5971cd fix: second polish — previous-page table scoring, figure confidence wiring, health hard-rule summary
- previous-page table: score_table_match() now adds previous_page evidence
  with caption_near_top/asset_near_previous_page_bottom, and asset_below_caption
  is gated to same-page only
- figure confidence: matched_figures.confidence now reads match_score.score
  instead of hardcoded 0.85/0.4 binary; match_score dict always present
- health: hard_rule_decision_count derived from decision log signals
  (structured_insert_decision + tail_promotion + candidate_resolution +
  candidate_forced) instead of fixed 0
2026-06-08 16:05:50 +08:00
Research Assistant
0aab860cf2 feat: OCR risk convergence — scored figure/table matching, layout confidence, insert scorer, health uncertainty metrics
- Figure matching: nearest-asset → scored multi-candidate selection with ambiguous state
- Table matching: vertical-nearest → score-based selection with prev/next page window
- Layout confidence: PageLayoutProfile gains confidence/evidence; eligible-body block filter
- Tail reorder: guarded by layout + tail confidence (low confidence preserves raw order)
- Structured insert: direct promotion → scorer + candidate + fallback state
- Health: adds 14 uncertainty metrics (ambiguous matches, unresolved clusters, forced inserts, low tail confidence)
- Audit: docs/ocr/high-risk-rule-audit.md documents 8 hard-rule decisions mapped to plans
- Spec + 6 implementation plans for agentic execution
2026-06-08 13:43:48 +08:00
Research Assistant
4dd9ec4691 fix: seal OCR state machine — settled statuses prevent re-entry, bare except replaced with error taxonomy 2026-06-08 12:45:43 +08:00
Research Assistant
05daf752ea fix: P0 exception order, status sync, retryable transition, P1 scorer warnings 2026-06-08 12:40:57 +08:00
Research Assistant
84112c3817 feat: classify OCR errors and degraded completions 2026-06-08 12:34:44 +08:00
Research Assistant
dbab9c5ddf feat: add confidence distributions to OCR health report 2026-06-08 12:27:40 +08:00
Research Assistant
c8b4846998 feat: add tail boundary score to DocumentStructure and health 2026-06-08 12:27:12 +08:00
Research Assistant
46937f2a0a feat: thread table match scores into table inventory 2026-06-08 12:25:55 +08:00
Research Assistant
944ef640d6 feat: thread figure caption scores into inventory evidence 2026-06-08 12:25:20 +08:00
Research Assistant
b8473dac47 feat: add OCR evidence scorers (figure_caption, table_match, tail_boundary) 2026-06-08 12:24:14 +08:00
Research Assistant
38527cf5d6 feat: add OCR decision log and health summaries 2026-06-08 12:22:41 +08:00
Research Assistant
04c942c8db chore: remove root debug scripts, relocate rebuild_verify to scripts/dev 2026-06-08 12:15:17 +08:00
Research Assistant
c036ef7670 fix(ocr): align object filenames with unresolved cluster IDs 2026-06-08 12:14:15 +08:00
Research Assistant
acd28d321e test: make OCR real-paper regressions reproducible 2026-06-08 12:13:11 +08:00
Research Assistant
a3c8dab389 docs: audit OCR production route and mark experimental modules 2026-06-08 12:13:07 +08:00
Research Assistant
64915aa3f2 fix(ocr): prevent empty wikilinks for unresolved clusters in renderer 2026-06-08 12:13:02 +08:00
Research Assistant
b7ca334aca fix(ocr): add stable cluster_id to unresolved clusters in figure inventory 2026-06-08 12:11:54 +08:00
Research Assistant
e39b82b99a fix: close structural convergence gaps — primary-path frontmatter anchor, health spine alignment, unresolved figure clusters, artifact cache cleanup 2026-06-08 00:24:54 +08:00
Research Assistant
c205ae1fd1 fix: render table captions inline with embeds 2026-06-07 23:06:09 +08:00
Research Assistant
d7974867cc fix: prevent frontmatter_noise rescue in page footer zone 2026-06-07 22:31:05 +08:00
Research Assistant
463cd4d7b1 fix: references, figure matching, and OCR structural fixes
- Fix references zone detection without heading (backward reference fallback)
- Fix reference_item silently dropped in _reorder_tail_run_fifo when no heading
- Fix figure proximity matching: use nearest legend regardless of column order
- Fix figure numbering in extract_and_write_objects: use actual figure_id
- Fix body→reference_item rescue: require text starts with numeric ref pattern
- Fix tail reading order rebuilt after block normalization
- A8E7SRVS: corrected figure-legend pairing on page 5
- CAQNW9Q2: references now render without heading
- K7R8PEKW: adjusted body retention threshold
2026-06-07 22:27:58 +08:00
Research Assistant
f9dff20cd8 feat: OCR complex layout and sidebar detection overhaul
- Add RegionPrepass data model for frontmatter/structured_insert/body classification
- Add PDF visual container detection (filled rects) for sidebar detection
- Add Box N and keyword-based sidebar anchor detection
- Add column-aware block expansion for mixed sidebar clusters
- Add _container_text from PDF text blocks for cross-column truncation
- Add degraded-mode heading level normalization with font size clustering
- Merge adjacent structured_insert blocks into single callout
- Fix metadata enrichment from Literature-hub note frontmatter for old papers
- Fix heading hierarchy using span font-size clusters when available
- Add real-paper regression harness (TSCKAVIS, CAQNW9Q2, A8E7SRVS, K7R8PEKW)
- Add body retention, heading, metadata, and callout acceptance tests
- 219 passed, 4 control guards passed
2026-06-07 21:35:57 +08:00
Research Assistant
9d48bf4122 test: preserve figure and tail mainline during frontmatter hardening 2026-06-07 16:23:36 +08:00
Research Assistant
8c49db1b11 fix: gate non-body insert by spine trust and region evidence 2026-06-07 16:23:06 +08:00
Research Assistant
fa418b218f feat: add final OCR PDF layout consistency audit 2026-06-07 16:20:04 +08:00
Research Assistant
ef73b3f0a1 feat: add explicit OCR no-span degraded mode 2026-06-07 16:13:09 +08:00
Research Assistant
a46fa2a297 feat: add structured insert candidate and region analysis 2026-06-07 16:09:35 +08:00
Research Assistant
a8f2a9fc50 feat: add OCR body spine quality diagnostics 2026-06-07 16:07:31 +08:00