Commit graph

38 commits

Author SHA1 Message Date
Research Assistant
d7974867cc fix: prevent frontmatter_noise rescue in page footer zone 2026-06-07 22:31:05 +08:00
Research Assistant
463cd4d7b1 fix: references, figure matching, and OCR structural fixes
- Fix references zone detection without heading (backward reference fallback)
- Fix reference_item silently dropped in _reorder_tail_run_fifo when no heading
- Fix figure proximity matching: use nearest legend regardless of column order
- Fix figure numbering in extract_and_write_objects: use actual figure_id
- Fix body→reference_item rescue: require text starts with numeric ref pattern
- Fix tail reading order rebuilt after block normalization
- A8E7SRVS: corrected figure-legend pairing on page 5
- CAQNW9Q2: references now render without heading
- K7R8PEKW: adjusted body retention threshold
2026-06-07 22:27:58 +08:00
Research Assistant
f9dff20cd8 feat: OCR complex layout and sidebar detection overhaul
- Add RegionPrepass data model for frontmatter/structured_insert/body classification
- Add PDF visual container detection (filled rects) for sidebar detection
- Add Box N and keyword-based sidebar anchor detection
- Add column-aware block expansion for mixed sidebar clusters
- Add _container_text from PDF text blocks for cross-column truncation
- Add degraded-mode heading level normalization with font size clustering
- Merge adjacent structured_insert blocks into single callout
- Fix metadata enrichment from Literature-hub note frontmatter for old papers
- Fix heading hierarchy using span font-size clusters when available
- Add real-paper regression harness (TSCKAVIS, CAQNW9Q2, A8E7SRVS, K7R8PEKW)
- Add body retention, heading, metadata, and callout acceptance tests
- 219 passed, 4 control guards passed
2026-06-07 21:35:57 +08:00
Research Assistant
fa418b218f feat: add final OCR PDF layout consistency audit 2026-06-07 16:20:04 +08:00
Research Assistant
ef73b3f0a1 feat: add explicit OCR no-span degraded mode 2026-06-07 16:13:09 +08:00
Research Assistant
a8f2a9fc50 feat: add OCR body spine quality diagnostics 2026-06-07 16:07:31 +08:00
Research Assistant
46eb6b70ee feat: derive body baselines from stable middle pages 2026-06-07 16:03:07 +08:00
Research Assistant
f10d244631 fix: protect page-1 paper titles from non_body_insert suppression 2026-06-07 12:53:24 +08:00
Research Assistant
644f14dad6 fix: gate non-body insert detection by spine reliability 2026-06-07 12:00:18 +08:00
Research Assistant
a723993bb9 feat: add explicit no-span degraded OCR mode 2026-06-07 11:48:58 +08:00
Research Assistant
2404a486f0 feat: expose OCR body spine quality and anchor diagnostics 2026-06-07 11:45:44 +08:00
Research Assistant
dcc07140b0 feat: anchor OCR body baselines from stable middle pages 2026-06-07 11:43:36 +08:00
Research Assistant
ab2f581ae9 fix: complete container backmatter detection with backmatter_boundary_candidate support and multi-column container child promotion 2026-06-07 01:28:26 +08:00
Research Assistant
0ac980ebbe fix: remove dead _verified_by_seed guard from reference demotion 2026-06-07 01:00:22 +08:00
Research Assistant
3e748dcb5d fix: gate weak boundary signals through backmatter_boundary_candidate 2026-06-07 00:59:22 +08:00
Research Assistant
ab24f19b91 fix: figure_id missing from matched figures, subfigure narrative hard reject 2026-06-06 19:31:44 +08:00
Research Assistant
b017e78828 fix: proper Funding handling in container backmatter, fix single-column tail reorder 2026-06-06 19:25:55 +08:00
Research Assistant
45768a6d5f fix: restore figure pipeline, fix Funding suppression, tighten candidate resolution 2026-06-06 19:11:51 +08:00
Research Assistant
f2e62c5773 fix: include figure_caption_candidate in non_body_insert detection 2026-06-06 18:54:03 +08:00
Research Assistant
9bfd4354ca fix: add candidate resolution and activation gates to document normalization 2026-06-06 18:46:20 +08:00
Research Assistant
5cbf6ae066 fix: make reference rescue depend on local reference zones 2026-06-06 17:39:15 +08:00
Research Assistant
32e4a67f8d refactor: add local reference zones and layout-aware tail boundary detection 2026-06-06 17:35:08 +08:00
Research Assistant
9f0bc39692 feat: derive layout-aware tail reading segments 2026-06-06 17:28:03 +08:00
Research Assistant
c21f04d7b6 feat: add page layout profiling for OCR document analysis 2026-06-06 17:23:41 +08:00
Research Assistant
cdf7deffc9 fix: catch body-width continuation fragments in non_body_insert detection
Block 35 in SAN9AYVR was a biography continuation fragment that escaped
non_body_insert detection because it shared the same width and font as
the body spine.  Added a second-pass expansion in
_detect_non_body_insert_clusters that catches orphan continuation
fragments (lowercase start, same font as insert cluster, adjacent to
detected insert) and marks them as non_body_insert.

Regression test: test_non_body_insert_catches_continuation_fragment
2026-06-06 17:14:43 +08:00
Research Assistant
4ecdee5ff6 fix: close residual non-body insert leakage from figure_caption blocks 2026-06-06 16:50:49 +08:00
Research Assistant
e41895d4d6 fix: add geometry guard to non_body_insert rescue — narrow blocks (<35% page width) stay non_body_insert regardless of font match; require strong non_body_insert_family quality for reinstatement 2026-06-06 16:22:46 +08:00
Research Assistant
0175fc3ab3 fix: tighten non_body_insert detection and rescue — exclude frontmatter_noise from role filter, require moderate+ family quality for reinstatement 2026-06-06 16:15:38 +08:00
Research Assistant
dcfbe8bb64 fix: exclude non-body insert media from figure asset pool 2026-06-06 16:06:36 +08:00
Research Assistant
69a1156449 fix: expand non_body_insert detection to catch frontmatter_noise blocks; use relative page bound
- Expand role filter in _detect_non_body_insert_clusters from body_paragraph
  only to include frontmatter_noise and unknown_structural
- Replace absolute page<=3 gate with relative bound based on body_end_page:
  max_early_page = min(3, body_end_page/4 + 1)
- Added test_detect_non_body_insert_catches_frontmatter_noise_blocks
2026-06-06 15:56:14 +08:00
Research Assistant
3dc06eb9db feat: derive section family profiles from normalized OCR structure 2026-06-06 15:49:50 +08:00
Research Assistant
562ae5ca93 fix: author anchor & non_body_insert — remove open-ended authors, add Zotero anchor, font-family cluster detection, furniture/received/published word-fix, affiliation keyword cleanup 2026-06-06 14:08:32 +08:00
Research Assistant
f40548e8a1 feat: enhance body spine detection with font-family collection from span metadata 2026-06-06 13:51:16 +08:00
Research Assistant
fd00a5add2 fix: guard rescue from promoting non_body_insert blocks to body roles 2026-06-06 13:50:24 +08:00
Research Assistant
b4951597e7 feat: add non_body_insert role for early-page non-body insert clusters 2026-06-06 13:48:30 +08:00
Research Assistant
674809a2ae refactor: converge structure truth — normalize before profile build, render consumes structure 2026-06-06 13:08:42 +08:00
Research Assistant
6bbe00204c feat: add section-aware OCR rescue pass 2026-06-06 12:34:53 +08:00
Research Assistant
e03a50ea53 refactor: extract OCR document structure analysis into ocr_document.py 2026-06-06 12:30:44 +08:00