Research Assistant
32e4a67f8d
refactor: add local reference zones and layout-aware tail boundary detection
2026-06-06 17:35:08 +08:00
Research Assistant
9f0bc39692
feat: derive layout-aware tail reading segments
2026-06-06 17:28:03 +08:00
Research Assistant
2c7ab8aa62
feat: persist layout-aware OCR tail analysis artifacts
2026-06-06 17:25:47 +08:00
Research Assistant
c21f04d7b6
feat: add page layout profiling for OCR document analysis
2026-06-06 17:23:41 +08:00
Research Assistant
c56ce87a1c
fix: add relational-operator spacing and Greek-letter preservation to math normalization
2026-06-06 17:19:33 +08:00
Research Assistant
cdf7deffc9
fix: catch body-width continuation fragments in non_body_insert detection
...
Block 35 in SAN9AYVR was a biography continuation fragment that escaped
non_body_insert detection because it shared the same width and font as
the body spine. Added a second-pass expansion in
_detect_non_body_insert_clusters that catches orphan continuation
fragments (lowercase start, same font as insert cluster, adjacent to
detected insert) and marks them as non_body_insert.
Regression test: test_non_body_insert_catches_continuation_fragment
2026-06-06 17:14:43 +08:00
Research Assistant
06d7abb2fa
feat: unify OCR math normalization across render and objects
2026-06-06 17:02:01 +08:00
Research Assistant
4ecdee5ff6
fix: close residual non-body insert leakage from figure_caption blocks
2026-06-06 16:50:49 +08:00
Research Assistant
e41895d4d6
fix: add geometry guard to non_body_insert rescue — narrow blocks (<35% page width) stay non_body_insert regardless of font match; require strong non_body_insert_family quality for reinstatement
2026-06-06 16:22:46 +08:00
Research Assistant
0175fc3ab3
fix: tighten non_body_insert detection and rescue — exclude frontmatter_noise from role filter, require moderate+ family quality for reinstatement
2026-06-06 16:15:38 +08:00
Research Assistant
463852462f
fix: stop legend-only figures from using fake unmatched assets
2026-06-06 16:08:26 +08:00
Research Assistant
dcfbe8bb64
fix: exclude non-body insert media from figure asset pool
2026-06-06 16:06:36 +08:00
Research Assistant
cb3f459ba2
fix: prioritize formal figure legends over panel subcaptions
2026-06-06 16:01:57 +08:00
Research Assistant
69a1156449
fix: expand non_body_insert detection to catch frontmatter_noise blocks; use relative page bound
...
- Expand role filter in _detect_non_body_insert_clusters from body_paragraph
only to include frontmatter_noise and unknown_structural
- Replace absolute page<=3 gate with relative bound based on body_end_page:
max_early_page = min(3, body_end_page/4 + 1)
- Added test_detect_non_body_insert_catches_frontmatter_noise_blocks
2026-06-06 15:56:14 +08:00
Research Assistant
67bd36c997
refactor: make document structure the single render truth
...
- build_structured_blocks now returns (rows, doc_structure) tuple
- ocr.py and ocr_rebuild.py pass document_structure to render_fulltext_markdown
- bare except Exception replaced with structured logging
2026-06-06 15:52:18 +08:00
Research Assistant
3dc06eb9db
feat: derive section family profiles from normalized OCR structure
2026-06-06 15:49:50 +08:00
Research Assistant
562ae5ca93
fix: author anchor & non_body_insert — remove open-ended authors, add Zotero anchor, font-family cluster detection, furniture/received/published word-fix, affiliation keyword cleanup
2026-06-06 14:08:32 +08:00
Research Assistant
1e10f9edd6
fix: add non_body_insert guard in frontmatter candidate extraction
2026-06-06 13:51:51 +08:00
Research Assistant
f40548e8a1
feat: enhance body spine detection with font-family collection from span metadata
2026-06-06 13:51:16 +08:00
Research Assistant
fd00a5add2
fix: guard rescue from promoting non_body_insert blocks to body roles
2026-06-06 13:50:24 +08:00
Research Assistant
b4951597e7
feat: add non_body_insert role for early-page non-body insert clusters
2026-06-06 13:48:30 +08:00
Research Assistant
936902a7d6
feat: add Zotero-anchored author alignment check in resolve_metadata
2026-06-06 13:33:20 +08:00
Research Assistant
c974ba7480
fix: remove open-ended authors text heuristics, anchor to page-1 zone only
2026-06-06 13:28:46 +08:00
Research Assistant
2bf7e73fff
feat: persist document_structure.json, thread structure truth through downstream pipeline
2026-06-06 13:10:56 +08:00
Research Assistant
674809a2ae
refactor: converge structure truth — normalize before profile build, render consumes structure
2026-06-06 13:08:42 +08:00
Research Assistant
5566b2f992
fix: map OCR bbox coordinates to PDF space before span extraction
2026-06-06 12:59:06 +08:00
Research Assistant
6bbe00204c
feat: add section-aware OCR rescue pass
2026-06-06 12:34:53 +08:00
Research Assistant
e03a50ea53
refactor: extract OCR document structure analysis into ocr_document.py
2026-06-06 12:30:44 +08:00
Research Assistant
43087401c1
docs: mark completed tasks in document analysis plan, keep only delta tasks
2026-06-06 12:23:08 +08:00
Research Assistant
2122450c8d
fix: use page role distribution (not hard page==1) to gate frontmatter_noise rules
2026-06-06 11:57:37 +08:00
Research Assistant
0258f0d66e
fix: restrict frontmatter_noise text-based rules to page 1 only, prevent body text misclassification
2026-06-06 11:53:57 +08:00
Research Assistant
fdc1cdf301
feat: extract span_metadata from source PDF via PyMuPDF, wire into both pipelines
2026-06-06 11:44:14 +08:00
Research Assistant
be818f4deb
fix: revert PaddleOCR span_metadata passthrough — source is PDF, not OCR engine
2026-06-06 11:42:40 +08:00
Research Assistant
687332d126
docs: correct span_metadata source to PDF (not PaddleOCR), revise plan
2026-06-06 11:42:14 +08:00
Research Assistant
118f643951
feat: second-pass cross-validation for low-confidence role assignments using span profiles
2026-06-06 11:27:44 +08:00
Research Assistant
187fd53efe
fix: consolidate duplicate is_visually_heading computation in _is_backmatter_boundary_heading
2026-06-06 11:24:52 +08:00
Research Assistant
9006807127
feat: dynamic heading family discovery with profile matching, remove hardcoded font-size thresholds and absolute page gate
2026-06-06 11:21:25 +08:00
Research Assistant
94de2dbbc7
feat: persist role_span_profiles.json and wire into rebuild pipeline
2026-06-06 11:16:43 +08:00
Research Assistant
5320e825ee
fix: remove dead constants, tighten profile quality test assertion
2026-06-06 11:15:41 +08:00
Research Assistant
2604a1c5cf
feat: create ocr_profiles.py with span extraction, profile aggregation, and cross-validation
2026-06-06 11:13:01 +08:00
Research Assistant
f67c7d5ac0
feat: carry span_metadata through raw and structured block pipeline
2026-06-06 11:08:44 +08:00
Research Assistant
bdb3eae8ac
docs: add unified span metadata spec and integration plan
2026-06-06 11:06:39 +08:00
Research Assistant
7c278fbc60
fix(ocr): properly cluster multi-panel figures — role assignment, cluster bbox crop, render visibility
2026-06-06 10:08:42 +08:00
Research Assistant
c235dbdc68
feat(ocr): journal layout generalization, OCR diagnostics, and debug scripts
2026-06-06 00:46:49 +08:00
Research Assistant
f9b5fb0541
feat(ocr): journal layout generalization — structured boundary, container regime, legend validation
...
- Task 1: TailBoundary namedtuple with body_end/backmatter_start/references_start
- Task 2: _classify_backmatter_form — 'container' (PeerJ) vs 'flat' (Frontiers) regime
- Task 3: _attach_container_bodies, _container_child flag for ### child headings
- Task 4: Style-family classification already complete
- Task 6: _is_formal_legend threshold raised to 100, more axis_words added,
low-confidence fallback with legend_uncertain flag for rejected captions
- 2GN9LMCW: authors correct, furniture suppressed, heading hierarchy,
container child headings, Figure 4 uncertain
- 7C8829BD: 26/26 markers, correct tail order, no regression
- 296/296 tests pass
2026-06-05 23:48:56 +08:00
Research Assistant
8a0f6dd0fb
feat(ocr): PeerJ-style remediation — frontmatter zoning, heading hierarchy, backmatter container
...
- Task 1: Strengthen first-page frontmatter zoning (author/affiliation/furniture zones)
- Task 2: Style-aware heading hierarchy with _infer_heading_level content heuristics
- Tasks 3+4: ADDITIONAL INFORMATION AND DECLARATIONS as backmatter_boundary_heading,
container ordering (boundary heading emitted first, sub-sections after)
- Task 5: Figure legend validation via _is_formal_legend (width/criterion/axis-vocab rejection)
- Verified on 2GN9LMCW: authors correct, furniture suppressed, heading hierarchy,
references last
- 7C8829BD regression-free: 26/26 markers, tail order intact
- 296/296 tests pass
2026-06-05 23:27:10 +08:00
Research Assistant
d52ea37e92
feat(ocr): tail regime remediation + style-aware heading profiles + boundary detection
...
- Task: Bidirectional body/backmatter boundary detection (forward/backward spine + tail spread reconciliation)
- Task: Style-aware heading profiles from PDF span_metadata (extract, cluster, disambiguate)
- Task: Lock tests for tail-candidate overreach, cross-page continuation, style-aware heading detection
- 296/296 tests pass, real-paper 7C8829BD verified
2026-06-05 22:47:13 +08:00
Research Assistant
2e2e8cb8e4
Revert "revert: restore original Phase 2 figure/table matching logic"
...
This reverts commit 0a2988b89f
2026-06-05 15:38:04 +08:00
Research Assistant
0a2988b89f
revert: restore original Phase 2 figure/table matching logic
2026-06-05 15:36:59 +08:00
Research Assistant
ac3c0c4b8e
fix: resolve source_pdf during legacy backfill for figure/table crop extraction
2026-06-05 15:29:55 +08:00
