obsidian-plugin/lllin000_PaperForge
1
0
Fork 0
mirror of https://github.com/lllin000/PaperForge.git synced 2026-07-22 17:00:23 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
1617 commits 4 branches 19 tags 51 MiB
Commit graph

93 commits

Author SHA1 Message Date
LLLin000
8dba963394 fix: complete reference detection fixes — 5 problem families 2026-07-03 01:46:27 +08:00
LLLin000
4afe32b6d5 docs: patch layout robustness execution gaps 2026-07-02 19:02:42 +08:00
LLLin000
e7a809f291 docs: tighten layout robustness implementation plan 2026-07-02 18:59:05 +08:00
LLLin000
33fa5e1973 docs: refine layout robustness design contracts 2026-07-02 18:46:05 +08:00
LLLin000
a6187f7135 docs: add layout robustness layer design spec 2026-07-02 18:38:47 +08:00
LLLin000
1791edd7cd docs: record round 2 truth audit + 3 bug fixes for 37LK5T97 2026-07-02 15:02:42 +08:00
LLLin000
5af3a1f8ca feat: asset-internal figure number recovery for rotated vector figures 
adds a metadata-only recovery pass that scans matched figure assets for
internal PDF line labels ("Figure N.", "Fig. N:") when the figure lacks
a figure_number.

new components:
- extract_pdf_lines_normalized() in ocr_pdf_spans.py — reads PDF rawdict
  lines page-by-page, normalizes coordinates to OCR space
- _recover_missing_figure_numbers_from_assets() — inventory pass between
  synthetic fallback and dedup
- _needs_asset_internal_figure_number_recovery() — gate supporting both
  synthetic_figure and figure_unknown entries
- _looks_like_internal_figure_label() — regex-based label detection
- _asset_edge_band_score() — geometric rejection for center-positioned
  lines or lines covering >15% asset area
- page_pdf_lines_by_page parameter added to build_figure_inventory()

verification: U746UJ7G figure_unknown_000 -> figure_002 with recovered
label "Plot of Criteria Time". 428 regression tests pass (6 new).
 2026-07-02 03:00:33 +08:00
LLLin000
01ba102605 fix: support Roman numeral and S-prefix table captions across roles/signatures/tables 
- _TABLE_PREFIX_PATTERN in ocr_roles.py, ocr_signatures.py, ocr_tables.py
  now matches Table I/II/III and Table S1/S2
- _extract_table_number and _extract_marker_signature parse Roman numbers
- figure_title generic fallback checks table prefix before figure caption
- ocr_families.py adds early table-prefix guard before reference family
 2026-07-02 00:47:18 +08:00
Research Assistant
1a9ad1d147 P1: backmatter boundary redesign — ref-anchored partition replaces second _reconcile_tail_spread 
reference_zone is the only hard boundary. Pre-ref disclosure headings (CRediT,
Ethics, Declaration) are normalized as local same-column runs — they never set
global boundaries. Four contracts enforced (Contract A-D in spec).

- 13 new helpers in ocr_document.py: _has_verified_reference_zone, _partition_by_reference_zone,
  _classify_same_page_block, _normalize_reference_roles_from_partition,
  _normalize_pre_ref_disclosure_runs, _build_tail_boundary_from_ref_partition, etc.
- infer_zones() stores effective_end_page for trimmed reference extent
- normalize_document_structure(): ref-partitioned path replaces second _reconcile_tail_spread;
  _promote_tail_body_candidates/_assign_tail_spread_ownership skipped when active
- 20 tests in test_backmatter_boundary.py
- 322/322 tests pass
 2026-06-28 01:13:54 +08:00
Research Assistant
b1fdbc30eb feat: P0 fixes (ref sort, caption insert) + P1 figure containment render hygiene 
P0:
- Fix 2: _ref_number_sort_key handles [N] bracket format (ocr_render.py:481)
- Fix 4: remove figure_caption from _INSERT_CANDIDATE_ROLES (ocr_document.py:3846)
- Fix 5: filter demoted body_paragraph from figure legends (ocr_figures.py:2950)

P1:
- figure containment render-hygiene pass after table inventory writeback
- 6 helpers: _cluster_bboxes_by_proximity, _is_contained,
  _highly_overlaps_any_matched_region, _figure_region_bbox,
  _matched_asset_keys, tag_figure_contained_text
- Wire call site in postprocess_ocr_result() (ocr.py:1891)

Infra:
- Sync workspace fulltext after rebuild (asset_index.py)
- Fix UnboundLocalError on index-refresh (sync_service.py)
- Add missing frontmatter fields to field registry (field_registry.yaml)
- 283 + 19 = 302 tests passing
 2026-06-27 22:56:22 +08:00
Research Assistant
653d0925c6 docs: apply 4 must-fix review changes to container admission spec+plan 
1. _has_container_text() gets pdf_page param for real _map_ocr_bbox_to_pdf_rect()
2. Call site rewritten as per-page three-phase loop (clear/ compute/ mark)
3. page-sized/crop-like features excluded from grouping_pool entirely
4. Merged rects replace children; accepted rects sorted by area descending
+ fix Stage Boundary count (5->9 helpers)
+ add 11th test for non-identity coordinate mapping
+ add _bbox_overlap_ratio zero-area/empty handling
+ add yellow flag note for black-border plot frame risk
 2026-06-26 22:25:23 +08:00
Research Assistant
9b07d68c10 docs: revise figure-number-inference plan with 9 review fixes 
- Add supersede note for spec's old integration wording
- Replace _FIGURE_NUMBER_PATTERN with new _FIGURE_MARKER_PATTERN regex (captures S prefix)
- Add _coerce_int_figure_number() helper (handles int/float/str from JSON)
- Add duplicate_known_main_numbers skip reason
- _FRONTMATTER_VISUAL_VETO: remove bare 'toc' substring risk
- Add _has_frontmatter_visual_veto() helper with word-boundary match
- Noise check: resolvable order key failure strategy
- Stage Boundary: include inventory[figure_number_inference] in mutation scope
- Use _format_figure_id() not literal string
- Add test_infer_duplicate_known_numbers_skips test case
- Frontmatter veto test: veto item must otherwise be eligible
 2026-06-26 20:29:15 +08:00
Research Assistant
c9ca8e2657 docs+fix: evidence-driven container admission + figure-number-inference spec/plan 
_extract_visual_container_rects:
- Replace flat width<100/height<50 gate with evidence-driven admission
- Add line-like veto (<=3pt, pure decorative separators)
- Add page-sized veto (>=60% page + near full edge = background/clip)
- Add vertical component merge (same x-range, gap <=5pt)
- Keep existing fill/border/thin-gray filter logic

spec: 2026-06-26-figure-number-fallback-by-sequence-design
- Fix integration point (after _promote_sequence_matches)
- Add 3-tier legend_bbox lookup
- Add _FRONTMATTER_VISUAL_VETO keyword list
- Add structured _extract_figure_marker() return
- Add skip-reason contract to inventory

plan: 2026-06-26-figure-number-inference-implementation
- 6 tasks, ~95min, single-commit PR
 2026-06-26 20:22:43 +08:00
Research Assistant
b4152abe10 A2: check table ownership (consumed_table_block_keys) before _SKIPPED_BODY_ROLES in render loop 
Moves the block_id fetch and consumed_table_block_keys check before
_SKIPPED_BODY_ROLES so table note removal is a contract-based decision
(ownership) not a role-based accident (footnote role).

Part of table ownership contract refactor.
 2026-06-25 22:27:35 +08:00
Research Assistant
d018a68fd6 fix: harden early OCR frontmatter-side zone inference 2026-06-25 15:57:53 +08:00
Research Assistant
1f7bd743ce fix: close out OCR merge-gate figure blockers 2026-06-24 01:09:38 +08:00
Research Assistant
147f5b77e1 docs: add convergence architecture spec/plans and dense-parent hardening spec/plan 2026-06-23 14:29:18 +08:00
Research Assistant
fb28eca1fe fix: converge figure ownership arbitration and add VFS8CBW2 dense-composite diagnostics 2026-06-23 02:42:22 +08:00
Research Assistant
224882f697 fix: govern local figure pairing and ownership fallbacks 2026-06-22 22:15:39 +08:00
Research Assistant
410151ea8e fix: reuse rebuild metadata and object extraction work 2026-06-22 22:15:25 +08:00
Research Assistant
5bbe40d08b fix: reject weak isolated layout clusters 2026-06-22 22:15:12 +08:00
Research Assistant
6522dc9aef docs: update PROJECT-MANAGEMENT with 2HEUD5P9 rebuild findings 2026-06-22 02:50:37 +08:00
Research Assistant
db69961c7c feat: convert body-zone footnotes with distinct font to callout 2026-06-22 01:51:10 +08:00
Research Assistant
69f6a82b52 docs: update PROJECT-MANAGEMENT with figure merge refactor + heading/backmatter fixes + cover page plan 2026-06-22 01:44:52 +08:00
Research Assistant
4cd0cf3d54 fix: heading detection, backmatter, table consumed, footnote reorder, figure crop + audit fixtures 2026-06-22 01:28:55 +08:00
Research Assistant
da2e018b19 feat: figure merge clustering spec + fixes for caption dedup and structural gate 
- Add global-distance-clustering figure merge design spec
- Fix ocr_roles.py: body_zone body_paragraph no longer promoted to
  figure_caption_candidate (prevents body mentions from entering legend pipeline)
- Fix ocr_document.py: preserve subsection_heading, reference_item,
  table_caption from structural-gate demotion
- Update PROJECT-MANAGEMENT.md with edge-case analysis
- Add ocr-vision-audit-master.md for CODE vs VISION separation
- Update SKILL.md dispatch rules with trust boundaries
 2026-06-21 21:32:30 +08:00
Research Assistant
70a7978de8 feat: complete OCR-v2 readiness gates + blind audit validation 
- Gate 1: completeness signals for page text coverage
- Gate 2: DW figure ownership generalization with strict regression
- Gate 3: reference boundary enforcement from structure
- Gate 4: layout coverage taxonomy in audit/coverage_ledger.json
- diff_audit.py: canonical role alias mapping + auto-normalize truth_role
- ocr_structural_gate.py: table caption text-evidence fallback
- ocr-canonical-roles.md: canonical role list for audit workflow
- ocr-truth-audit.md: pre-flight checklist enforces canonical roles
- Blind audit: 5 unseen papers across domains, all PASS with coverage
 2026-06-19 00:35:21 +08:00
Research Assistant
7e0b69b3ff docs: add OCR real-paper regression design 2026-06-14 21:13:39 +08:00
Research Assistant
123234b93b docs: add OCR-v2 closure gap remediation plan 2026-06-14 11:41:05 +08:00
Research Assistant
711bb3c86a docs: add OCR real-paper regression harness implementation plan 2026-06-13 15:22:45 +08:00
Research Assistant
c65ff39c4d docs: add fixture layout, debug bundle, segment-order, and consumption refinements 2026-06-13 15:20:17 +08:00
Research Assistant
9cb8fb7b04 docs: apply 6 spec refinements for agent execution readiness 2026-06-13 15:16:54 +08:00
Research Assistant
df82820e66 docs: add consumption assertion field to fixture schema 2026-06-13 15:11:45 +08:00
Research Assistant
da00c083e6 docs: add audited OCR-v2 failure families 2026-06-13 15:04:02 +08:00
Research Assistant
e1ecf125ce docs: add OCR production-path regression design 2026-06-13 14:40:15 +08:00
Research Assistant
d9f76917e9 style: ruff format after OCR role gate implementation 2026-06-12 15:11:33 +08:00
Research Assistant
eb3b73ae78 docs: prune superseded OCR worktree plans 2026-06-09 12:55:05 +08:00
Research Assistant
0aab860cf2 feat: OCR risk convergence — scored figure/table matching, layout confidence, insert scorer, health uncertainty metrics 
- Figure matching: nearest-asset → scored multi-candidate selection with ambiguous state
- Table matching: vertical-nearest → score-based selection with prev/next page window
- Layout confidence: PageLayoutProfile gains confidence/evidence; eligible-body block filter
- Tail reorder: guarded by layout + tail confidence (low confidence preserves raw order)
- Structured insert: direct promotion → scorer + candidate + fallback state
- Health: adds 14 uncertainty metrics (ambiguous matches, unresolved clusters, forced inserts, low tail confidence)
- Audit: docs/ocr/high-risk-rule-audit.md documents 8 hard-rule decisions mapped to plans
- Spec + 6 implementation plans for agentic execution
 2026-06-08 13:43:48 +08:00
Research Assistant
c974ba7480 fix: remove open-ended authors text heuristics, anchor to page-1 zone only 2026-06-06 13:28:46 +08:00
Research Assistant
43087401c1 docs: mark completed tasks in document analysis plan, keep only delta tasks 2026-06-06 12:23:08 +08:00
Research Assistant
687332d126 docs: correct span_metadata source to PDF (not PaddleOCR), revise plan 2026-06-06 11:42:14 +08:00
Research Assistant
bdb3eae8ac docs: add unified span metadata spec and integration plan 2026-06-06 11:06:39 +08:00
Research Assistant
7c278fbc60 fix(ocr): properly cluster multi-panel figures — role assignment, cluster bbox crop, render visibility 2026-06-06 10:08:42 +08:00
Research Assistant
c235dbdc68 feat(ocr): journal layout generalization, OCR diagnostics, and debug scripts 2026-06-06 00:46:49 +08:00
Research Assistant
f9b5fb0541 feat(ocr): journal layout generalization — structured boundary, container regime, legend validation 
- Task 1: TailBoundary namedtuple with body_end/backmatter_start/references_start
- Task 2: _classify_backmatter_form — 'container' (PeerJ) vs 'flat' (Frontiers) regime
- Task 3: _attach_container_bodies, _container_child flag for ### child headings
- Task 4: Style-family classification already complete
- Task 6: _is_formal_legend threshold raised to 100, more axis_words added,
  low-confidence fallback with legend_uncertain flag for rejected captions
- 2GN9LMCW: authors correct, furniture suppressed, heading hierarchy,
  container child headings, Figure 4 uncertain
- 7C8829BD: 26/26 markers, correct tail order, no regression
- 296/296 tests pass
 2026-06-05 23:48:56 +08:00
Research Assistant
d52ea37e92 feat(ocr): tail regime remediation + style-aware heading profiles + boundary detection 
- Task: Bidirectional body/backmatter boundary detection (forward/backward spine + tail spread reconciliation)
- Task: Style-aware heading profiles from PDF span_metadata (extract, cluster, disambiguate)
- Task: Lock tests for tail-candidate overreach, cross-page continuation, style-aware heading detection
- 296/296 tests pass, real-paper 7C8829BD verified
 2026-06-05 22:47:13 +08:00
Research Assistant
1779b8b0ff docs: add legacy OCR backfill and sync integration plan 2026-06-05 12:49:01 +08:00
Research Assistant
5f31d2924f docs: add OCR phase4.1 cleanup and phase5 plan 2026-06-05 12:00:22 +08:00
Research Assistant
f3fe86038e docs: add OCR structured pipeline phase4 plan 2026-06-05 10:56:54 +08:00
Research Assistant
1963a091fc docs: add OCR structured pipeline phase3 plan 2026-06-05 01:01:50 +08:00