Commit graph

84 commits

Author SHA1 Message Date
LLLin000
f52124bcf7 docs: retrieval recovery design — Approach A (Python CLI sole owner) recommended 2026-07-10 23:01:57 +08:00
LLLin000
45022c3e16 docs: archive superpowers specs/plans, commit PR9A-C plan files
- Archive 173 design/plan/analysis docs from docs/superpowers/ →
  docs/archive/superpowers/ (all features shipped)
- Commit 5 PR9A-C plan files to docs/plans/ (historical record)
- Rebuild plugin bundle (main.js: 2172→49 lines, minified)
- Fix settings.ts type assertion
2026-07-09 00:39:39 +08:00
LLLin000
288829bb5b feat: PR 1 — nested structure tree + body_units correctness
- RenderOutput dataclass with heading_events/emitted_block_events tracking
- Stack-algorithm structure tree with own/subtree block IDs
- Recursive body_units builder with role helper + token cap splitting
- Schema v4: section_path_json, section_level, section_title, part_ordinal
- Phase 4/5 reorder in ocr_rebuild.py and ocr.py
- 12 new test cases for tree nesting, own_block_ids, rendered-order intervals
2026-07-06 16:42:08 +08:00
LLLin000
c34ebbce81 feat: rebuild speed-up with parallel rebuild, threaded asset cropping, maintenance refactor, embed progress, CLI docs 2026-07-05 19:43:07 +08:00
LLLin000
a21fdf304b docs: tighten rebuild safety spec 2026-07-05 16:10:07 +08:00
LLLin000
35e66197b4 docs: add fulltext rebuild safety design 2026-07-05 16:00:54 +08:00
LLLin000
88838c1842 docs: keep paper lookup at identity boundary 2026-07-05 15:06:23 +08:00
LLLin000
9bc12bd6a0 docs: tighten Layer 4 retrieval spec 2026-07-05 02:22:17 +08:00
LLLin000
f8d928d6ed docs: design Layer 4 retrieval substrate 2026-07-05 02:05:27 +08:00
LLLin000
b4edea8ee6 docs: spec cleanup — remove duplicate old UI, add row model display fields + mapping contract 2026-07-05 00:27:44 +08:00
LLLin000
ac55a519db docs: maintenance tab design spec v2 — action-first labels, legacy handling, cache strategy 2026-07-05 00:24:08 +08:00
LLLin000
b07ac7a177 docs: add Layer 2 spec and implementation plan 2026-07-04 23:32:55 +08:00
LLLin000
fc968f550a docs: update spec with 8 corrections, add implementation plan
8 spec corrections from review:
1. PR 1: field-based return, not lines; insertion before title
2. PR 2: explicit body_paragraph escape in assign_block_role
3. PR 2: caption syntax supports delimiter + title patterns
4. PR 3: use existing ref_heading_block, no new scan
5. PR 3: same_page_tail_blocks + column check, test body rendering
6. PR 4: split None into full_width vs ambiguous_center
7. PR 4: column check before safe_auto_match, no sidecar exception
8. PR 2: fixture verification criteria per-fixture, not blanket
2026-07-04 21:39:50 +08:00
LLLin000
bdf96af6fd docs: four-PR architecture hardening spec
Based on layout truth audit findings (6 bug patterns), designs 4
independent PRs with precise code locations, helper contracts,
column-aware zone inference, caption formalness tightening,
and render frontmatter fallback. Includes before/after Mermaid
diagrams, test specs, and execution order.
2026-07-04 21:34:02 +08:00
LLLin000
26715a9d7e feat: layout-category truth audit complete
Workstream X: 11 papers audited across 6 layout classes,
5 verified via vision subagents. 6 real bug patterns identified
with code locations and root cause analysis.

Bug patterns:
A - _is_obviously_formal_figure_caption heuristic too aggressive
    (ocr_roles.py:193)
B - Cross-column figure asset mis-assignment ignores boundaries
    (ocr_figures.py:704)
C - Render frontmatter skip silently drops authors/affiliations
    (ocr_render.py)
D - Two-column same-page boundary body→backmatter zone pollution
    (ocr_document.py)
E - Frontmatter headings misclassified (ocr_roles.py)
F - Supplementary-only PDFs not handled

Full report: docs/superpowers/analysis/2026-07-05-layout-truth-audit-findings.md

Also: codebase-memory-mcp 0.8.1 installed and repo indexed.
2026-07-04 21:26:15 +08:00
LLLin000
268462c781 docs: sync OCR records after A/B/C merge 2026-07-04 18:00:38 +08:00
LLLin000
3cda942f96 fix: appendix numbering cross-cutting bugs (TABLE regex, int block_id, table alpha prefix, weak-caption tie-break)
3 distinct bugs found and fixed:

1. TABLE in figure regex (_FIGURE_NUMBER_PATTERN)
   Removed TABLE/Table from figure regex — caused table captions to be
   extracted as figure numbers and roadmapped into figure inventory.
   Added unit tests (table_numeric_caption_is_not_a_figure_number,
   table_appendix_caption_marker_has_no_figure_number).

2. int block_id type mismatch in cross-page lookup
   CrossPageSettlementPass and PrimarySamePagePass compared block_id
   with === but deduped_legends use int keys while ResourceRef stores
   str.  Fixed by casting both to str.  Table continuation lookup
   missed page filter, picking wrong page's same block_id.  Added
   page filter + int_block_id tests.

3. Table appendix support gaps
   - Table prefix regex only accepted digits/roman, not 'TABLE A1'.
     Added [A-Z]\d+ token and strip-leading-alpha in parse.
   - _is_validation_first_table_candidate didn't cover figure_title
     raw_label blocks with table_caption_like style.  Added second
     gate.
   - _is_weak_explicit_table_caption only checked table_caption roles.
     Extended to include validation-first candidates.
   - Same-page tie-break didn't apply for weak-explicit captions.
     Ported _bare_table_tie_break into vnext pass.
   - figure_caption_candidate not excluded from note attachment.
   - Continuation merge stripped leading duplicate table marker.

M84CTEM9 vault verification: 6/6 figures matched (3 main + 3
appendix + assets), 4/4 tables (Table 1,2 + Table A1,A2 + assets),
0 false positives, 0 figure asset leakage.
2026-07-04 01:29:06 +08:00
LLLin000
edc8541cd8 docs(ocr): refine pairing framework spec 2026-07-03 19:21:55 +08:00
LLLin000
11355322e6 docs(ocr): add pairing framework design spec 2026-07-03 19:15:15 +08:00
LLLin000
239d44bfe4 feat(ocr): checkpoint layout robustness and figure vnext groundwork on master 2026-07-03 17:48:50 +08:00
LLLin000
8dba963394 fix: complete reference detection fixes — 5 problem families 2026-07-03 01:46:27 +08:00
LLLin000
33fa5e1973 docs: refine layout robustness design contracts 2026-07-02 18:46:05 +08:00
LLLin000
a6187f7135 docs: add layout robustness layer design spec 2026-07-02 18:38:47 +08:00
LLLin000
1791edd7cd docs: record round 2 truth audit + 3 bug fixes for 37LK5T97 2026-07-02 15:02:42 +08:00
LLLin000
01ba102605 fix: support Roman numeral and S-prefix table captions across roles/signatures/tables
- _TABLE_PREFIX_PATTERN in ocr_roles.py, ocr_signatures.py, ocr_tables.py
  now matches Table I/II/III and Table S1/S2
- _extract_table_number and _extract_marker_signature parse Roman numbers
- figure_title generic fallback checks table prefix before figure caption
- ocr_families.py adds early table-prefix guard before reference family
2026-07-02 00:47:18 +08:00
Research Assistant
1a9ad1d147 P1: backmatter boundary redesign — ref-anchored partition replaces second _reconcile_tail_spread
reference_zone is the only hard boundary. Pre-ref disclosure headings (CRediT,
Ethics, Declaration) are normalized as local same-column runs — they never set
global boundaries. Four contracts enforced (Contract A-D in spec).

- 13 new helpers in ocr_document.py: _has_verified_reference_zone, _partition_by_reference_zone,
  _classify_same_page_block, _normalize_reference_roles_from_partition,
  _normalize_pre_ref_disclosure_runs, _build_tail_boundary_from_ref_partition, etc.
- infer_zones() stores effective_end_page for trimmed reference extent
- normalize_document_structure(): ref-partitioned path replaces second _reconcile_tail_spread;
  _promote_tail_body_candidates/_assign_tail_spread_ownership skipped when active
- 20 tests in test_backmatter_boundary.py
- 322/322 tests pass
2026-06-28 01:13:54 +08:00
Research Assistant
b1fdbc30eb feat: P0 fixes (ref sort, caption insert) + P1 figure containment render hygiene
P0:
- Fix 2: _ref_number_sort_key handles [N] bracket format (ocr_render.py:481)
- Fix 4: remove figure_caption from _INSERT_CANDIDATE_ROLES (ocr_document.py:3846)
- Fix 5: filter demoted body_paragraph from figure legends (ocr_figures.py:2950)

P1:
- figure containment render-hygiene pass after table inventory writeback
- 6 helpers: _cluster_bboxes_by_proximity, _is_contained,
  _highly_overlaps_any_matched_region, _figure_region_bbox,
  _matched_asset_keys, tag_figure_contained_text
- Wire call site in postprocess_ocr_result() (ocr.py:1891)

Infra:
- Sync workspace fulltext after rebuild (asset_index.py)
- Fix UnboundLocalError on index-refresh (sync_service.py)
- Add missing frontmatter fields to field registry (field_registry.yaml)
- 283 + 19 = 302 tests passing
2026-06-27 22:56:22 +08:00
Research Assistant
653d0925c6 docs: apply 4 must-fix review changes to container admission spec+plan
1. _has_container_text() gets pdf_page param for real _map_ocr_bbox_to_pdf_rect()
2. Call site rewritten as per-page three-phase loop (clear/ compute/ mark)
3. page-sized/crop-like features excluded from grouping_pool entirely
4. Merged rects replace children; accepted rects sorted by area descending
+ fix Stage Boundary count (5->9 helpers)
+ add 11th test for non-identity coordinate mapping
+ add _bbox_overlap_ratio zero-area/empty handling
+ add yellow flag note for black-border plot frame risk
2026-06-26 22:25:23 +08:00
Research Assistant
c9ca8e2657 docs+fix: evidence-driven container admission + figure-number-inference spec/plan
_extract_visual_container_rects:
- Replace flat width<100/height<50 gate with evidence-driven admission
- Add line-like veto (<=3pt, pure decorative separators)
- Add page-sized veto (>=60% page + near full edge = background/clip)
- Add vertical component merge (same x-range, gap <=5pt)
- Keep existing fill/border/thin-gray filter logic

spec: 2026-06-26-figure-number-fallback-by-sequence-design
- Fix integration point (after _promote_sequence_matches)
- Add 3-tier legend_bbox lookup
- Add _FRONTMATTER_VISUAL_VETO keyword list
- Add structured _extract_figure_marker() return
- Add skip-reason contract to inventory

plan: 2026-06-26-figure-number-inference-implementation
- 6 tasks, ~95min, single-commit PR
2026-06-26 20:22:43 +08:00
Research Assistant
b4152abe10 A2: check table ownership (consumed_table_block_keys) before _SKIPPED_BODY_ROLES in render loop
Moves the block_id fetch and consumed_table_block_keys check before
_SKIPPED_BODY_ROLES so table note removal is a contract-based decision
(ownership) not a role-based accident (footnote role).

Part of table ownership contract refactor.
2026-06-25 22:27:35 +08:00
Research Assistant
d018a68fd6 fix: harden early OCR frontmatter-side zone inference 2026-06-25 15:57:53 +08:00
Research Assistant
147f5b77e1 docs: add convergence architecture spec/plans and dense-parent hardening spec/plan 2026-06-23 14:29:18 +08:00
Research Assistant
fb28eca1fe fix: converge figure ownership arbitration and add VFS8CBW2 dense-composite diagnostics 2026-06-23 02:42:22 +08:00
Research Assistant
224882f697 fix: govern local figure pairing and ownership fallbacks 2026-06-22 22:15:39 +08:00
Research Assistant
410151ea8e fix: reuse rebuild metadata and object extraction work 2026-06-22 22:15:25 +08:00
Research Assistant
6522dc9aef docs: update PROJECT-MANAGEMENT with 2HEUD5P9 rebuild findings 2026-06-22 02:50:37 +08:00
Research Assistant
db69961c7c feat: convert body-zone footnotes with distinct font to callout 2026-06-22 01:51:10 +08:00
Research Assistant
4cd0cf3d54 fix: heading detection, backmatter, table consumed, footnote reorder, figure crop + audit fixtures 2026-06-22 01:28:55 +08:00
Research Assistant
da2e018b19 feat: figure merge clustering spec + fixes for caption dedup and structural gate
- Add global-distance-clustering figure merge design spec
- Fix ocr_roles.py: body_zone body_paragraph no longer promoted to
  figure_caption_candidate (prevents body mentions from entering legend pipeline)
- Fix ocr_document.py: preserve subsection_heading, reference_item,
  table_caption from structural-gate demotion
- Update PROJECT-MANAGEMENT.md with edge-case analysis
- Add ocr-vision-audit-master.md for CODE vs VISION separation
- Update SKILL.md dispatch rules with trust boundaries
2026-06-21 21:32:30 +08:00
Research Assistant
70a7978de8 feat: complete OCR-v2 readiness gates + blind audit validation
- Gate 1: completeness signals for page text coverage
- Gate 2: DW figure ownership generalization with strict regression
- Gate 3: reference boundary enforcement from structure
- Gate 4: layout coverage taxonomy in audit/coverage_ledger.json
- diff_audit.py: canonical role alias mapping + auto-normalize truth_role
- ocr_structural_gate.py: table caption text-evidence fallback
- ocr-canonical-roles.md: canonical role list for audit workflow
- ocr-truth-audit.md: pre-flight checklist enforces canonical roles
- Blind audit: 5 unseen papers across domains, all PASS with coverage
2026-06-19 00:35:21 +08:00
Research Assistant
7e0b69b3ff docs: add OCR real-paper regression design 2026-06-14 21:13:39 +08:00
Research Assistant
c65ff39c4d docs: add fixture layout, debug bundle, segment-order, and consumption refinements 2026-06-13 15:20:17 +08:00
Research Assistant
9cb8fb7b04 docs: apply 6 spec refinements for agent execution readiness 2026-06-13 15:16:54 +08:00
Research Assistant
df82820e66 docs: add consumption assertion field to fixture schema 2026-06-13 15:11:45 +08:00
Research Assistant
da00c083e6 docs: add audited OCR-v2 failure families 2026-06-13 15:04:02 +08:00
Research Assistant
e1ecf125ce docs: add OCR production-path regression design 2026-06-13 14:40:15 +08:00
Research Assistant
d9f76917e9 style: ruff format after OCR role gate implementation 2026-06-12 15:11:33 +08:00
Research Assistant
eb3b73ae78 docs: prune superseded OCR worktree plans 2026-06-09 12:55:05 +08:00
Research Assistant
0aab860cf2 feat: OCR risk convergence — scored figure/table matching, layout confidence, insert scorer, health uncertainty metrics
- Figure matching: nearest-asset → scored multi-candidate selection with ambiguous state
- Table matching: vertical-nearest → score-based selection with prev/next page window
- Layout confidence: PageLayoutProfile gains confidence/evidence; eligible-body block filter
- Tail reorder: guarded by layout + tail confidence (low confidence preserves raw order)
- Structured insert: direct promotion → scorer + candidate + fallback state
- Health: adds 14 uncertainty metrics (ambiguous matches, unresolved clusters, forced inserts, low tail confidence)
- Audit: docs/ocr/high-risk-rule-audit.md documents 8 hard-rule decisions mapped to plans
- Spec + 6 implementation plans for agentic execution
2026-06-08 13:43:48 +08:00
Research Assistant
687332d126 docs: correct span_metadata source to PDF (not PaddleOCR), revise plan 2026-06-06 11:42:14 +08:00