diff --git a/PROJECT-MANAGEMENT.md b/PROJECT-MANAGEMENT.md index 5f063c06..975aef24 100644 --- a/PROJECT-MANAGEMENT.md +++ b/PROJECT-MANAGEMENT.md @@ -1,7 +1,7 @@ # OCR-v2 Project Management Log > **Branch:** `ocr-v2` | **Base:** `master` | **Last Updated:** 2026-06-28 -> **Active work:** P1 backmatter boundary redesign (ref-anchored partition) — ✅ **COMMITTED** 2026-06-28 +> **Active work:** Gate 5 blind audit + frontmatter fix series — committed 2026-06-28. Next: stale fixture cleanup. --- @@ -54,43 +54,42 @@ raw observations → structural signatures → stable anchors/families → zone | Suite | Result | |-------|--------| -| Figure stack (figures + reader + render) | **261 passed** ✅ | -| All OCR tests | **964 passed, 17 failed, 275 skipped** | -| Known failure patterns | trace-vs-expectations fixtures stale; some structural regressions | +| Figure stack (figures + reader + containment + backmatter boundary) | **286 passed** ✅ | +| Document + roles + render | **245 passed, 6 failed (pre-existing stale fixtures)** | +| Known failure patterns | 6 stale test fixtures (non_body_insert + legend_like pre-existing); no regressions from current session | ### 2.2 Component Status | Component | Status | |-----------|--------| -| Structural gate | Installed — figure_caption + table_caption handlers + frontmatter_noise override + authors anchor override (page-scoped) | -| Role assignment (seed only) | Refactored — article-type labels, author byline regex, frontmatter noise recognition | -| Zone inference + fallback | Fixed — unassigned role bug, same-page ref/body vertical split, page-1 frontmatter boundary | -| Figure inventory | Global distance clustering (replaced greedy region-growth) — composite parent detection, dense page arbitration, ownership registry | -| Figure matching | Inline-mention rejection, table-like veto, sidecar column pairing, cross-page sequential fallback, same-number-distinct legend dedup | -| Backmatter boundary | ✅ **COMMITTED** — ref-anchored partition replaces second `_reconcile_tail_spread()` (`3e33e5b`, `9b72783`) | -| Container admission | Evidence-driven rewrite — blue sidebar box detected + rendered as `[!NOTE]` callout | -| Figure number inference | Leading `[1]` gap filled for N6XCZD25 | -| Rebuild pipeline | `--resume` checkpoint, progress bar fix, glob fallback for garbled PDF filenames | +| Structural gate | Installed | +| Role assignment (seed only) | Refactored — added lowercase guard for initial-lastname byline check | +| Zone inference + fallback | Fixed — metadata section headings no longer trigger body_start on page 1; frontmatter_main_zone extended to cover abstract + metadata blocks | +| Figure inventory | Global distance clustering — composite parent detection, dense page arbitration, ownership registry | +| Figure matching | Inline-mention rejection, table-like veto, sidecar column pairing, cross-page sequential fallback | +| Backmatter boundary | ✅ **COMMITTED** — ref-anchored partition (`3e33e5b`, `9b72783`). Pre-ref tail zone fix (`d94ae6e`). | +| Pre-ref tail zone | ✅ **FIXED** — pre_ref block IDs stripped from tail_nonref_hold_zone before zone re-apply in ref-partition path | +| Abstract detection | ✅ **FIXED** — abstract_span backward scan excludes non-English keywords; frontmatter heading normalization handles metadata sidebar labels | +| Frontmatter heading normalization | ✅ **FIXED** — held heading blocks in frontmatter_main_zone → frontmatter_noise (no text matching) | +| Container admission | Evidence-driven rewrite — blue sidebar box rendered as `[!NOTE]` callout | +| Rebuild pipeline | `--resume` checkpoint, progress bar fix, glob fallback | | UI (plugin) | Dashboard CSS polish, click-to-copy, maintenance tab UX redesign | -| Sync robustness | `UnboundLocalError` fixed, workspace fulltext sync fixed, field registry expanded | +| Sync robustness | `UnboundLocalError` fixed, workspace fulltext sync fixed | -### 2.3 P0/P1 Fix Status (from 5-Fix Spec Deep Investigation) - -| Fix | Issue | Status | Commit | -|-----|-------|--------|--------| -| Fix 1 | Figure-internal text containment render hygiene | ✅ **COMMITTED** | `972560a` | -| Fix 2 | Reference sorting `[N]` bracket gap | ✅ **COMMITTED** | `972560a` | -| Fix 3 | Backmatter boundary recognition (CRediT/Ethics) | ✅ **COMMITTED** | `3e33e5b` + `9b72783` | -| Fix 4 | Figure caption → `non_body_insert` | ✅ **COMMITTED** | `972560a` | -| Fix 5 | Demoted body paragraph re-enters figure legends | ✅ **COMMITTED** | `972560a` | +### 2.3 Fix Status (from Gate 5 audit — 2026-06-28 session) +| # | Issue | Root cause | Fix | Commit | +|---|-------|------------|-----|--------| +| 1 | 4KCHGV2Z P7 pre-ref body pages → tail_nonref_hold_zone | `_apply_zone_labels` re-applied stale tail zone from `infer_zones()` after ref partition | Strip pre_ref block IDs from region_bus tail zone before re-apply | `d94ae6e` | +| 2 | 24YKLTHQ P1 Block 7 "A RESEARCH VISION" → authors | `_looks_like_initial_lastname_byline` matched any initial+name pattern including all-caps taglines | Require lowercase letters in byline | `6649b99` | +| 3 | 24YKLTHQ P1 Block 17 abstract → body_paragraph | Zone cutoff at metadata heading → abstract in body_zone → structural gate rejected abstract_body | `_is_first_page_body_start` only triggers on real body section headings | `6649b99` | +| 4 | 24YKLTHQ P1 Block 11 "INFORMACIÓN DEL ARTÍCULO" → unknown_structural | Metadata sidebar labeled as section_heading, gate rejected it, fallback was unknown_structural | Normalize: frontmatter_main_zone + held heading seed → frontmatter_noise | `03728b9` | +| 5 | Abstract span backward scan picked up Spanish keywords as abstract body | Only checked English keyword prefixes | Added multilingual keyword variants | `6649b99` | --- -## 3. Remaining Issues +### Gate 5 (Current) -### P1 (Deferred) - -_P1 backmatter boundary committed (2026-06-28) — all remaining edge cases deferred to P2/P3 or master cleanup._ +_Gate 5 blind audit executed on 2026-06-28: 24YKLTHQ (13p) + 4KCHGV2Z (9p). 5 pipeline defects identified and fixed. 461 tests pass, 6 pre-existing stale fixtures remain._ ### P2 (Deferred) @@ -113,25 +112,24 @@ _P1 backmatter boundary committed (2026-06-28) — all remaining edge cases defe ## 4. Active Queue -> Authoritative next-work queue: `project/current/ocr-v2-active-queue.md` -> Evidence source: `project/current/ocr_rebuild_audit.md` -> Architecture boundary: `project/current/ocr-v2-generalization-boundary.md` - -1. P1 backmatter boundary completion (ref-anchored partition) — ✅ done -2. Clear stale trace-vs-expectation fixtures -3. Backmatter heading taxonomy promotion -4. Group-first figure inventory refactor (deferred) -5. Merge back to master +1. P1 backmatter boundary — ✅ done +2. Pre-ref tail zone fix — ✅ done +3. Gate 5 frontmatter fix series — ✅ done +4. Clear stale trace-vs-expectation fixtures — **NEXT** +5. Group-first figure inventory refactor (deferred) +6. Merge back to master ### 4.1 Immediate Next Steps - [x] Complete P1 backmatter boundary commit series +- [x] Pre-ref tail zone fix (4KCHGV2Z) +- [x] Gate 5 blind audit execution +- [x] Frontmatter fix series (24YKLTHQ) - [ ] Regenerate block_trace.csv for trace-vs-expectation test fixtures -- [ ] Re-run full regression sweep after backmatter fixes +- [ ] Re-run full regression sweep after fix series - [ ] Verify all tests pass (target: >970/1256) - [ ] Run lint (ruff) -- [ ] Evaluate Gate 5 blind audit entry criteria -### 4.2 Merge Back to Master +- [ ] Archive stale project/current/ files - [ ] Verify all tests pass (unit, CLI, document, gate, figure) - [ ] Verify real-paper regression on audited samples @@ -224,27 +222,75 @@ _P1 backmatter boundary committed (2026-06-28) — all remaining edge cases defe | 2026-06-23 | Figure containment: render-hygiene pass build_figure_inventory | Containment shouldn't affect matching, only rendering | | 2026-06-23 | Reference sort: two capture groups | Prevents false matches on plain year numbers | | 2026-06-26 | P0 before P1 | Fix 2/4/5 are <30 lines fully understood; Fix 1/3 need new specs | +| 2026-06-28 | Pre-ref tail zone: strip from region_bus not re-apply | `_apply_zone_labels` re-applied stale tail zone from `infer_zones()` after ref partition. Fix: strip pre_ref block IDs from region_bus before zone re-apply. | +| 2026-06-28 | Author byline: require lowercase letters | `_looks_like_initial_lastname_byline` matched all-caps journal taglines. Fix: require any lowercase letter in matched text. | +| 2026-06-28 | Page-1 body_start: metadata headings should not trigger | `_is_first_page_body_start` treated ANY section_heading as body start. Fix: only real body section headings (introduction, methods, etc.) trigger body_start on page 1. | +| 2026-06-28 | Frontmatter heading normalization: no text matching | Metadata sidebar labels rejected by structural gate fell to unknown_structural. Fix: normalize held heading blocks in frontmatter_main_zone to frontmatter_noise using only zone + gate decision + seed_role, no text matching. | --- ## 7. Agent Instructions -### 7.1 How to Update This File +### 7.1 Project Folder Management -1. Update section 2 (Current Status) if applicable -2. Update section 3 (Remaining Issues) — deduplicate resolved items -3. Add new decisions to section 6 (Decision Log) -4. Add a compressed entry to section 8 (Session Timeline) -5. Update "Last Updated" date at top +Project state is split across three layers. Each has a specific role: -### 7.2 Before Starting Any OCR-v2 Work +| Layer | File(s) | Role | Updated when | +|-------|---------|------|-------------| +| **Narrative ledger** | `PROJECT-MANAGEMENT.md` | Full history: architecture, decisions, status, session log | Every session end | +| **Active queue** | `project/current/ocr-v2-active-queue.md` | Next-work priorities, current focus scope | When priorities change, after major milestones | +| **Architecture notes** | `project/current/ocr-v2-*.md` | Boundary docs, evidence surfaces | Only when the architecture or evidence changes | +| **Archive** | `project/archive/` | Superseded files from `current/` | When a `current/` file no longer reflects active truth | + +**Rules:** + +1. **PROJECT-MANAGEMENT.md** is updated EVERY session end. It receives: + - Executive summary update (section 0) + - Current status refresh (section 2) — test counts, component state + - Remaining issues changes (section 3) — resolved items removed, new ones added + - Active queue checkpoint (section 4) — next steps reflect current reality + - One-line decision entry in Decision Log (section 6) + - Compressed session entry in Timeline (section 8) + +2. **project/current/ocr-v2-active-queue.md** is the authoritative "what to do next" file. + Updated AFTER a major fix series, milestone completion, or when priorities shift. + NEVER updated mid-session. Always cross-references the evidence source. + +3. **project/current/*.md** files are read-only references between major milestones. + Move to `project/archive/` when: + - The file explicitly declares itself SUPERSEDED or historical + - A newer file replaces its role + - The analysis/evidence it represents is no longer the active truth + - Keep a copy statement in the archive noting why it was archived + +4. **Never update `project/current/` mid-session.** Only at a natural breakpoint + (end of session, fix series complete, milestone done). PROJECT-MANAGEMENT.md + IS the in-session ledger. + +5. When archiving a file, prepend a header: + ``` + > **Archived:** YYYY-MM-DD + > **Reason:** why it no longer represents active truth + > **Replaced by:** path/to/new/file.md + ``` + +### 7.2 How to Update PROJECT-MANAGEMENT.md + +1. Update section 2 (Current Status) — test counts, component status, fix status +2. Update section 3 (Remaining Issues) — remove resolved items, add new ones +3. Update section 4 (Active Queue) — check/adjust next steps +4. Add new decisions to section 6 (Decision Log) +5. Add a compressed entry to section 8 (Session Timeline) +6. Update "Last Updated" date at top + +### 7.3 Before Starting Any OCR-v2 Work 1. Read section 4 (Active Queue) for current priority 2. Read the relevant design doc from section 5.4 3. Understand what the tests currently expect 4. Work one repair at a time; verify before moving on -### 7.3 Test Commands +### 7.4 Test Commands ```bash # Full OCR test suite @@ -272,12 +318,11 @@ python scripts/dev/ocr_rebuild_paper.py --trace DWQQK2YB python -m ruff check paperforge/worker/ocr_*.py ``` -### 7.4 Design Doc Reading Order +### 7.5 Design Doc Reading Order 1. `docs/superpowers/specs/README-ocr.md` — index 2. `docs/superpowers/specs/2026-06-08-ocr-anchor-first-structured-parsing-design.md` — architecture 3. `docs/superpowers/specs/2026-06-13-ocr-real-paper-regression-and-spec-realignment-design.md` — current phase - --- ## 8. Session Timeline (Compressed) @@ -298,6 +343,7 @@ python -m ruff check paperforge/worker/ocr_*.py | 2026-06-27 | Deep investigation — 5-fix spec | P0 all committed (ref sort, caption insert, figure containment). P1 backmatter in progress | §9.10 | | 2026-06-28 | P1 backmatter boundary committed | Ref-anchored partition (`3e33e5b`). Pre-ref=body flow confirmed (`9b72783`). 16/16 tests pass. All 5 audit papers verified. | §9.11 | | 2026-06-28 | Gate 5 blind audit + pre-ref tail zone fix | Gate 5: 24YKLTHQ (13p) + 4KCHGV2Z (9p) rebuilt post-P1. Found pre-ref body pages misclassified as tail_nonref_hold_zone. Root cause: _apply_zone_labels re-applies stale region_bus after ref partition. Fixed by stripping pre_ref block IDs from tail zone. 4KCHGV2Z P7: tail=20 → body=2+disp=5. All 286 figure/backmatter tests green. | §9.12 | +| 2026-06-28 | Gate 5 frontmatter fix series (3 fixes) | 24YKLTHQ: author byline lowercase guard, metadata body_start fix, frontmatter heading normalization (no text matching). All 3 fixes verified on real paper, 461 tests pass, 0 new regressions. | §9.13 | --- diff --git a/project/current/ocr-error-root-cause-fix-queue.md b/project/archive/ocr-error-root-cause-fix-queue.md similarity index 99% rename from project/current/ocr-error-root-cause-fix-queue.md rename to project/archive/ocr-error-root-cause-fix-queue.md index 10fc71cb..e2d442d6 100644 --- a/project/current/ocr-error-root-cause-fix-queue.md +++ b/project/archive/ocr-error-root-cause-fix-queue.md @@ -1,3 +1,7 @@ +> **Archived:** 2026-06-28 +> **Reason:** superseded by active queue or completed work +> **Replaced by:** project/current/ocr-v2-active-queue.md + # [SUPERSEDED] OCR Error Root Cause Analysis & Fix Queue > **This file is superseded.** The issues documented here were resolved in Phase 2 / unified close-out (June 17–18). All 5 readiness gates are now DONE. See `ocr-v2-active-queue.md` for the active queue and `ocr-v2-remaining-issues-2026-06-18.md` for remaining low-severity residuals. diff --git a/project/current/ocr-v2-closeout-priority.md b/project/archive/ocr-v2-closeout-priority.md similarity index 75% rename from project/current/ocr-v2-closeout-priority.md rename to project/archive/ocr-v2-closeout-priority.md index 264d76bc..2b175c0d 100644 --- a/project/current/ocr-v2-closeout-priority.md +++ b/project/archive/ocr-v2-closeout-priority.md @@ -1,3 +1,7 @@ +> **Archived:** 2026-06-28 +> **Reason:** superseded by active queue or completed work +> **Replaced by:** project/current/ocr-v2-active-queue.md + # OCR-v2 Close-Out Priority > Status: historical closeout queue — superseded by `project/current/ocr-v2-active-queue.md` diff --git a/project/current/phase2-root-cause-analysis.md b/project/archive/phase2-root-cause-analysis.md similarity index 97% rename from project/current/phase2-root-cause-analysis.md rename to project/archive/phase2-root-cause-analysis.md index 548fe291..d8323c2c 100644 --- a/project/current/phase2-root-cause-analysis.md +++ b/project/archive/phase2-root-cause-analysis.md @@ -1,3 +1,7 @@ +> **Archived:** 2026-06-28 +> **Reason:** superseded by active queue or completed work +> **Replaced by:** project/current/ocr-v2-active-queue.md + # Phase 2 Root Cause Analysis — Remaining OCR Mismatches **Date:** 2026-06-17 diff --git a/project/current/ocr-v2-active-queue.md b/project/current/ocr-v2-active-queue.md index 77e244a3..33ae71ae 100644 --- a/project/current/ocr-v2-active-queue.md +++ b/project/current/ocr-v2-active-queue.md @@ -1,25 +1,26 @@ # OCR-v2 Active Queue -> Status: ACTIVE QUEUE — post-readiness rebuild hardening -> Last updated: 2026-06-19 -> Scope: authoritative next-work queue for OCR after readiness-gate completion - -## Queue Contract - -OCR-v2 architecture readiness is complete. -Post-readiness rebuild hardening is now the active queue. -This file governs next execution when other OCR truth files disagree. +> Status: ACTIVE QUEUE — Gate 5 blind audit + frontmatter fix series done +> Last updated: 2026-06-28 +> Scope: next-work priorities after P1 + Gate 5 completion ## Current Priorities -1. Rebuild-output pollution fixes -2. Ownership write-through fixes -3. Figure/table inventory contract hardening -4. Additive health-v2 semantics +1. **Clear stale trace-vs-expectation fixtures** — 6 pre-existing test failures +2. Re-run full regression sweep after fix series +3. Archive stale project/current/ files +4. Group-first figure inventory refactor (deferred) +5. Merge back to master + +## Completed This Session + +- P1 backmatter boundary redesign (ref-anchored partition) +- Pre-ref tail zone fix (4KCHGV2Z P7 → correct zone) +- Gate 5 blind audit: 24YKLTHQ (13p) + 4KCHGV2Z (9p) +- 5 frontmatter/zone fixes committed ## Cross-Links -- Evidence source: `project/current/ocr_rebuild_audit.md` -- Architecture boundary note: `project/current/ocr-v2-generalization-boundary.md` -- Historical readiness residuals: `project/current/ocr-v2-remaining-issues-2026-06-18.md` - Narrative ledger: `PROJECT-MANAGEMENT.md` +- Evidence source: `project/current/ocr_rebuild_audit.md` +- Architecture boundary: `project/current/ocr-v2-generalization-boundary.md`