mirror of
https://github.com/lllin000/PaperForge.git
synced 2026-07-22 06:50:53 +00:00
project: update for Gate 5 session — PM rules, archive, active queue
- PROJECT-MANAGEMENT.md: new project folder management rules (7.1), updated status, fix table, decision log, session timeline, active queue - project/current/ocr-v2-active-queue.md: refreshed for Gate 5 completion - project/archive/: moved 3 stale current/ files with archive headers (ocr-v2-closeout-priority.md, phase2-root-cause-analysis.md, ocr-error-root-cause-fix-queue.md)
This commit is contained in:
Research Assistant
parent
71160ceca8
commit
7b0bb41a9c
5 changed files with 124 additions and 65 deletions
|
|
@ -1,7 +1,7 @@
|
|||
# OCR-v2 Project Management Log
|
||||
|
||||
> **Branch:** `ocr-v2` | **Base:** `master` | **Last Updated:** 2026-06-28
|
||||
> **Active work:** P1 backmatter boundary redesign (ref-anchored partition) — ✅ **COMMITTED** 2026-06-28
|
||||
> **Active work:** Gate 5 blind audit + frontmatter fix series — committed 2026-06-28. Next: stale fixture cleanup.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -54,43 +54,42 @@ raw observations → structural signatures → stable anchors/families → zone
|
|||
|
||||
| Suite | Result |
|
||||
|-------|--------|
|
||||
| Figure stack (figures + reader + render) | **261 passed** ✅ |
|
||||
| All OCR tests | **964 passed, 17 failed, 275 skipped** |
|
||||
| Known failure patterns | trace-vs-expectations fixtures stale; some structural regressions |
|
||||
| Figure stack (figures + reader + containment + backmatter boundary) | **286 passed** ✅ |
|
||||
| Document + roles + render | **245 passed, 6 failed (pre-existing stale fixtures)** |
|
||||
| Known failure patterns | 6 stale test fixtures (non_body_insert + legend_like pre-existing); no regressions from current session |
|
||||
|
||||
### 2.2 Component Status
|
||||
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| Structural gate | Installed — figure_caption + table_caption handlers + frontmatter_noise override + authors anchor override (page-scoped) |
|
||||
| Role assignment (seed only) | Refactored — article-type labels, author byline regex, frontmatter noise recognition |
|
||||
| Zone inference + fallback | Fixed — unassigned role bug, same-page ref/body vertical split, page-1 frontmatter boundary |
|
||||
| Figure inventory | Global distance clustering (replaced greedy region-growth) — composite parent detection, dense page arbitration, ownership registry |
|
||||
| Figure matching | Inline-mention rejection, table-like veto, sidecar column pairing, cross-page sequential fallback, same-number-distinct legend dedup |
|
||||
| Backmatter boundary | ✅ **COMMITTED** — ref-anchored partition replaces second `_reconcile_tail_spread()` (`3e33e5b`, `9b72783`) |
|
||||
| Container admission | Evidence-driven rewrite — blue sidebar box detected + rendered as `[!NOTE]` callout |
|
||||
| Figure number inference | Leading `[1]` gap filled for N6XCZD25 |
|
||||
| Rebuild pipeline | `--resume` checkpoint, progress bar fix, glob fallback for garbled PDF filenames |
|
||||
| Structural gate | Installed |
|
||||
| Role assignment (seed only) | Refactored — added lowercase guard for initial-lastname byline check |
|
||||
| Zone inference + fallback | Fixed — metadata section headings no longer trigger body_start on page 1; frontmatter_main_zone extended to cover abstract + metadata blocks |
|
||||
| Figure inventory | Global distance clustering — composite parent detection, dense page arbitration, ownership registry |
|
||||
| Figure matching | Inline-mention rejection, table-like veto, sidecar column pairing, cross-page sequential fallback |
|
||||
| Backmatter boundary | ✅ **COMMITTED** — ref-anchored partition (`3e33e5b`, `9b72783`). Pre-ref tail zone fix (`d94ae6e`). |
|
||||
| Pre-ref tail zone | ✅ **FIXED** — pre_ref block IDs stripped from tail_nonref_hold_zone before zone re-apply in ref-partition path |
|
||||
| Abstract detection | ✅ **FIXED** — abstract_span backward scan excludes non-English keywords; frontmatter heading normalization handles metadata sidebar labels |
|
||||
| Frontmatter heading normalization | ✅ **FIXED** — held heading blocks in frontmatter_main_zone → frontmatter_noise (no text matching) |
|
||||
| Container admission | Evidence-driven rewrite — blue sidebar box rendered as `[!NOTE]` callout |
|
||||
| Rebuild pipeline | `--resume` checkpoint, progress bar fix, glob fallback |
|
||||
| UI (plugin) | Dashboard CSS polish, click-to-copy, maintenance tab UX redesign |
|
||||
| Sync robustness | `UnboundLocalError` fixed, workspace fulltext sync fixed, field registry expanded |
|
||||
| Sync robustness | `UnboundLocalError` fixed, workspace fulltext sync fixed |
|
||||
|
||||
### 2.3 P0/P1 Fix Status (from 5-Fix Spec Deep Investigation)
|
||||
|
||||
| Fix | Issue | Status | Commit |
|
||||
|-----|-------|--------|--------|
|
||||
| Fix 1 | Figure-internal text containment render hygiene | ✅ **COMMITTED** | `972560a` |
|
||||
| Fix 2 | Reference sorting `[N]` bracket gap | ✅ **COMMITTED** | `972560a` |
|
||||
| Fix 3 | Backmatter boundary recognition (CRediT/Ethics) | ✅ **COMMITTED** | `3e33e5b` + `9b72783` |
|
||||
| Fix 4 | Figure caption → `non_body_insert` | ✅ **COMMITTED** | `972560a` |
|
||||
| Fix 5 | Demoted body paragraph re-enters figure legends | ✅ **COMMITTED** | `972560a` |
|
||||
### 2.3 Fix Status (from Gate 5 audit — 2026-06-28 session)
|
||||
|
||||
| # | Issue | Root cause | Fix | Commit |
|
||||
|---|-------|------------|-----|--------|
|
||||
| 1 | 4KCHGV2Z P7 pre-ref body pages → tail_nonref_hold_zone | `_apply_zone_labels` re-applied stale tail zone from `infer_zones()` after ref partition | Strip pre_ref block IDs from region_bus tail zone before re-apply | `d94ae6e` |
|
||||
| 2 | 24YKLTHQ P1 Block 7 "A RESEARCH VISION" → authors | `_looks_like_initial_lastname_byline` matched any initial+name pattern including all-caps taglines | Require lowercase letters in byline | `6649b99` |
|
||||
| 3 | 24YKLTHQ P1 Block 17 abstract → body_paragraph | Zone cutoff at metadata heading → abstract in body_zone → structural gate rejected abstract_body | `_is_first_page_body_start` only triggers on real body section headings | `6649b99` |
|
||||
| 4 | 24YKLTHQ P1 Block 11 "INFORMACIÓN DEL ARTÍCULO" → unknown_structural | Metadata sidebar labeled as section_heading, gate rejected it, fallback was unknown_structural | Normalize: frontmatter_main_zone + held heading seed → frontmatter_noise | `03728b9` |
|
||||
| 5 | Abstract span backward scan picked up Spanish keywords as abstract body | Only checked English keyword prefixes | Added multilingual keyword variants | `6649b99` |
|
||||
---
|
||||
|
||||
## 3. Remaining Issues
|
||||
### Gate 5 (Current)
|
||||
|
||||
### P1 (Deferred)
|
||||
|
||||
_P1 backmatter boundary committed (2026-06-28) — all remaining edge cases deferred to P2/P3 or master cleanup._
|
||||
_Gate 5 blind audit executed on 2026-06-28: 24YKLTHQ (13p) + 4KCHGV2Z (9p). 5 pipeline defects identified and fixed. 461 tests pass, 6 pre-existing stale fixtures remain._
|
||||
|
||||
### P2 (Deferred)
|
||||
|
||||
|
|
@ -113,25 +112,24 @@ _P1 backmatter boundary committed (2026-06-28) — all remaining edge cases defe
|
|||
|
||||
## 4. Active Queue
|
||||
|
||||
> Authoritative next-work queue: `project/current/ocr-v2-active-queue.md`
|
||||
> Evidence source: `project/current/ocr_rebuild_audit.md`
|
||||
> Architecture boundary: `project/current/ocr-v2-generalization-boundary.md`
|
||||
|
||||
1. P1 backmatter boundary completion (ref-anchored partition) — ✅ done
|
||||
2. Clear stale trace-vs-expectation fixtures
|
||||
3. Backmatter heading taxonomy promotion
|
||||
4. Group-first figure inventory refactor (deferred)
|
||||
5. Merge back to master
|
||||
1. P1 backmatter boundary — ✅ done
|
||||
2. Pre-ref tail zone fix — ✅ done
|
||||
3. Gate 5 frontmatter fix series — ✅ done
|
||||
4. Clear stale trace-vs-expectation fixtures — **NEXT**
|
||||
5. Group-first figure inventory refactor (deferred)
|
||||
6. Merge back to master
|
||||
|
||||
### 4.1 Immediate Next Steps
|
||||
|
||||
- [x] Complete P1 backmatter boundary commit series
|
||||
- [x] Pre-ref tail zone fix (4KCHGV2Z)
|
||||
- [x] Gate 5 blind audit execution
|
||||
- [x] Frontmatter fix series (24YKLTHQ)
|
||||
- [ ] Regenerate block_trace.csv for trace-vs-expectation test fixtures
|
||||
- [ ] Re-run full regression sweep after backmatter fixes
|
||||
- [ ] Re-run full regression sweep after fix series
|
||||
- [ ] Verify all tests pass (target: >970/1256)
|
||||
- [ ] Run lint (ruff)
|
||||
- [ ] Evaluate Gate 5 blind audit entry criteria
|
||||
### 4.2 Merge Back to Master
|
||||
- [ ] Archive stale project/current/ files
|
||||
|
||||
- [ ] Verify all tests pass (unit, CLI, document, gate, figure)
|
||||
- [ ] Verify real-paper regression on audited samples
|
||||
|
|
@ -224,27 +222,75 @@ _P1 backmatter boundary committed (2026-06-28) — all remaining edge cases defe
|
|||
| 2026-06-23 | Figure containment: render-hygiene pass build_figure_inventory | Containment shouldn't affect matching, only rendering |
|
||||
| 2026-06-23 | Reference sort: two capture groups | Prevents false matches on plain year numbers |
|
||||
| 2026-06-26 | P0 before P1 | Fix 2/4/5 are <30 lines fully understood; Fix 1/3 need new specs |
|
||||
| 2026-06-28 | Pre-ref tail zone: strip from region_bus not re-apply | `_apply_zone_labels` re-applied stale tail zone from `infer_zones()` after ref partition. Fix: strip pre_ref block IDs from region_bus before zone re-apply. |
|
||||
| 2026-06-28 | Author byline: require lowercase letters | `_looks_like_initial_lastname_byline` matched all-caps journal taglines. Fix: require any lowercase letter in matched text. |
|
||||
| 2026-06-28 | Page-1 body_start: metadata headings should not trigger | `_is_first_page_body_start` treated ANY section_heading as body start. Fix: only real body section headings (introduction, methods, etc.) trigger body_start on page 1. |
|
||||
| 2026-06-28 | Frontmatter heading normalization: no text matching | Metadata sidebar labels rejected by structural gate fell to unknown_structural. Fix: normalize held heading blocks in frontmatter_main_zone to frontmatter_noise using only zone + gate decision + seed_role, no text matching. |
|
||||
|
||||
---
|
||||
|
||||
## 7. Agent Instructions
|
||||
|
||||
### 7.1 How to Update This File
|
||||
### 7.1 Project Folder Management
|
||||
|
||||
1. Update section 2 (Current Status) if applicable
|
||||
2. Update section 3 (Remaining Issues) — deduplicate resolved items
|
||||
3. Add new decisions to section 6 (Decision Log)
|
||||
4. Add a compressed entry to section 8 (Session Timeline)
|
||||
5. Update "Last Updated" date at top
|
||||
Project state is split across three layers. Each has a specific role:
|
||||
|
||||
### 7.2 Before Starting Any OCR-v2 Work
|
||||
| Layer | File(s) | Role | Updated when |
|
||||
|-------|---------|------|-------------|
|
||||
| **Narrative ledger** | `PROJECT-MANAGEMENT.md` | Full history: architecture, decisions, status, session log | Every session end |
|
||||
| **Active queue** | `project/current/ocr-v2-active-queue.md` | Next-work priorities, current focus scope | When priorities change, after major milestones |
|
||||
| **Architecture notes** | `project/current/ocr-v2-*.md` | Boundary docs, evidence surfaces | Only when the architecture or evidence changes |
|
||||
| **Archive** | `project/archive/` | Superseded files from `current/` | When a `current/` file no longer reflects active truth |
|
||||
|
||||
**Rules:**
|
||||
|
||||
1. **PROJECT-MANAGEMENT.md** is updated EVERY session end. It receives:
|
||||
- Executive summary update (section 0)
|
||||
- Current status refresh (section 2) — test counts, component state
|
||||
- Remaining issues changes (section 3) — resolved items removed, new ones added
|
||||
- Active queue checkpoint (section 4) — next steps reflect current reality
|
||||
- One-line decision entry in Decision Log (section 6)
|
||||
- Compressed session entry in Timeline (section 8)
|
||||
|
||||
2. **project/current/ocr-v2-active-queue.md** is the authoritative "what to do next" file.
|
||||
Updated AFTER a major fix series, milestone completion, or when priorities shift.
|
||||
NEVER updated mid-session. Always cross-references the evidence source.
|
||||
|
||||
3. **project/current/*.md** files are read-only references between major milestones.
|
||||
Move to `project/archive/` when:
|
||||
- The file explicitly declares itself SUPERSEDED or historical
|
||||
- A newer file replaces its role
|
||||
- The analysis/evidence it represents is no longer the active truth
|
||||
- Keep a copy statement in the archive noting why it was archived
|
||||
|
||||
4. **Never update `project/current/` mid-session.** Only at a natural breakpoint
|
||||
(end of session, fix series complete, milestone done). PROJECT-MANAGEMENT.md
|
||||
IS the in-session ledger.
|
||||
|
||||
5. When archiving a file, prepend a header:
|
||||
```
|
||||
> **Archived:** YYYY-MM-DD
|
||||
> **Reason:** why it no longer represents active truth
|
||||
> **Replaced by:** path/to/new/file.md
|
||||
```
|
||||
|
||||
### 7.2 How to Update PROJECT-MANAGEMENT.md
|
||||
|
||||
1. Update section 2 (Current Status) — test counts, component status, fix status
|
||||
2. Update section 3 (Remaining Issues) — remove resolved items, add new ones
|
||||
3. Update section 4 (Active Queue) — check/adjust next steps
|
||||
4. Add new decisions to section 6 (Decision Log)
|
||||
5. Add a compressed entry to section 8 (Session Timeline)
|
||||
6. Update "Last Updated" date at top
|
||||
|
||||
### 7.3 Before Starting Any OCR-v2 Work
|
||||
|
||||
1. Read section 4 (Active Queue) for current priority
|
||||
2. Read the relevant design doc from section 5.4
|
||||
3. Understand what the tests currently expect
|
||||
4. Work one repair at a time; verify before moving on
|
||||
|
||||
### 7.3 Test Commands
|
||||
### 7.4 Test Commands
|
||||
|
||||
```bash
|
||||
# Full OCR test suite
|
||||
|
|
@ -272,12 +318,11 @@ python scripts/dev/ocr_rebuild_paper.py --trace DWQQK2YB
|
|||
python -m ruff check paperforge/worker/ocr_*.py
|
||||
```
|
||||
|
||||
### 7.4 Design Doc Reading Order
|
||||
### 7.5 Design Doc Reading Order
|
||||
|
||||
1. `docs/superpowers/specs/README-ocr.md` — index
|
||||
2. `docs/superpowers/specs/2026-06-08-ocr-anchor-first-structured-parsing-design.md` — architecture
|
||||
3. `docs/superpowers/specs/2026-06-13-ocr-real-paper-regression-and-spec-realignment-design.md` — current phase
|
||||
|
||||
---
|
||||
|
||||
## 8. Session Timeline (Compressed)
|
||||
|
|
@ -298,6 +343,7 @@ python -m ruff check paperforge/worker/ocr_*.py
|
|||
| 2026-06-27 | Deep investigation — 5-fix spec | P0 all committed (ref sort, caption insert, figure containment). P1 backmatter in progress | §9.10 |
|
||||
| 2026-06-28 | P1 backmatter boundary committed | Ref-anchored partition (`3e33e5b`). Pre-ref=body flow confirmed (`9b72783`). 16/16 tests pass. All 5 audit papers verified. | §9.11 |
|
||||
| 2026-06-28 | Gate 5 blind audit + pre-ref tail zone fix | Gate 5: 24YKLTHQ (13p) + 4KCHGV2Z (9p) rebuilt post-P1. Found pre-ref body pages misclassified as tail_nonref_hold_zone. Root cause: _apply_zone_labels re-applies stale region_bus after ref partition. Fixed by stripping pre_ref block IDs from tail zone. 4KCHGV2Z P7: tail=20 → body=2+disp=5. All 286 figure/backmatter tests green. | §9.12 |
|
||||
| 2026-06-28 | Gate 5 frontmatter fix series (3 fixes) | 24YKLTHQ: author byline lowercase guard, metadata body_start fix, frontmatter heading normalization (no text matching). All 3 fixes verified on real paper, 461 tests pass, 0 new regressions. | §9.13 |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +1,7 @@
|
|||
> **Archived:** 2026-06-28
|
||||
> **Reason:** superseded by active queue or completed work
|
||||
> **Replaced by:** project/current/ocr-v2-active-queue.md
|
||||
|
||||
# [SUPERSEDED] OCR Error Root Cause Analysis & Fix Queue
|
||||
|
||||
> **This file is superseded.** The issues documented here were resolved in Phase 2 / unified close-out (June 17–18). All 5 readiness gates are now DONE. See `ocr-v2-active-queue.md` for the active queue and `ocr-v2-remaining-issues-2026-06-18.md` for remaining low-severity residuals.
|
||||
|
|
@ -1,3 +1,7 @@
|
|||
> **Archived:** 2026-06-28
|
||||
> **Reason:** superseded by active queue or completed work
|
||||
> **Replaced by:** project/current/ocr-v2-active-queue.md
|
||||
|
||||
# OCR-v2 Close-Out Priority
|
||||
|
||||
> Status: historical closeout queue — superseded by `project/current/ocr-v2-active-queue.md`
|
||||
|
|
@ -1,3 +1,7 @@
|
|||
> **Archived:** 2026-06-28
|
||||
> **Reason:** superseded by active queue or completed work
|
||||
> **Replaced by:** project/current/ocr-v2-active-queue.md
|
||||
|
||||
# Phase 2 Root Cause Analysis — Remaining OCR Mismatches
|
||||
|
||||
**Date:** 2026-06-17
|
||||
|
|
@ -1,25 +1,26 @@
|
|||
# OCR-v2 Active Queue
|
||||
|
||||
> Status: ACTIVE QUEUE — post-readiness rebuild hardening
|
||||
> Last updated: 2026-06-19
|
||||
> Scope: authoritative next-work queue for OCR after readiness-gate completion
|
||||
|
||||
## Queue Contract
|
||||
|
||||
OCR-v2 architecture readiness is complete.
|
||||
Post-readiness rebuild hardening is now the active queue.
|
||||
This file governs next execution when other OCR truth files disagree.
|
||||
> Status: ACTIVE QUEUE — Gate 5 blind audit + frontmatter fix series done
|
||||
> Last updated: 2026-06-28
|
||||
> Scope: next-work priorities after P1 + Gate 5 completion
|
||||
|
||||
## Current Priorities
|
||||
|
||||
1. Rebuild-output pollution fixes
|
||||
2. Ownership write-through fixes
|
||||
3. Figure/table inventory contract hardening
|
||||
4. Additive health-v2 semantics
|
||||
1. **Clear stale trace-vs-expectation fixtures** — 6 pre-existing test failures
|
||||
2. Re-run full regression sweep after fix series
|
||||
3. Archive stale project/current/ files
|
||||
4. Group-first figure inventory refactor (deferred)
|
||||
5. Merge back to master
|
||||
|
||||
## Completed This Session
|
||||
|
||||
- P1 backmatter boundary redesign (ref-anchored partition)
|
||||
- Pre-ref tail zone fix (4KCHGV2Z P7 → correct zone)
|
||||
- Gate 5 blind audit: 24YKLTHQ (13p) + 4KCHGV2Z (9p)
|
||||
- 5 frontmatter/zone fixes committed
|
||||
|
||||
## Cross-Links
|
||||
|
||||
- Evidence source: `project/current/ocr_rebuild_audit.md`
|
||||
- Architecture boundary note: `project/current/ocr-v2-generalization-boundary.md`
|
||||
- Historical readiness residuals: `project/current/ocr-v2-remaining-issues-2026-06-18.md`
|
||||
- Narrative ledger: `PROJECT-MANAGEMENT.md`
|
||||
- Evidence source: `project/current/ocr_rebuild_audit.md`
|
||||
- Architecture boundary: `project/current/ocr-v2-generalization-boundary.md`
|
||||
|
|
|
|||
Loading…
Reference in a new issue