test: lock OCR readiness baseline and restore coverage ledger

audit/coverage_ledger.json

@@ -0,0 +1,13 @@
+{
+  "audit_root": "audit",
+  "papers": [
+    {"paper_key": "2GN9LMCW", "layout_tags": ["single_column", "special_structure"], "risk_tags": ["special_structure"]},
+    {"paper_key": "6FGDBFQN", "layout_tags": ["multi_column", "side_caption"], "risk_tags": ["figure_heavy"]},
+    {"paper_key": "A8E7SRVS", "layout_tags": ["multi_column"], "risk_tags": ["table_heavy"]},
+    {"paper_key": "CAQNW9Q2", "layout_tags": ["multi_column", "same_page_ref_body_split"], "risk_tags": ["reference_boundary_sensitive", "frontmatter_sensitive"]},
+    {"paper_key": "DWQQK2YB", "layout_tags": ["multi_column", "preproof_frontmatter", "post_reference_biography"], "risk_tags": ["frontmatter_sensitive", "figure_heavy", "cross_page_caption"]},
+    {"paper_key": "K7R8PEKW", "layout_tags": ["single_column"], "risk_tags": ["frontmatter_sensitive"]},
+    {"paper_key": "SAN9AYVR", "layout_tags": ["multi_column"], "risk_tags": ["special_structure"]},
+    {"paper_key": "TSCKAVIS", "layout_tags": ["single_column", "review_callout"], "risk_tags": ["special_structure", "table_heavy"]}
+  ]
+}

project/current/ocr-v2-closeout-priority.md

@@ -1,8 +1,8 @@
 # OCR-v2 Close-Out Priority
 
-> Status: active authoritative execution topic
-> Last updated: 2026-06-17
-> Owner file for next work: `docs/superpowers/plans/2026-06-17-ocr-v2-closeout-single-plan.md`
+> Status: active authoritative readiness queue
+> Last updated: 2026-06-18
+> Owner file for next work: `docs/superpowers/plans/2026-06-18-ocr-v2-readiness-master-plan.md`
 
 ## Why This File Exists
 
@@ -30,12 +30,18 @@ The unified close-out plan has been executed, and P0-P2 layout close-out is reco
 - Regression suite: 202P / 1F / 43S
 - DW diff audit: 57% verified; CAQ diff audit: 88% verified
 
-## Remaining blockers before merge:
-- DWQQK2YB figure ownership on mixed post-reference figure pages (group-first refactor deferred)
-- completeness-check layer implementation (next slice)
+That pass is no longer the active execution thread. The branch has now moved into readiness-gate work: finish the first four gates required for "state healthy," then run unseen-paper blind audit as the next-stage gate.
+
+## Remaining readiness blockers before "state healthy"
+- Gate 1: completeness-check layer implementation
+- Gate 2: figure ownership generalization on mixed post-reference and multi-panel pages
+- Gate 3: ordering/boundary authority still partly depends on renderer-side repair
+- Gate 4: audit-paper taxonomy and coverage ledger are not yet formal enough for generalization claims
 
 ## Next Work
-1. Implement fuzzy OCR completeness-check layer (described in `docs/superpowers/specs/2026-06-18-ocr-completeness-check-design.md`)
+1. Execute `docs/superpowers/plans/2026-06-18-ocr-v2-readiness-master-plan.md`
+2. Implement Gate 1 first: fuzzy OCR completeness-check layer (`docs/superpowers/specs/2026-06-18-ocr-completeness-check-design.md`)
+3. Keep Gate 5 (unseen-paper blind audit) out of the current cycle; use it only after Gates 1-4 are complete
 
 ## Authoritative Reference
-This file is the tie-breaker. If another document disagrees about what to do next, this file wins until it is archived.
+This file is the tie-breaker for the active readiness queue. `project/current/ocr-v2-generalization-boundary.md` remains the broader architecture note, and `docs/superpowers/specs/2026-06-18-ocr-v2-readiness-gates-design.md` defines the readiness model the queue is following.

project/current/ocr-v2-remaining-issues-2026-06-18.md

@@ -1,18 +1,24 @@
 # OCR-v2 Remaining Issues & Root Cause Analysis
 
 > Date: 2026-06-18
-> Baseline: After 9.7 close-out, ahead of DW repair phase
+> Baseline: readiness-gates transition after 9.9 DW repair phase
 
-## Active Residuals After 9.7
+## Active Readiness Residuals
 
-1. **DWQQK2YB figure ownership still over-claims or stays ambiguous** on mixed post-reference figure pages (group-first figure inventory refactor deferred)
-2. **Backmatter heading taxonomy remains partially conservative by design**; only fix if verified zone semantics require it
-3. **Completeness-check layer is specified and pending implementation**
+1. **Gate 1 completeness-check layer is specified but not implemented** in the structured-block and rendered-coverage paths
+2. **Gate 2 figure ownership still carries a trust gap** on mixed post-reference and multi-panel pages; current hard case still tolerates ambiguity in practice
+3. **Gate 3 ordering/boundary authority is not fully upstream**; renderer-side repair remains part of the correctness story
+4. **Gate 4 layout coverage is under-formalized**; old fixture-ledger assumptions are stale, and the readiness ledger needs to live against the real audit-paper corpus instead
+5. **Backmatter heading taxonomy remains partially conservative by design**; only promote it if Gate 3 boundary evidence requires it
+6. **Gate 5 unseen-paper blind audit is intentionally deferred** until Gates 1-4 are complete
 
 ## Priority Summary
 
 | # | Issue | Priority | Type | Status |
 |---|-------|----------|------|--------|
-| 1 | DW figure ownership | P0 | Ownership gap | Deferred (architectural) |
-| 2 | Backmatter heading | P3 | Conservative by design | Deferred |
-| 3 | Completeness check | P1 | Feature gap | Planned next slice |
+| 1 | Completeness check | P1 | Readiness gate | Active next slice |
+| 2 | Figure ownership | P0 | Trust / ownership gap | Active after Gate 1 |
+| 3 | Ordering authority | P1 | Structural authority gap | Active after Gate 2 |
+| 4 | Layout coverage ledger + taxonomy | P1 | Test / evidence gap | Active after Gate 3 |
+| 5 | Backmatter heading taxonomy | P3 | Conservative by design | Deferred unless Gate 3 needs it |
+| 6 | Unseen-paper blind audit | Next stage | Validation gate | Deferred by design |

scripts/dev/ocr_render_annotated_pages.py

@@ -0,0 +1,181 @@
+"""Render fitz-based annotated PDF pages for OCR gold audits.
+
+Reads `block_trace.csv` from the test fixtures, resolves the source PDF from the
+Literature-hub vault, and writes one annotated PNG per PDF page.
+
+Usage:
+    python scripts/dev/ocr_render_annotated_pages.py CAQNW9Q2 DWQQK2YB
+    python scripts/dev/ocr_render_annotated_pages.py --all-gold
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+import sys
+from pathlib import Path
+
+import fitz
+from PIL import Image
+
+
+_THIS_FILE = Path(__file__).resolve()
+_REPO_ROOT = _THIS_FILE.parents[2]
+_FIXTURE_ROOT = _REPO_ROOT / "tests" / "fixtures" / "ocr_real_papers"
+_LEDGER_PATH = _REPO_ROOT / "audit" / "coverage_ledger.json"
+_VAULT = Path(r"D:\L\OB\Literature-hub")
+_OCR_ROOT = _VAULT / "System" / "PaperForge" / "ocr"
+
+ROLE_COLORS = {
+    "paper_title": (0.0, 0.4, 1.0),
+    "authors": (0.0, 0.7, 0.8),
+    "abstract_heading": (0.4, 0.0, 0.9),
+    "abstract_body": (0.6, 0.0, 0.9),
+    "section_heading": (1.0, 0.2, 0.2),
+    "subsection_heading": (1.0, 0.5, 0.0),
+    "body_paragraph": (0.0, 0.7, 0.2),
+    "reference_heading": (0.7, 0.1, 0.1),
+    "reference_item": (0.7, 0.2, 0.2),
+    "backmatter_heading": (0.5, 0.2, 0.7),
+    "backmatter_body": (0.5, 0.3, 0.7),
+    "structured_insert": (0.9, 0.0, 0.7),
+    "frontmatter_support": (0.2, 0.7, 0.9),
+    "frontmatter_noise": (0.4, 0.4, 0.4),
+    "noise": (0.5, 0.5, 0.5),
+    "unknown_structural": (0.2, 0.2, 0.2),
+}
+
+
+def _load_ledger() -> dict:
+    return json.loads(_LEDGER_PATH.read_text(encoding="utf-8"))
+
+
+def _gold_keys() -> list[str]:
+    return [paper["paper_key"] for paper in _load_ledger().get("papers", [])]
+
+
+def _read_trace(key: str) -> list[dict[str, str]]:
+    path = _FIXTURE_ROOT / key / "block_trace.csv"
+    with path.open("r", encoding="utf-8-sig", newline="") as fh:
+        return list(csv.DictReader(fh))
+
+
+def _resolve_source_pdf(key: str) -> Path:
+    note_candidates = list(_VAULT.glob(f"Resources/Literature/**/{key}.md"))
+    for note_path in note_candidates:
+        text = note_path.read_text(encoding="utf-8")
+        for line in text.splitlines():
+            if not line.startswith("pdf_path:"):
+                continue
+            start = line.find("[[")
+            end = line.find("]]", start + 2)
+            if start >= 0 and end > start:
+                rel = line[start + 2:end]
+                pdf_path = _VAULT / rel.replace("/", "\\")
+                if pdf_path.exists():
+                    return pdf_path
+
+    meta_path = _OCR_ROOT / key / "raw" / "source_metadata.json"
+    if meta_path.exists():
+        meta = json.loads(meta_path.read_text(encoding="utf-8"))
+        source = meta.get("source_pdf")
+        if source and Path(source).exists():
+            return Path(source)
+
+    local = _OCR_ROOT / key / "source.pdf"
+    if local.exists():
+        return local
+
+    raise FileNotFoundError(f"Could not resolve source PDF for {key}")
+
+
+def _parse_bbox(raw: str) -> fitz.Rect:
+    bbox = json.loads(raw)
+    return fitz.Rect(float(bbox[0]), float(bbox[1]), float(bbox[2]), float(bbox[3]))
+
+
+def _ocr_page_image_path(key: str, page_number: int) -> Path:
+    page_dir = _OCR_ROOT / key / "pages"
+    for suffix in ("png", "jpg", "jpeg"):
+        candidate = page_dir / f"page_{page_number:03d}.{suffix}"
+        if candidate.exists():
+            return candidate
+    raise FileNotFoundError(f"Missing OCR page image for {key} page {page_number}")
+
+
+def _scale_rect_to_pdf(key: str, page_number: int, rect: fitz.Rect, pdf_rect: fitz.Rect) -> fitz.Rect:
+    image_path = _ocr_page_image_path(key, page_number)
+    image_width, image_height = Image.open(image_path).size
+    x_scale = pdf_rect.width / image_width
+    y_scale = pdf_rect.height / image_height
+    return fitz.Rect(
+        rect.x0 * x_scale,
+        rect.y0 * y_scale,
+        rect.x1 * x_scale,
+        rect.y1 * y_scale,
+    )
+
+
+def _role_color(role: str) -> tuple[float, float, float]:
+    return ROLE_COLORS.get(role, (0.1, 0.1, 0.1))
+
+
+def render_annotated_pages(key: str, *, zoom: float = 2.0) -> Path:
+    pdf_path = _resolve_source_pdf(key)
+    trace_rows = _read_trace(key)
+    out_dir = _FIXTURE_ROOT / key / "annotated_pages"
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    doc = fitz.open(str(pdf_path))
+    rows_by_page: dict[int, list[dict[str, str]]] = {}
+    for row in trace_rows:
+        rows_by_page.setdefault(int(row["page"]), []).append(row)
+
+    for page_number in range(1, doc.page_count + 1):
+        page = doc.load_page(page_number - 1)
+        boxes = rows_by_page.get(page_number, [])
+        shape = page.new_shape()
+        for row in boxes:
+            rect = _scale_rect_to_pdf(key, page_number, _parse_bbox(row["bbox"]), page.rect)
+            role = row.get("role", "")
+            color = _role_color(role)
+            shape.draw_rect(rect)
+            shape.finish(color=color, width=0.8)
+            label = f"{row.get('block_id', '')}:{role}"
+            page.insert_text(
+                fitz.Point(rect.x0, max(8, rect.y0 - 2)),
+                label,
+                fontsize=5,
+                color=color,
+                overlay=True,
+            )
+        shape.commit(overlay=True)
+        pix = page.get_pixmap(matrix=fitz.Matrix(zoom, zoom), alpha=False)
+        pix.save(str(out_dir / f"page_{page_number:03d}.png"))
+
+    doc.close()
+    return out_dir
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("keys", nargs="*")
+    parser.add_argument("--all-gold", action="store_true")
+    parser.add_argument("--zoom", type=float, default=2.0)
+    args = parser.parse_args(argv)
+
+    keys = args.keys
+    if args.all_gold:
+        keys = _gold_keys()
+    if not keys:
+        parser.error("provide paper keys or use --all-gold")
+
+    for key in keys:
+        out_dir = render_annotated_pages(key, zoom=args.zoom)
+        print(f"[OK] {key}: {out_dir}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv[1:]))

tests/test_ocr_real_paper_audit_contracts.py

@@ -0,0 +1,35 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+
+FIXTURE_ROOT = Path(__file__).resolve().parent / "fixtures" / "ocr_real_papers"
+LEDGER_PATH = Path(__file__).resolve().parents[1] / "audit" / "coverage_ledger.json"
+
+
+def _load_manifest() -> dict:
+    return json.loads(LEDGER_PATH.read_text(encoding="utf-8"))
+
+
+def test_gold_set_covers_readiness_layout_classes() -> None:
+    manifest = _load_manifest()
+    all_layouts = {tag for paper in manifest["papers"] for tag in paper.get("layout_tags", [])}
+    assert "preproof_frontmatter" in all_layouts
+    assert "same_page_ref_body_split" in all_layouts
+    assert any(tag in all_layouts for tag in {"side_caption", "multi_panel"})
+    assert "post_reference_biography" in all_layouts
+    assert any(tag in all_layouts for tag in {"review_callout", "special_structure"})
+
+
+def test_layout_class_manifest_has_named_representatives() -> None:
+    manifest = _load_manifest()
+    by_tag: dict[str, set[str]] = {}
+    for paper in manifest["papers"]:
+        for tag in paper.get("layout_tags", []):
+            by_tag.setdefault(tag, set()).add(paper["paper_key"])
+
+    assert by_tag["preproof_frontmatter"]
+    assert by_tag["same_page_ref_body_split"]
+    assert by_tag["post_reference_biography"]
+    assert by_tag["review_callout"] or by_tag["special_structure"]

tests/test_ocr_real_paper_regressions.py

@@ -17,7 +17,8 @@ from pathlib import Path
 import pytest
 
 FIXTURE_ROOT = Path(__file__).resolve().parent / "fixtures" / "ocr_real_papers"
-MANIFEST_PATH = FIXTURE_ROOT / "coverage_manifest.json"
+LEDGER_PATH = Path(__file__).resolve().parents[1] / "audit" / "coverage_ledger.json"
+MANIFEST_PATH = LEDGER_PATH
 
 # ---------------------------------------------------------------------------
 # Fixture helper loaders
@@ -77,8 +78,9 @@ def _reader_figure_index(reader_payload: dict) -> tuple[dict[int, dict], dict[in
 
 
 def _load_reader_payload_from_vault(key: str) -> dict:
-    manifest = _load_manifest()
-    vault = Path(manifest["vault"])
+    vault = _real_ocr_vault()
+    if vault is None:
+        pytest.skip("PAPERFORGE_REAL_OCR_VAULT not set for vault fallback read")
     path = vault / "System" / "PaperForge" / "ocr" / key / "structure" / "reader_figures.json"
     return _load_json(path)  # type: ignore[return-value]
 
@@ -386,7 +388,7 @@ def test_a8e7srvs_page_level_production_pipeline(tmp_path: Path) -> None:
 
 def test_gold_figure_merge_ownership_contracts(tmp_path: Path) -> None:
     manifest = _load_manifest()
-    keys = [paper["paper_key"] for paper in manifest.get("gold_papers", [])]
+    keys = [paper["paper_key"] for paper in manifest.get("papers", [])]
     failures: list[str] = []
     for key in keys:
         expectations = _load_expectations(key)
@@ -472,6 +474,16 @@ def test_dwqqk2yb_ownership_no_longer_mega_merges_same_page_assets(tmp_path: Pat
     assert 3 in matched or 3 in ambiguous, "Fig 3 should be captured"
 
 
+def test_dwqqk2yb_figure3_is_fully_owned_not_merely_captured(tmp_path: Path) -> None:
+    """Gate 2 red regression: DW Figure 3 must be strictly matched, not left ambiguous."""
+    result = replay_production_pipeline("DWQQK2YB", tmp_path)
+    reader_payload = result["reader_payload"]
+    matched, ambiguous = _reader_figure_index(reader_payload)
+
+    assert 3 in matched, "Fig 3 should be matched, not left ambiguous"
+    assert 3 not in ambiguous, "Fig 3 ambiguity is no longer acceptable after Gate 2"
+
+
 def test_6fgdbfqn_same_page_narrow_caption_ownership(tmp_path: Path) -> None:
     key = "6FGDBFQN"
     expectations = _load_expectations(key)