diff --git a/docs/superpowers/plans/2026-07-04-ocr-pipeline-v3-implementation-plan.md b/docs/superpowers/plans/2026-07-04-ocr-pipeline-v3-implementation-plan.md new file mode 100644 index 00000000..be151004 --- /dev/null +++ b/docs/superpowers/plans/2026-07-04-ocr-pipeline-v3-implementation-plan.md @@ -0,0 +1,675 @@ +# OCR Pipeline V3 Split Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Split the current monolithic normalize path into `pre_match_normalize(...)` and `post_match_normalize(...)`, keep the public `block["role"]` contract intact, and gate the entire v3 path behind `OCR_PIPELINE_V3` so legacy callers stay untouched by default. + +**Architecture:** Keep the legacy path as the default. Add a seed-only mode to `build_structured_blocks(...)`, then let the v3 path run pre-match normalization, figure/table matching, and post-match normalization in `ocr.py`. To keep blast radius low, the first v3 implementation uses shadow calls into `normalize_document_structure(...)` to compute candidate/final roles while preserving the new sequencing and matching contract. + +**Tech Stack:** Python 3, existing PaperForge OCR workers, `DocumentStructure`, `ocr_tail_settlement.py`, pytest, environment toggle via `OCR_PIPELINE_V3` + +## Global Constraints + +- Workstream C only. Workstreams A and B are already landed on this branch and must stay green. +- `block["role"]` remains the final public role field. +- Under v3, matching code must not require final `role`; it must work from `seed_role`, `role_candidate`, `raw_label`, `bbox`, `zone`, and ownership evidence when present. +- The entire v3 path is behind `OCR_PIPELINE_V3`; legacy behavior is the default and must remain byte-for-byte compatible for existing callers. +- Do not rewrite figure/table pairing logic in this workstream. Change only the inputs those builders accept. +- Do not change object writeback sequencing from Workstream A. `apply_object_writebacks(...)` remains after figure/table inventory exists. +- Reuse the existing `settle_tail_and_backmatter(...)` module from Workstream B. Do not re-inline tail settlement. +- If v3 parity is not green, stop at the gate. Do not force-switch callers to v3. + +--- + +## File Structure + +- **Create:** `paperforge/worker/ocr_pre_match_normalize.py` + - v3 candidate-only normalization. Preserves `role = seed_role`, writes `role_candidate` from a shadow normalize pass. +- **Create:** `paperforge/worker/ocr_post_match_normalize.py` + - v3 final role commit after figure/table inventories exist. Reuses tail settlement from Workstream B. +- **Modify:** `paperforge/worker/ocr_blocks.py` + - add `normalize_mode="seed_only" | "legacy"` to `build_structured_blocks(...)` + - return rows/doc before legacy normalize when v3 wants seed-only rows +- **Modify:** `paperforge/worker/ocr.py` + - add `_ocr_pipeline_v3_enabled()` + - branch between legacy path and v3 pre-match → inventory build → post-match path +- **Modify:** `paperforge/worker/ocr_figures.py` + - add a helper that resolves match-time role from `role_candidate` / `role` / `seed_role` + - replace role checks used by figure matching with the helper +- **Modify:** `paperforge/worker/ocr_tables.py` + - same candidate-role helper approach for table matching +- **Create:** `tests/test_ocr_pipeline_v3.py` + - toggle tests, pre-match tests, post-match tests, and one figure/table contract test each + +--- + +### Task 1: C0 — Add the v3 toggle and seed-only build path + +**Files:** +- Create: `paperforge/worker/ocr_pre_match_normalize.py` +- Create: `paperforge/worker/ocr_post_match_normalize.py` +- Modify: `paperforge/worker/ocr_blocks.py` +- Modify: `paperforge/worker/ocr.py` +- Create: `tests/test_ocr_pipeline_v3.py` + +**Interfaces:** +- Consumes: + - current seed-row construction in `build_structured_blocks(...)` + - current top-level OCR flow in `ocr.py` +- Produces: + - `_ocr_pipeline_v3_enabled() -> bool` + - `build_structured_blocks(..., normalize_mode: str = "legacy")` + - `pre_match_normalize(rows: list[dict], *, source_frontmatter_anchors: dict | None = None, document_structure: DocumentStructure | None = None) -> tuple[list[dict], DocumentStructure]` + - `post_match_normalize(rows: list[dict], figure_inventory: dict, table_inventory: dict, *, document_structure: DocumentStructure, source_frontmatter_anchors: dict | None = None) -> tuple[list[dict], DocumentStructure]` + +- [ ] **Step 1: Write the failing tests** + +```python +# tests/test_ocr_pipeline_v3.py +from __future__ import annotations + + +def test_ocr_pipeline_v3_enabled_defaults_false(monkeypatch) -> None: + from paperforge.worker.ocr import _ocr_pipeline_v3_enabled + + monkeypatch.delenv("OCR_PIPELINE_V3", raising=False) + + assert _ocr_pipeline_v3_enabled() is False + + +def test_ocr_pipeline_v3_enabled_truthy(monkeypatch) -> None: + from paperforge.worker.ocr import _ocr_pipeline_v3_enabled + + monkeypatch.setenv("OCR_PIPELINE_V3", "1") + + assert _ocr_pipeline_v3_enabled() is True + + +def test_build_structured_blocks_seed_only_skips_legacy_normalize(monkeypatch) -> None: + import paperforge.worker.ocr_document as ocr_document + from paperforge.worker.ocr_blocks import build_structured_blocks + + def boom(*args, **kwargs): + raise AssertionError("legacy normalize should not run in seed_only mode") + + monkeypatch.setattr(ocr_document, "normalize_document_structure", boom) + + raw_blocks = [ + { + "block_id": "r1", + "page": 1, + "raw_label": "text", + "text": "Minimal body text.", + "bbox": [100, 100, 420, 140], + } + ] + + rows, doc = build_structured_blocks(raw_blocks, normalize_mode="seed_only") + + assert len(rows) == 1 + assert rows[0]["role"] == rows[0]["seed_role"] + assert doc is not None +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: +`python -m pytest tests/test_ocr_pipeline_v3.py::test_ocr_pipeline_v3_enabled_defaults_false tests/test_ocr_pipeline_v3.py::test_ocr_pipeline_v3_enabled_truthy tests/test_ocr_pipeline_v3.py::test_build_structured_blocks_seed_only_skips_legacy_normalize -v --tb=short` + +Expected: FAIL because `_ocr_pipeline_v3_enabled`, `normalize_mode`, and the new modules do not exist yet. + +- [ ] **Step 3: Write minimal implementation** + +```python +# paperforge/worker/ocr.py +import os + + +def _ocr_pipeline_v3_enabled() -> bool: + return os.environ.get("OCR_PIPELINE_V3", "").strip().lower() in {"1", "true", "yes", "on"} +``` + +```python +# paperforge/worker/ocr_blocks.py +def build_structured_blocks( + raw_blocks: list[dict], + source_metadata: dict | None = None, + structure_output_dir: str | Path | None = None, + normalize_mode: str = "legacy", +) -> tuple[list[dict], DocumentStructure]: + ... + body_family_anchor = discover_body_family_anchor(rows, page_count=total_pages) + doc_structure = DocumentStructure(body_family_anchor=body_family_anchor) + ... + if normalize_mode == "seed_only": + return rows, doc_structure + ... + doc_structure, rows = normalize_document_structure(rows, source_frontmatter_anchors=_sfm_anchors) + ... + settle_tail_and_backmatter(structured_blocks=rows, document_structure=doc_structure) + return rows, doc_structure +``` + +```python +# paperforge/worker/ocr_pre_match_normalize.py +from __future__ import annotations + +from paperforge.worker.ocr_document import DocumentStructure + + +def pre_match_normalize( + rows: list[dict], + *, + source_frontmatter_anchors: dict | None = None, + document_structure: DocumentStructure | None = None, +) -> tuple[list[dict], DocumentStructure]: + return rows, document_structure or DocumentStructure() +``` + +```python +# paperforge/worker/ocr_post_match_normalize.py +from __future__ import annotations + +from paperforge.worker.ocr_document import DocumentStructure + + +def post_match_normalize( + rows: list[dict], + figure_inventory: dict, + table_inventory: dict, + *, + document_structure: DocumentStructure, + source_frontmatter_anchors: dict | None = None, +) -> tuple[list[dict], DocumentStructure]: + return rows, document_structure +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: +`python -m pytest tests/test_ocr_pipeline_v3.py::test_ocr_pipeline_v3_enabled_defaults_false tests/test_ocr_pipeline_v3.py::test_ocr_pipeline_v3_enabled_truthy tests/test_ocr_pipeline_v3.py::test_build_structured_blocks_seed_only_skips_legacy_normalize -v --tb=short` + +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/worker/ocr.py paperforge/worker/ocr_blocks.py paperforge/worker/ocr_pre_match_normalize.py paperforge/worker/ocr_post_match_normalize.py tests/test_ocr_pipeline_v3.py +git commit -m "feat: add OCR pipeline v3 toggle and seed-only path" +``` + +--- + +### Task 2: C1 — Implement pre-match normalize and the matching contract + +**Files:** +- Modify: `paperforge/worker/ocr_pre_match_normalize.py` +- Modify: `paperforge/worker/ocr_figures.py` +- Modify: `paperforge/worker/ocr_tables.py` +- Modify: `tests/test_ocr_pipeline_v3.py` + +**Interfaces:** +- Consumes: + - seed-only rows from `build_structured_blocks(..., normalize_mode="seed_only")` + - legacy `normalize_document_structure(...)` on a shadow copy +- Produces: + - `role_candidate` filled from shadow normalize results + - figure/table builders that accept `role_candidate` instead of requiring final `role` + +- [ ] **Step 1: Write the failing tests** + +```python +# tests/test_ocr_pipeline_v3.py +from __future__ import annotations + + +def test_pre_match_normalize_preserves_public_role_and_sets_role_candidate(monkeypatch) -> None: + from paperforge.worker.ocr_document import DocumentStructure + import paperforge.worker.ocr_pre_match_normalize as pre + + def fake_normalize(rows, source_frontmatter_anchors=None, pdf_path=None): + shadow_rows = [dict(r) for r in rows] + shadow_rows[0]["role"] = "figure_caption_candidate" + return DocumentStructure(), shadow_rows + + monkeypatch.setattr(pre, "normalize_document_structure", fake_normalize) + + rows = [ + { + "block_id": "c1", + "page": 1, + "role": "body_paragraph", + "seed_role": "body_paragraph", + "text": "Figure 1. Example caption text.", + "bbox": [100, 100, 520, 160], + } + ] + + out_rows, doc = pre.pre_match_normalize(rows, source_frontmatter_anchors=None, document_structure=DocumentStructure()) + + assert out_rows[0]["role"] == "body_paragraph" + assert out_rows[0]["role_candidate"] == "figure_caption_candidate" + assert doc is not None + + +def test_figure_inventory_accepts_role_candidate_caption_blocks() -> None: + from paperforge.worker.ocr_figures import build_figure_inventory_vnext + + blocks = [ + { + "block_id": "cap1", + "page": 2, + "role": "body_paragraph", + "role_candidate": "figure_caption_candidate", + "seed_role": "figure_caption", + "raw_label": "figure_title", + "zone": "display_zone", + "style_family": "legend_like", + "marker_signature": {"type": "figure_number"}, + "text": "Figure 1. Example caption", + "bbox": [100, 100, 420, 140], + }, + { + "block_id": "asset1", + "page": 2, + "role": "media_asset", + "role_candidate": "media_asset", + "seed_role": "media_asset", + "raw_label": "image", + "zone": "display_zone", + "text": "", + "bbox": [100, 160, 420, 380], + }, + ] + + inv = build_figure_inventory_vnext(blocks) + + assert inv["matched_figures"] or inv["figure_legends"] + + +def test_table_inventory_accepts_role_candidate_caption_blocks() -> None: + from paperforge.worker.ocr_tables import build_table_inventory_vnext + + blocks = [ + { + "block_id": "cap1", + "page": 3, + "role": "body_paragraph", + "role_candidate": "table_caption_candidate", + "seed_role": "table_caption", + "raw_label": "figure_title", + "zone": "display_zone", + "style_family": "table_caption_like", + "marker_signature": {"type": "table_number"}, + "text": "Table 1. Outcomes", + "bbox": [100, 100, 420, 140], + }, + { + "block_id": "asset1", + "page": 3, + "role": "media_asset", + "role_candidate": "media_asset", + "seed_role": "media_asset", + "raw_label": "table", + "zone": "display_zone", + "text": "", + "bbox": [100, 160, 420, 380], + }, + ] + + inv = build_table_inventory_vnext(blocks) + + assert inv["tables"] +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: +`python -m pytest tests/test_ocr_pipeline_v3.py::test_pre_match_normalize_preserves_public_role_and_sets_role_candidate tests/test_ocr_pipeline_v3.py::test_figure_inventory_accepts_role_candidate_caption_blocks tests/test_ocr_pipeline_v3.py::test_table_inventory_accepts_role_candidate_caption_blocks -v --tb=short` + +Expected: FAIL because `pre_match_normalize(...)` does not populate `role_candidate`, and figure/table builders still key too heavily off final `role`. + +- [ ] **Step 3: Write minimal implementation** + +```python +# paperforge/worker/ocr_pre_match_normalize.py +from __future__ import annotations + +from paperforge.worker.ocr_document import DocumentStructure, normalize_document_structure + + +def pre_match_normalize( + rows: list[dict], + *, + source_frontmatter_anchors: dict | None = None, + document_structure: DocumentStructure | None = None, +) -> tuple[list[dict], DocumentStructure]: + live_rows = [dict(row) for row in rows] + shadow_doc, shadow_rows = normalize_document_structure( + [dict(row) for row in rows], + source_frontmatter_anchors=source_frontmatter_anchors, + ) + for live, shadow in zip(live_rows, shadow_rows): + live["role_candidate"] = shadow.get("role") or shadow.get("seed_role") or live.get("seed_role") + live["zone"] = shadow.get("zone", live.get("zone")) + live["style_family"] = shadow.get("style_family", live.get("style_family")) + live["marker_signature"] = shadow.get("marker_signature", live.get("marker_signature")) + live["render_default"] = live.get("render_default", True) + live["role"] = live.get("seed_role") or live.get("role") + return live_rows, document_structure or shadow_doc +``` + +```python +# paperforge/worker/ocr_figures.py + +def _match_role(block: dict) -> str: + return str(block.get("role_candidate") or block.get("role") or block.get("seed_role") or "") +``` + +```python +# paperforge/worker/ocr_figures.py +# replace match-time role reads like: +role = str(block.get("role") or "") +# with: +role = _match_role(block) +``` + +Replace that pattern anywhere figure matching decides whether a block is a caption candidate, figure asset, or excluded demoted body paragraph. Do not change post-writeback render-only checks. + +```python +# paperforge/worker/ocr_tables.py + +def _match_role(block: dict) -> str: + return str(block.get("role_candidate") or block.get("role") or block.get("seed_role") or "") +``` + +```python +# paperforge/worker/ocr_tables.py +# replace match-time role reads like: +role = str(block.get("role", "") or "") +# with: +role = _match_role(block) +``` + +Replace only the role reads that affect caption/asset selection for matching. Leave write-back and render-only code alone. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: +`python -m pytest tests/test_ocr_pipeline_v3.py::test_pre_match_normalize_preserves_public_role_and_sets_role_candidate tests/test_ocr_pipeline_v3.py::test_figure_inventory_accepts_role_candidate_caption_blocks tests/test_ocr_pipeline_v3.py::test_table_inventory_accepts_role_candidate_caption_blocks -v --tb=short` + +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/worker/ocr_pre_match_normalize.py paperforge/worker/ocr_figures.py paperforge/worker/ocr_tables.py tests/test_ocr_pipeline_v3.py +git commit -m "feat: add pre-match normalize and candidate-role matching" +``` + +--- + +### Task 3: C2 — Implement post-match normalize and top-level v3 orchestration + +**Files:** +- Modify: `paperforge/worker/ocr_post_match_normalize.py` +- Modify: `paperforge/worker/ocr.py` +- Modify: `tests/test_ocr_pipeline_v3.py` + +**Interfaces:** +- Consumes: + - pre-match rows with `role_candidate` + - `figure_inventory` and `table_inventory` + - existing `DocumentStructure` +- Produces: + - final public `role` + - reuse of `settle_tail_and_backmatter(...)` after post-match commit + - v3 sequencing in `ocr.py` + +- [ ] **Step 1: Write the failing tests** + +```python +# tests/test_ocr_pipeline_v3.py +from __future__ import annotations + + +def test_post_match_normalize_commits_shadow_role_back_to_public_role(monkeypatch) -> None: + from paperforge.worker.ocr_document import DocumentStructure + import paperforge.worker.ocr_post_match_normalize as post + + def fake_normalize(rows, source_frontmatter_anchors=None, pdf_path=None): + shadow_rows = [dict(r) for r in rows] + shadow_rows[0]["role"] = "figure_caption" + shadow_rows[0]["role_source"] = "shadow_post_match" + return DocumentStructure(), shadow_rows + + monkeypatch.setattr(post, "normalize_document_structure", fake_normalize) + + rows = [ + { + "block_id": "c1", + "page": 1, + "role": "body_paragraph", + "seed_role": "body_paragraph", + "role_candidate": "figure_caption_candidate", + "text": "Figure 1. Example caption text.", + "bbox": [100, 100, 520, 160], + } + ] + + out_rows, doc = post.post_match_normalize( + rows, + {"matched_figures": []}, + {"tables": []}, + document_structure=DocumentStructure(), + source_frontmatter_anchors=None, + ) + + assert out_rows[0]["role"] == "figure_caption" + assert out_rows[0]["role_candidate"] == "figure_caption_candidate" + assert out_rows[0]["role_source"] == "shadow_post_match" + assert doc is not None +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: +`python -m pytest tests/test_ocr_pipeline_v3.py::test_post_match_normalize_commits_shadow_role_back_to_public_role -v --tb=short` + +Expected: FAIL because `post_match_normalize(...)` is still a no-op. + +- [ ] **Step 3: Write minimal implementation** + +```python +# paperforge/worker/ocr_post_match_normalize.py +from __future__ import annotations + +from paperforge.worker.ocr_document import DocumentStructure, normalize_document_structure +from paperforge.worker.ocr_tail_settlement import settle_tail_and_backmatter + + +def post_match_normalize( + rows: list[dict], + figure_inventory: dict, + table_inventory: dict, + *, + document_structure: DocumentStructure, + source_frontmatter_anchors: dict | None = None, +) -> tuple[list[dict], DocumentStructure]: + live_rows = [dict(row) for row in rows] + shadow_doc, shadow_rows = normalize_document_structure( + [dict(row) for row in rows], + source_frontmatter_anchors=source_frontmatter_anchors, + ) + for live, shadow in zip(live_rows, shadow_rows): + live["role"] = shadow.get("role", live.get("role")) + live["role_source"] = shadow.get("role_source", live.get("role_source")) + live["role_confidence"] = shadow.get("role_confidence", live.get("role_confidence")) + live["role_candidate"] = live.get("role_candidate") or shadow.get("role") + live["render_default"] = shadow.get("render_default", live.get("render_default")) + live["index_default"] = shadow.get("index_default", live.get("index_default")) + settle_tail_and_backmatter(structured_blocks=live_rows, document_structure=shadow_doc) + return live_rows, shadow_doc +``` + +```python +# paperforge/worker/ocr.py +from paperforge.worker.ocr_pre_match_normalize import pre_match_normalize +from paperforge.worker.ocr_post_match_normalize import post_match_normalize + +... +normalize_mode = "seed_only" if _ocr_pipeline_v3_enabled() else "legacy" +structured, doc_structure = build_structured_blocks( + all_raw_blocks, + source_metadata=source_meta, + structure_output_dir=artifacts.blocks_structured.parent, + normalize_mode=normalize_mode, +) +if _ocr_pipeline_v3_enabled(): + structured, doc_structure = pre_match_normalize( + structured, + source_frontmatter_anchors=getattr(doc_structure, "source_frontmatter_anchors", None), + document_structure=doc_structure, + ) +write_structured_blocks_jsonl(artifacts.blocks_structured, structured) +... +figure_inventory = build_figure_inventory(structured) +... +table_inventory = build_table_inventory(structured) +if _ocr_pipeline_v3_enabled(): + structured, doc_structure = post_match_normalize( + structured, + figure_inventory, + table_inventory, + document_structure=doc_structure, + source_frontmatter_anchors=getattr(doc_structure, "source_frontmatter_anchors", None), + ) +# object writeback stays here +apply_object_writebacks( + structured_blocks=structured, + figure_inventory=figure_inventory, + table_inventory=table_inventory, +) +``` + +The top-level order under v3 is now: +1. seed-only block build +2. pre-match normalize +3. figure/table inventory build +4. post-match normalize +5. object writeback + +The legacy path stays unchanged. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: +`python -m pytest tests/test_ocr_pipeline_v3.py::test_post_match_normalize_commits_shadow_role_back_to_public_role -v --tb=short` + +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/worker/ocr.py paperforge/worker/ocr_post_match_normalize.py tests/test_ocr_pipeline_v3.py +git commit -m "feat: wire OCR pipeline v3 pre-match and post-match flow" +``` + +--- + +### Task 4: C3 — Parity gate for v3 + +**Files:** +- Modify: `tests/test_ocr_pipeline_v3.py` +- Reuse: `tests/test_ocr_tail_settlement.py` +- Reuse: `tests/test_ocr_object_writeback.py` +- Reuse: `tests/test_ocr_rendering.py` +- Reuse: `tests/test_appendix_figure_numbering.py` + +**Interfaces:** +- Consumes: + - C0/C1/C2 v3 path + - A/B regressions +- Produces: + - go / no-go for turning on v3 in any real caller + +- [ ] **Step 1: Add one v3 integration smoke test** + +```python +# tests/test_ocr_pipeline_v3.py +from __future__ import annotations + + +def test_build_structured_blocks_legacy_default_still_matches_seed_contract() -> None: + from paperforge.worker.ocr_blocks import build_structured_blocks + + raw_blocks = [ + { + "block_id": "r1", + "page": 1, + "raw_label": "text", + "text": "Minimal body text.", + "bbox": [100, 100, 420, 140], + } + ] + + rows, _ = build_structured_blocks(raw_blocks) + + assert rows[0]["role"] + assert rows[0]["seed_role"] +``` + +- [ ] **Step 2: Run the v3-focused suite** + +Run: +`python -m pytest tests/test_ocr_pipeline_v3.py tests/test_ocr_tail_settlement.py -q --tb=line` + +Expected: PASS + +- [ ] **Step 3: Run cross-workstream guard suite** + +Run: +`python -m pytest tests/test_ocr_object_writeback.py tests/test_ocr_rendering.py::test_body_renderer_skips_consumed_object_owned_blocks tests/test_appendix_figure_numbering.py -q --tb=line` + +Expected: PASS + +- [ ] **Step 4: Run tail/backmatter parity suite** + +Run: +`python -m pytest tests/test_ocr_rendering.py::test_tail_zone_noise_band_guard tests/test_ocr_rendering.py::test_tail_zone_supplementary_material_not_noise tests/test_ocr_rendering.py::test_tail_candidate_overreach_does_not_absorb_late_body tests/test_ocr_rendering.py::test_cross_page_funding_continuation_preserves_order tests/test_ocr_rendering.py::test_mixed_tail_page_keeps_late_body_out_of_funding_and_attaches_real_funding tests/test_ocr_rendering.py::test_backmatter_boundary_normalizes_child_sections_before_references -q --tb=line` + +Expected: PASS + +- [ ] **Step 5: Stop rule** + +Decision rule: +- If all C tests and all A/B regressions are green, Workstream C is complete but still optional behind `OCR_PIPELINE_V3`. +- Do **not** flip the default to v3 in this workstream. +- Any future default-on change requires a separate micro-plan after corpus-diff review. + +- [ ] **Step 6: Commit** + +```bash +git add tests/test_ocr_pipeline_v3.py +git commit -m "test: add OCR pipeline v3 parity gate" +``` + +--- + +## Self-Review + +- **Spec coverage:** This plan covers Workstream C from `2026-07-04-ocr-pipeline-deepening-design.md`: pre-match module, post-match module, role-field contract, matching contract, and behind-toggle migration rule. +- **Grounding against current code:** The plan matches the current placement of figure/table inventory building in `ocr.py` and the current seed-row construction in `ocr_blocks.py`. That is why the v3 split is staged across both files rather than only inside `normalize_document_structure(...)`. +- **Placeholder scan:** No `TBD` / `TODO` implementation placeholders remain. +- **Scope control:** The plan keeps v3 behind `OCR_PIPELINE_V3` and explicitly does not flip the default. +- **Type consistency:** `role` stays final public role; `role_candidate` is candidate-only; `seed_role` remains the early guess. + +## Execution Handoff + +**Plan complete and saved to `docs/superpowers/plans/2026-07-04-ocr-pipeline-v3-implementation-plan.md`. Two execution options:** + +**1. Subagent-Driven (recommended)** - I dispatch a fresh subagent per task, review between tasks, fast iteration + +**2. Inline Execution** - Execute tasks in this session using executing-plans, batch execution with checkpoints + +**Which approach?** diff --git a/docs/superpowers/plans/2026-07-04-ocr-tail-settlement-implementation-plan.md b/docs/superpowers/plans/2026-07-04-ocr-tail-settlement-implementation-plan.md new file mode 100644 index 00000000..a455db41 --- /dev/null +++ b/docs/superpowers/plans/2026-07-04-ocr-tail-settlement-implementation-plan.md @@ -0,0 +1,662 @@ +# OCR Tail Settlement Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Extract the current tail/body/backmatter settlement logic into a dedicated module, preserve current behavior while removing cross-module leakage between `normalize_document_structure()` and `build_structured_blocks()`, then add a settlement report and a hard corpus-diff gate before any policy change. + +**Architecture:** Keep the current OCR pipeline order unchanged. First move the existing tail settlement implementation into `paperforge/worker/ocr_tail_settlement.py` and route current call sites through it without changing behavior. Then introduce `TailSettlementReport` so normalize-phase promotion/exclusion and build-phase restore both write into one report attached to `DocumentStructure`. B2 is a verification gate, not a policy rewrite: if the diff is not green, stop. + +**Tech Stack:** Python 3, existing PaperForge OCR workers, `DocumentStructure`, pytest, current tail/backmatter render regressions in `tests/test_ocr_rendering.py` + +## Global Constraints + +- Workstream B only: implement **B0 / B1 / B2**; do not touch Workstream C. +- Workstream A is already landed on this branch (`feat/ocr-tail-settlement`, `964e05b`); do not regress `ocr_object_writeback` behavior. +- B0/B1 are **behavior-preserving extraction only**. Do **not** change tail/backmatter/reference policy in those tasks. +- Do **not** reorder the main OCR pipeline. +- Do **not** change figure/table ownership behavior. +- Keep `block["role"]` as the current public final role contract. +- Because `DocumentStructure` is only materialized late in `normalize_document_structure()` (`ocr_document.py:5678` today), normalize-phase extraction must use module-local helpers first; the public `settle_tail_and_backmatter(...)` wrapper only owns the post-normalize pass until B1 attaches the shared report. +- B2 may only adjust policy after the corpus diff is green. If the diff is not green, stop after instrumentation/reporting and do **not** commit any policy change. + +--- + +## File Structure + +- **Create:** `paperforge/worker/ocr_tail_settlement.py` + - Owns the extracted tail settlement helpers and the public `settle_tail_and_backmatter(...)` wrapper. + - Hosts `promote_backmatter_heading_candidates(...)`, `exclude_tail_nonref_from_body_flow(...)`, `restore_numbered_body_from_tail_hold(...)`. + - Hosts `TailSettlementReport` in B1. +- **Modify:** `paperforge/worker/ocr_document.py` + - Replace the inline backmatter-candidate promotion loop and direct tail exclusion call with imports from `ocr_tail_settlement.py`. + - Attach the shared `TailSettlementReport` to `DocumentStructure` in B1. +- **Modify:** `paperforge/worker/ocr_blocks.py` + - Replace the direct `_exclude_tail_nonref_from_body_flow(...)` / `_restore_numbered_body_from_tail_hold(...)` imports and calls with `settle_tail_and_backmatter(...)`. +- **Create:** `tests/test_ocr_tail_settlement.py` + - Unit tests for B0/B1 extraction behavior and report contents. +- **Reuse existing tests:** `tests/test_ocr_rendering.py` + - Existing tail/backmatter regressions remain the acceptance suite for behavior-preserving extraction. + +--- + +### Task 1: B0 — Extract the existing tail settlement seam into one deep module + +**Files:** +- Create: `paperforge/worker/ocr_tail_settlement.py` +- Modify: `paperforge/worker/ocr_document.py` +- Modify: `paperforge/worker/ocr_blocks.py` +- Create: `tests/test_ocr_tail_settlement.py` + +**Interfaces:** +- Consumes: + - `structured_blocks: list[dict]` + - optional `document_structure: object | None` + - current block fields already produced by `normalize_document_structure()` (`role`, `seed_role`, `zone`, `style_family`, `marker_signature`, `text`, `bbox`) +- Produces: + - `promote_backmatter_heading_candidates(blocks: list[dict]) -> None` + - `exclude_tail_nonref_from_body_flow(blocks: list[dict]) -> None` + - `restore_numbered_body_from_tail_hold(blocks: list[dict]) -> None` + - `settle_tail_and_backmatter(*, structured_blocks: list[dict], document_structure: object | None = None) -> dict` + - behavior-preserving delegation of the current helper order + +- [ ] **Step 1: Write the failing tests** + +```python +# tests/test_ocr_tail_settlement.py +from __future__ import annotations + + +def test_promote_backmatter_heading_candidates_promotes_same_page_followers() -> None: + from paperforge.worker.ocr_tail_settlement import promote_backmatter_heading_candidates + + blocks = [ + { + "block_id": "h1", + "page": 10, + "role": "backmatter_heading_candidate", + "seed_role": "backmatter_heading_candidate", + "text": "Funding", + "bbox": [100, 100, 260, 130], + }, + { + "block_id": "b1", + "page": 10, + "role": "body_paragraph", + "seed_role": "body_paragraph", + "text": "This work was supported by Grant A.", + "bbox": [100, 150, 520, 220], + }, + { + "block_id": "b2", + "page": 10, + "role": "body_paragraph", + "seed_role": "body_paragraph", + "text": "The funders had no role in study design.", + "bbox": [100, 230, 520, 300], + }, + ] + + promote_backmatter_heading_candidates(blocks) + + assert blocks[0]["role"] == "backmatter_heading" + assert blocks[1]["role"] == "backmatter_body" + assert blocks[2]["role"] == "backmatter_body" + + +def test_settle_tail_and_backmatter_preserves_tail_hold_restore() -> None: + from paperforge.worker.ocr_tail_settlement import settle_tail_and_backmatter + + blocks = [ + { + "block_id": "h1", + "page": 11, + "role": "section_heading", + "zone": "tail_nonref_hold_zone", + "style_family": "heading_like", + "marker_signature": {"type": "heading_numbered"}, + "text": "5. Conclusions", + }, + { + "block_id": "b1", + "page": 11, + "role": "body_paragraph", + "seed_role": "body_paragraph", + "zone": "tail_nonref_hold_zone", + "text": "Funding: supported by Grant A.", + }, + { + "block_id": "b2", + "page": 11, + "role": "backmatter_body", + "zone": "tail_nonref_hold_zone", + "style_family": "body_like", + "marker_signature": {"type": "none"}, + "text": "This section returns to the main conclusions.", + }, + ] + + report = settle_tail_and_backmatter(structured_blocks=blocks, document_structure=None) + + assert blocks[1]["role"] == "backmatter_body" + assert blocks[2]["role"] == "body_paragraph" + assert report["applied_count"] == 0 +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: +`python -m pytest tests/test_ocr_tail_settlement.py::test_promote_backmatter_heading_candidates_promotes_same_page_followers tests/test_ocr_tail_settlement.py::test_settle_tail_and_backmatter_preserves_tail_hold_restore -v --tb=short` + +Expected: FAIL with `ModuleNotFoundError: No module named 'paperforge.worker.ocr_tail_settlement'` + +- [ ] **Step 3: Write minimal implementation** + +```python +# paperforge/worker/ocr_tail_settlement.py +from __future__ import annotations + +import re + +from paperforge.worker.ocr_decisions import record_decision + +_BACKMATTER_TITLE_DENY_LIST: frozenset[str] = frozenset() +_BACKMATTER_BODY_SIGNALS = re.compile( + r"\b(?:declare|conflict|interest|funding|support|grant|author|contribut|" + r"acknowledge|thank|ethic|review|approv|consent|availab|data|material|" + r"competing|financial|disclos|report|none|no conflict|nothing to declare)\b", + re.IGNORECASE, +) + + +def _block_text(block: dict) -> str: + return str(block.get("text") or block.get("block_content") or "") + + +def _next_nonempty_block_same_page(blocks: list[dict], idx: int) -> dict | None: + page = blocks[idx].get("page") + if page is None: + return None + for j in range(idx + 1, len(blocks)): + if blocks[j].get("page") != page: + break + if _block_text(blocks[j]).strip(): + return blocks[j] + return None + + +def _canonical_section_text(block: dict) -> str: + text = _block_text(block).strip().lower() + if re.match(r"^(?:\w\s)+\w$", text): + text = re.sub(r"\s+", "", text) + return text + + +def _looks_like_tail_body(block: dict) -> bool: + text = _block_text(block).strip() + if not text: + return False + words = text.split() + if len(words) > 80: + return False + role = block.get("role", "") + if role in {"section_heading", "subsection_heading", "sub_subsection_heading"}: + return False + return bool(_BACKMATTER_BODY_SIGNALS.search(text)) + + +def _looks_like_backmatter_body_text(text: str) -> bool: + lower = text.lower() + markers = ( + "conflict of interest", + "declaration", + "publisher", + "author contributions", + "funding", + "acknowledg", + "data availability", + "supplement", + "ethics", + "copyright", + ) + return any(marker in lower for marker in markers) + + +def promote_backmatter_heading_candidates(blocks: list[dict]) -> None: + for idx, block in enumerate(blocks): + is_candidate = ( + block.get("role") == "backmatter_heading_candidate" + or block.get("seed_role") == "backmatter_heading_candidate" + ) + if not is_candidate or block.get("role") == "backmatter_heading": + continue + next_body = _next_nonempty_block_same_page(blocks, idx) + if next_body and next_body.get("role") in {"body_paragraph", "backmatter_body"} and _looks_like_tail_body(next_body): + old_role = block.get("role") + block["role"] = "backmatter_heading" + block.setdefault("role_confidence", 0.6) + if old_role != block["role"]: + record_decision( + block, + stage="backmatter_candidate_promotion", + old_role=old_role, + new_role=block["role"], + reason="backmatter heading candidate promoted: followed by tail-like body on same page", + ) + for j in range(idx + 1, len(blocks)): + if blocks[j].get("page") != block.get("page"): + break + if blocks[j].get("role") == "body_paragraph": + old_follower_role = blocks[j].get("role") + blocks[j]["role"] = "backmatter_body" + blocks[j].setdefault("role_confidence", 0.6) + if old_follower_role != blocks[j]["role"]: + record_decision( + blocks[j], + stage="backmatter_candidate_promotion", + old_role=old_follower_role, + new_role=blocks[j]["role"], + reason="follower body converted to backmatter_body under confirmed heading", + ) + break + + +def exclude_tail_nonref_from_body_flow(blocks: list[dict]) -> None: + for block in blocks: + effective_role = block.get("role") + if effective_role == "unassigned": + effective_role = block.get("seed_role") + if effective_role != "body_paragraph": + continue + if block.get("zone") != "tail_nonref_hold_zone": + continue + text = _block_text(block) + if not _looks_like_backmatter_body_text(text): + continue + old_role = block.get("role") + block["role"] = "backmatter_body" + if block.get("seed_role") == "body_paragraph": + block["seed_role"] = "backmatter_body" + if old_role != block["role"]: + record_decision( + block, + stage="tail_nonref_exclusion", + old_role=old_role, + new_role=block["role"], + reason="tail non-reference body block excluded from body flow", + ) + block.setdefault("evidence", []).append("tail_nonref_hold_zone excluded from body flow") + + +def restore_numbered_body_from_tail_hold(blocks: list[dict]) -> None: + active_numbered_body = False + for block in blocks: + role = block.get("role") + text = _canonical_section_text(block) + marker_type = str(((block.get("marker_signature") or {}).get("type")) or "none") + + if role in {"reference_heading", "backmatter_heading", "backmatter_boundary_heading"}: + active_numbered_body = False + continue + + if role in {"section_heading", "subsection_heading", "sub_subsection_heading"}: + active_numbered_body = ( + ( + marker_type in {"heading_numbered", "heading_arabic", "heading_decimal", "heading_roman", "heading_alpha"} + or (block.get("zone") == "tail_nonref_hold_zone" and str(block.get("style_family") or "") == "heading_like") + ) + and text not in _BACKMATTER_TITLE_DENY_LIST + ) + continue + + if role == "backmatter_body" and active_numbered_body: + block["role"] = "body_paragraph" + + +def settle_tail_and_backmatter(*, structured_blocks: list[dict], document_structure: object | None = None) -> dict: + exclude_tail_nonref_from_body_flow(structured_blocks) + restore_numbered_body_from_tail_hold(structured_blocks) + return { + "promoted_backmatter_heading_ids": [], + "converted_to_backmatter_body_ids": [], + "restored_body_paragraph_ids": [], + "applied_count": 0, + } +``` + +```python +# paperforge/worker/ocr_document.py +# add near the top-level imports +from paperforge.worker.ocr_tail_settlement import ( + promote_backmatter_heading_candidates, + exclude_tail_nonref_from_body_flow, +) + +# inside normalize_document_structure(...), replace the inline loop + direct helper call: +promote_backmatter_heading_candidates(blocks) +exclude_tail_nonref_from_body_flow(blocks) + +# later, replace the second direct helper call too: +exclude_tail_nonref_from_body_flow(blocks) +``` + +```python +# paperforge/worker/ocr_blocks.py +# replace: +# from paperforge.worker.ocr_document import ( +# _exclude_tail_nonref_from_body_flow, +# _restore_numbered_body_from_tail_hold, +# ) +# _exclude_tail_nonref_from_body_flow(rows) +# _restore_numbered_body_from_tail_hold(rows) + +from paperforge.worker.ocr_tail_settlement import settle_tail_and_backmatter + +settle_tail_and_backmatter(structured_blocks=rows, document_structure=doc_structure) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: +`python -m pytest tests/test_ocr_tail_settlement.py::test_promote_backmatter_heading_candidates_promotes_same_page_followers tests/test_ocr_tail_settlement.py::test_settle_tail_and_backmatter_preserves_tail_hold_restore -v --tb=short` + +Expected: PASS + +- [ ] **Step 5: Run the behavior-preserving tail regression subset** + +Run: +`python -m pytest tests/test_ocr_rendering.py::test_tail_zone_noise_band_guard tests/test_ocr_rendering.py::test_tail_zone_supplementary_material_not_noise tests/test_ocr_rendering.py::test_tail_candidate_overreach_does_not_absorb_late_body tests/test_ocr_rendering.py::test_cross_page_funding_continuation_preserves_order tests/test_ocr_rendering.py::test_mixed_tail_page_keeps_late_body_out_of_funding_and_attaches_real_funding tests/test_ocr_rendering.py::test_backmatter_boundary_normalizes_child_sections_before_references -q --tb=line` + +Expected: PASS, because B0 is extraction-only. + +- [ ] **Step 6: Commit** + +```bash +git add paperforge/worker/ocr_tail_settlement.py paperforge/worker/ocr_document.py paperforge/worker/ocr_blocks.py tests/test_ocr_tail_settlement.py +git commit -m "refactor: extract OCR tail settlement helpers" +``` + +--- + +### Task 2: B1 — Add `TailSettlementReport` and attach it to `DocumentStructure` + +**Files:** +- Modify: `paperforge/worker/ocr_tail_settlement.py` +- Modify: `paperforge/worker/ocr_document.py` +- Modify: `paperforge/worker/ocr_blocks.py` +- Modify: `tests/test_ocr_tail_settlement.py` + +**Interfaces:** +- Consumes: + - extracted B0 helper functions from `ocr_tail_settlement.py` + - `DocumentStructure` from `ocr_document.py` +- Produces: + - `TailSettlementReport` + - `DocumentStructure.tail_settlement_report` + - report accumulation across normalize-phase promotion/exclusion and build-phase restore + +- [ ] **Step 1: Write the failing tests** + +```python +# tests/test_ocr_tail_settlement.py +from __future__ import annotations + + +def test_settle_tail_and_backmatter_reports_promotions_conversions_and_restores() -> None: + from paperforge.worker.ocr_document import DocumentStructure + from paperforge.worker.ocr_tail_settlement import ( + TailSettlementReport, + promote_backmatter_heading_candidates, + exclude_tail_nonref_from_body_flow, + settle_tail_and_backmatter, + ) + + blocks = [ + { + "block_id": "h1", + "page": 10, + "role": "backmatter_heading_candidate", + "seed_role": "backmatter_heading_candidate", + "text": "Funding", + "bbox": [100, 100, 260, 130], + }, + { + "block_id": "b1", + "page": 10, + "role": "body_paragraph", + "seed_role": "body_paragraph", + "zone": "tail_nonref_hold_zone", + "text": "Funding: supported by Grant A.", + "bbox": [100, 150, 520, 220], + }, + { + "block_id": "h2", + "page": 11, + "role": "section_heading", + "zone": "tail_nonref_hold_zone", + "style_family": "heading_like", + "marker_signature": {"type": "heading_numbered"}, + "text": "5. Conclusions", + }, + { + "block_id": "b2", + "page": 11, + "role": "backmatter_body", + "zone": "tail_nonref_hold_zone", + "style_family": "body_like", + "marker_signature": {"type": "none"}, + "text": "This section returns to the main conclusions.", + }, + ] + + report = TailSettlementReport() + promote_backmatter_heading_candidates(blocks, report=report) + exclude_tail_nonref_from_body_flow(blocks, report=report) + + doc = DocumentStructure(tail_settlement_report=report) + returned = settle_tail_and_backmatter(structured_blocks=blocks, document_structure=doc) + + assert returned is report + assert report.promoted_backmatter_heading_ids == ["h1"] + assert report.converted_to_backmatter_body_ids == ["b1"] + assert report.restored_body_paragraph_ids == ["b2"] + assert doc.tail_settlement_report is report +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: +`python -m pytest tests/test_ocr_tail_settlement.py::test_settle_tail_and_backmatter_reports_promotions_conversions_and_restores -v --tb=short` + +Expected: FAIL on missing `TailSettlementReport`, missing `report=` parameters, or missing `DocumentStructure.tail_settlement_report` + +- [ ] **Step 3: Write minimal implementation** + +```python +# paperforge/worker/ocr_tail_settlement.py +from dataclasses import dataclass, field + + +@dataclass +class TailSettlementReport: + promoted_backmatter_heading_ids: list[str] = field(default_factory=list) + converted_to_backmatter_body_ids: list[str] = field(default_factory=list) + restored_body_paragraph_ids: list[str] = field(default_factory=list) + + @property + def applied_count(self) -> int: + return ( + len(self.promoted_backmatter_heading_ids) + + len(self.converted_to_backmatter_body_ids) + + len(self.restored_body_paragraph_ids) + ) +``` + +```python +# paperforge/worker/ocr_tail_settlement.py +# update helper signatures + +def promote_backmatter_heading_candidates(blocks: list[dict], report: TailSettlementReport | None = None) -> None: + ... + if old_role != block["role"] and report is not None: + report.promoted_backmatter_heading_ids.append(str(block.get("block_id"))) + ... + if old_follower_role != blocks[j]["role"] and report is not None: + report.converted_to_backmatter_body_ids.append(str(blocks[j].get("block_id"))) + + +def exclude_tail_nonref_from_body_flow(blocks: list[dict], report: TailSettlementReport | None = None) -> None: + ... + if old_role != block["role"] and report is not None: + report.converted_to_backmatter_body_ids.append(str(block.get("block_id"))) + + +def restore_numbered_body_from_tail_hold(blocks: list[dict], report: TailSettlementReport | None = None) -> None: + ... + if role == "backmatter_body" and active_numbered_body: + block["role"] = "body_paragraph" + if report is not None: + report.restored_body_paragraph_ids.append(str(block.get("block_id"))) + + +def settle_tail_and_backmatter( + *, structured_blocks: list[dict], document_structure: object | None = None +) -> TailSettlementReport: + report = getattr(document_structure, "tail_settlement_report", None) if document_structure is not None else None + if report is None: + report = TailSettlementReport() + if document_structure is not None: + document_structure.tail_settlement_report = report + exclude_tail_nonref_from_body_flow(structured_blocks, report=report) + restore_numbered_body_from_tail_hold(structured_blocks, report=report) + return report +``` + +```python +# paperforge/worker/ocr_document.py +from typing import TYPE_CHECKING +... +if TYPE_CHECKING: + from paperforge.worker.ocr_tail_settlement import TailSettlementReport +... +@dataclass +class DocumentStructure: + ... + tail_settlement_report: "TailSettlementReport | None" = None +``` + +```python +# paperforge/worker/ocr_document.py +# inside normalize_document_structure(...) +from paperforge.worker.ocr_tail_settlement import TailSettlementReport + +... +settlement_report = TailSettlementReport() +promote_backmatter_heading_candidates(blocks, report=settlement_report) +exclude_tail_nonref_from_body_flow(blocks, report=settlement_report) +... +exclude_tail_nonref_from_body_flow(blocks, report=settlement_report) +... +doc_structure = DocumentStructure( + ..., + tail_settlement_report=settlement_report, +) +``` + +```python +# paperforge/worker/ocr_blocks.py +settle_tail_and_backmatter(structured_blocks=rows, document_structure=doc_structure) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: +`python -m pytest tests/test_ocr_tail_settlement.py::test_settle_tail_and_backmatter_reports_promotions_conversions_and_restores -v --tb=short` + +Expected: PASS + +- [ ] **Step 5: Run B0 + B1 focused suite** + +Run: +`python -m pytest tests/test_ocr_tail_settlement.py tests/test_ocr_rendering.py::test_tail_zone_noise_band_guard tests/test_ocr_rendering.py::test_tail_zone_supplementary_material_not_noise tests/test_ocr_rendering.py::test_tail_candidate_overreach_does_not_absorb_late_body tests/test_ocr_rendering.py::test_cross_page_funding_continuation_preserves_order tests/test_ocr_rendering.py::test_mixed_tail_page_keeps_late_body_out_of_funding_and_attaches_real_funding tests/test_ocr_rendering.py::test_backmatter_boundary_normalizes_child_sections_before_references -q --tb=line` + +Expected: PASS + +- [ ] **Step 6: Commit** + +```bash +git add paperforge/worker/ocr_tail_settlement.py paperforge/worker/ocr_document.py paperforge/worker/ocr_blocks.py tests/test_ocr_tail_settlement.py +git commit -m "feat: add OCR tail settlement report" +``` + +--- + +### Task 3: B2 — Corpus-diff gate before any policy change + +**Files:** +- No production-file changes by default. +- Reuse: `tests/test_ocr_tail_settlement.py` +- Reuse: `tests/test_ocr_rendering.py` +- Reuse: `tests/test_ocr_object_writeback.py` +- Reuse: `tests/test_appendix_figure_numbering.py` + +**Interfaces:** +- Consumes: + - B0/B1 extracted tail-settlement module and report + - existing render regressions for tail/backmatter ordering + - existing Workstream A tests to prove no cross-regression +- Produces: + - go / no-go decision for any future tail policy edits + - **no policy change commit unless diff is green** + +- [ ] **Step 1: Run the full tail/backmatter acceptance suite** + +Run: +`python -m pytest tests/test_ocr_tail_settlement.py tests/test_ocr_rendering.py::test_tail_zone_noise_band_guard tests/test_ocr_rendering.py::test_tail_zone_supplementary_material_not_noise tests/test_ocr_rendering.py::test_tail_candidate_overreach_does_not_absorb_late_body tests/test_ocr_rendering.py::test_cross_page_funding_continuation_preserves_order tests/test_ocr_rendering.py::test_mixed_tail_page_keeps_late_body_out_of_funding_and_attaches_real_funding tests/test_ocr_rendering.py::test_backmatter_boundary_normalizes_child_sections_before_references -q --tb=line` + +Expected: PASS + +- [ ] **Step 2: Run cross-workstream guard suite** + +Run: +`python -m pytest tests/test_ocr_object_writeback.py tests/test_ocr_rendering.py::test_body_renderer_skips_consumed_object_owned_blocks tests/test_appendix_figure_numbering.py -q --tb=line` + +Expected: PASS — Workstream B must not disturb Workstream A. + +- [ ] **Step 3: Evaluate whether any policy change is even needed** + +Decision rule: +- If **all** B0/B1 tests are green and the tail/backmatter render regressions remain green, **stop here**. Workstream B is complete as an extraction/reporting phase. +- If a future branch still needs a tail policy change, create a fresh micro-spec and micro-plan for that change; do **not** smuggle policy edits into this workstream after a green diff. + +- [ ] **Step 4: Commit only if code changed in this task** + +```bash +# Default case after a green diff gate: no code changed in B2, so no commit. +git status --short +# +# Expected: no output +# +# If you intentionally tightened the gate by editing only +# tests/test_ocr_tail_settlement.py in this task, use: +git add tests/test_ocr_tail_settlement.py +git commit -m "chore: lock OCR tail settlement diff gate" +``` + +--- + +## Self-Review + +- **Spec coverage:** This plan covers Workstream B from `2026-07-04-ocr-pipeline-deepening-design.md`: B0 extraction, B1 report, B2 diff gate. It intentionally does **not** start Workstream C. +- **Grounding against current code:** The plan explicitly reflects the current split call order: normalize-phase promotion/exclusion still happens before `DocumentStructure` exists (`ocr_document.py:5474-5521` today), while build-phase exclusion/restore still happens in `ocr_blocks.py:350-356` today. That sequencing is preserved. +- **Placeholder scan:** No `TODO`/`TBD` placeholders remain. B2 is intentionally a stop gate, not an unfinished implementation. +- **Type consistency:** B0 starts with a dict return for the wrapper because the module is brand-new; B1 upgrades that wrapper to `TailSettlementReport` and attaches it to `DocumentStructure`. The helper names are consistent across all tasks. + +## Execution Handoff + +**Plan complete and saved to `docs/superpowers/plans/2026-07-04-ocr-tail-settlement-implementation-plan.md`. Two execution options:** + +**1. Subagent-Driven (recommended)** - I dispatch a fresh subagent per task, review between tasks, fast iteration + +**2. Inline Execution** - Execute tasks in this session using executing-plans, batch execution with checkpoints + +**Which approach?** diff --git a/docs/superpowers/plans/2026-07-04-pre-merge-verification-plan.md b/docs/superpowers/plans/2026-07-04-pre-merge-verification-plan.md new file mode 100644 index 00000000..bf54fdc9 --- /dev/null +++ b/docs/superpowers/plans/2026-07-04-pre-merge-verification-plan.md @@ -0,0 +1,213 @@ +# OCR A/B/C Pre-Merge Verification Plan + +**Branch:** `feat/ocr-tail-settlement` + +**Current verdict:** not merge-ready yet. + +**Why:** fresh regression suite is green, but the current diff still has a few spec / regression blockers that need to be closed before merge. + +--- + +## 1. Fresh evidence already collected + +Ran on `feat/ocr-tail-settlement`: + +```bash +python -m pytest tests/test_ocr_pipeline_v3.py tests/test_ocr_tail_settlement.py tests/test_ocr_object_writeback.py tests/test_appendix_figure_numbering.py tests/test_ocr_rendering.py -q --tb=line +``` + +Observed: + +- `95 passed` + +This proves the current focused suite is green. + +It does **not** prove merge readiness by itself, because the remaining blockers are contract gaps and path-coverage gaps. + +--- + +## 2. Merge blockers to clear + +### Blocker A — page-qualified object writeback lookup + +**Observed code:** `paperforge/worker/ocr_object_writeback.py:111-123` + +```python +block_by_id: dict[str, dict] = {} +for b in structured_blocks: + bid = b.get("block_id") + if bid is not None: + block_by_id[str(bid)] = b +... +block = block_by_id.get(asset_bid) +``` + +**Risk:** if `block_id` is reused on different pages, the later block overwrites the earlier one. That can stamp ownership onto the wrong block and hide the wrong content at render time. + +**Required fix:** +- key by `(page, block_id)` instead of `block_id` +- use `(figure.page, asset.block_id)` / `(table.page, asset_block_id)` for lookups +- add a regression test with duplicated `block_id` across pages + +**Merge gate:** do not merge before this is fixed and covered. + +--- + +### Blocker B — contained figure text still bypasses the ownership contract + +**Observed code:** `paperforge/worker/ocr_figures.py:5677-5680` + +```python +block["_figure_contained"] = True +if role in _LEAK_ROLES: + block["role"] = "figure_inner_text" +``` + +**Observed gap:** `tests/test_ocr_object_writeback.py` covers side-adjacent text and consumed-block ids, but does **not** cover contained text. + +**Spec requirement:** `docs/superpowers/specs/2026-07-04-ocr-pipeline-deepening-design.md:468-476` +- contained figure text +- side-adjacent figure text +- idempotent writeback +- consumed-block contract visible both on block and inventory + +**Required fix:** +- route contained-text claims through the same ownership-evidence path as side-adjacent text +- stamp `_object_owner_*`, `_object_association_reason`, `_object_consumed` +- add consumed block ids to the owning figure inventory entry +- add a contained-text regression test to `tests/test_ocr_object_writeback.py` + +**Merge gate:** do not merge before contained text is under the same contract as side-adjacent text. + +--- + +### Blocker C — v3 path skips legacy rescue behavior + +**Observed flow:** +- `paperforge/worker/ocr_blocks.py` returns early in `normalize_mode="seed_only"` +- `paperforge/worker/ocr.py` then runs `pre_match_normalize(...)` and `post_match_normalize(...)` +- the legacy `rescue_roles_with_document_context(...)` pass is not run on the v3 path + +**Risk:** `OCR_PIPELINE_V3=1` can diverge from legacy on papers that rely on rescue-phase body/reference/heading correction. + +**Required action:** choose one and prove it: +1. **Preferred:** run `rescue_roles_with_document_context(...)` in the v3 path at the equivalent stage, or +2. prove by regression corpus that v3 no longer needs it + +**Merge gate:** because v3 stays default-off, this is a **caveat blocker**, not a release blocker for legacy behavior. It must still be addressed before calling the v3 path merge-ready. + +--- + +### Blocker D — no real-paper parity gate for `OCR_PIPELINE_V3=1` + +**Observed coverage:** current v3 tests are unit/synthetic only. + +**Missing proof:** no curated old-vs-new corpus diff on real OCR fixtures for: +- frontmatter +- backmatter +- reference boundaries +- figure/table matching parity + +**Required fix:** add a small real-paper parity gate, not a giant suite. + +Recommended corpus: +- one frontmatter-sensitive paper +- one tail/backmatter-sensitive paper +- one figure-heavy paper +- one table-heavy paper + +For each, compare legacy vs v3 on: +- final `role` +- `render_default` +- `index_default` +- figure/table inventory counts + +**Merge gate:** v3 is not merge-ready as a completed workstream until this gate exists and is green. + +--- + +## 3. Merge-before-merge execution order + +### Step 1 — fix object writeback lookup + +Run after the fix: + +```bash +python -m pytest tests/test_ocr_object_writeback.py -q --tb=line +``` + +Expected: +- all current writeback tests pass +- new duplicate-`block_id` cross-page regression test passes + +--- + +### Step 2 — unify contained-text ownership contract + +Run after the fix: + +```bash +python -m pytest tests/test_ocr_object_writeback.py tests/test_ocr_rendering.py::test_body_renderer_skips_consumed_object_owned_blocks -q --tb=line +``` + +Expected: +- contained-text regression passes +- side-adjacent regression still passes +- renderer still skips consumed blocks + +--- + +### Step 3 — restore or replace v3 rescue equivalence + +Run after the fix: + +```bash +python -m pytest tests/test_ocr_pipeline_v3.py tests/test_ocr_rendering.py::test_tail_zone_noise_band_guard tests/test_ocr_rendering.py::test_tail_zone_supplementary_material_not_noise tests/test_ocr_rendering.py::test_tail_candidate_overreach_does_not_absorb_late_body tests/test_ocr_rendering.py::test_cross_page_funding_continuation_preserves_order tests/test_ocr_rendering.py::test_backmatter_boundary_normalizes_child_sections_before_references -q --tb=line +``` + +Expected: +- v3 suite green +- tail/backmatter parity still green + +--- + +### Step 4 — add real-paper v3 parity gate + +Minimum acceptance command after adding it: + +```bash +python -m pytest tests/test_ocr_pipeline_v3.py -q --tb=line +``` + +Expected: +- synthetic tests green +- new real-paper parity tests green + +--- + +### Step 5 — final merge suite + +Run this fresh immediately before merge: + +```bash +python -m pytest tests/test_ocr_pipeline_v3.py tests/test_ocr_tail_settlement.py tests/test_ocr_object_writeback.py tests/test_appendix_figure_numbering.py tests/test_ocr_rendering.py -q --tb=line +``` + +Expected: +- `0 failed` + +If this is green **and** blockers A-D are closed, the branch is merge-ready. + +--- + +## 4. Merge decision rule + +Only merge when all are true: + +- cross-page duplicate `block_id` writeback bug is fixed +- contained text uses the same ownership contract as side-adjacent text +- v3 rescue equivalence is restored or explicitly proven unnecessary +- v3 real-paper parity gate exists and is green +- final fresh merge suite is green + +Until then: **ready for more work, not ready to merge**. diff --git a/paperforge/worker/ocr.py b/paperforge/worker/ocr.py index 1319bfe0..e6237598 100644 --- a/paperforge/worker/ocr.py +++ b/paperforge/worker/ocr.py @@ -69,8 +69,6 @@ from paperforge.worker.ocr_errors import ( ) from paperforge.worker.ocr_figures import ( build_figure_inventory, - tag_figure_contained_text, - write_back_figure_roles, write_figure_inventory, ) from paperforge.worker.ocr_metadata import ( @@ -80,9 +78,9 @@ from paperforge.worker.ocr_metadata import ( ) from paperforge.worker.ocr_tables import ( build_table_inventory, - write_back_table_roles, write_table_inventory, ) +from paperforge.worker.ocr_object_writeback import apply_object_writebacks from paperforge.worker.sync import ( load_control_actions, load_export_rows, @@ -103,6 +101,12 @@ OCR_QUEUE_STATUSES = { } OCR_SETTLED_STATUSES = {"done", "done_degraded", "fatal_error", "blocked"} + + +def _ocr_pipeline_v3_enabled() -> bool: + import os + + return os.environ.get("OCR_PIPELINE_V3", "").strip().lower() in {"1", "true", "yes", "on"} _LEGACY_HARD_DEGRADE_PREFIXES = ( "rendered text gaps", ) @@ -1855,11 +1859,21 @@ def postprocess_ocr_result(vault: Path, key: str, all_results: list[dict]) -> tu write_raw_blocks_jsonl(artifacts.blocks_raw, all_raw_blocks) + normalize_mode = "seed_only" if _ocr_pipeline_v3_enabled() else "legacy" structured, doc_structure = build_structured_blocks( all_raw_blocks, source_metadata=source_meta, structure_output_dir=artifacts.blocks_structured.parent, + normalize_mode=normalize_mode, ) + if _ocr_pipeline_v3_enabled(): + from paperforge.worker.ocr_pre_match_normalize import pre_match_normalize + + structured, doc_structure = pre_match_normalize( + structured, + source_frontmatter_anchors=getattr(doc_structure, "source_frontmatter_anchors", None), + document_structure=doc_structure, + ) write_structured_blocks_jsonl(artifacts.blocks_structured, structured) # Write role-level span profiles from paperforge.worker.ocr_profiles import write_role_span_profiles @@ -1878,9 +1892,7 @@ def postprocess_ocr_result(vault: Path, key: str, all_results: list[dict]) -> tu ) write_resolved_metadata(metadata_dir / "resolved_metadata.json", resolved) - # --- Phase 2: figure inventory --- figure_inventory = build_figure_inventory(structured) - write_back_figure_roles(figure_inventory, structured) # --- Author bio: residual figure inventory cleanup (Pass B, P1) --- from paperforge.worker.ocr_bio import ( @@ -1910,23 +1922,32 @@ def postprocess_ocr_result(vault: Path, key: str, all_results: list[dict]) -> tu # --- Phase 2: table inventory --- table_inventory = build_table_inventory(structured) - from paperforge.worker.ocr_figures import attach_ownership_conflicts, resolve_media_asset_conflicts + # --- Phase 2: object writeback --- + if _ocr_pipeline_v3_enabled(): + from paperforge.worker.ocr_post_match_normalize import post_match_normalize - resolve_media_asset_conflicts(figure_inventory, table_inventory) - attach_ownership_conflicts(figure_inventory, table_inventory) + structured, doc_structure = post_match_normalize( + structured, + figure_inventory, + table_inventory, + document_structure=doc_structure, + source_frontmatter_anchors=getattr(doc_structure, "source_frontmatter_anchors", None), + ) + + apply_object_writebacks( + structured_blocks=structured, + figure_inventory=figure_inventory, + table_inventory=table_inventory, + ) write_figure_inventory( artifacts.blocks_structured.parent / "figure_inventory.json", figure_inventory, ) - write_back_table_roles(table_inventory, structured) write_table_inventory( artifacts.blocks_structured.parent / "table_inventory.json", table_inventory, ) - # Figure containment render hygiene: tag text blocks inside figure regions - tag_figure_contained_text(structured, figure_inventory.get("matched_figures", [])) - # Re-persist structured blocks with writeback roles (table_html, figure_asset) # ponytail: writes entire list again; if throughput matters, write only changed blocks write_structured_blocks_jsonl(artifacts.blocks_structured, structured) diff --git a/paperforge/worker/ocr_blocks.py b/paperforge/worker/ocr_blocks.py index 37cbee0b..0d2ed1aa 100644 --- a/paperforge/worker/ocr_blocks.py +++ b/paperforge/worker/ocr_blocks.py @@ -190,9 +190,17 @@ def build_structured_blocks( raw_blocks: list[dict], source_metadata: dict | None = None, structure_output_dir: str | Path | None = None, + normalize_mode: str = "legacy", ) -> tuple[list[dict], Any]: """Build structured blocks with role assignment, normalization, and rescue. + Args: + raw_blocks: OCR raw blocks. + source_metadata: Optional source metadata for frontmatter anchors. + structure_output_dir: Optional output dir for document structure JSON. + normalize_mode: "legacy" (default) runs full normalize pipeline; + "seed_only" returns rows with seed_role as role before legacy normalize. + Returns (rows, doc_structure_or_None) so callers can pass the document structure artifact downstream without re-computing it. """ @@ -316,6 +324,12 @@ def build_structured_blocks( ) doc_structure.source_frontmatter_anchors = _sfm_anchors + # Early return for v3 seed-only mode — skip legacy normalize pipeline. + if normalize_mode == "seed_only": + for row in rows: + row["role"] = row["seed_role"] + return rows, doc_structure + # Normalize document structure (backmatter boundary, role regime, tail promotion) from paperforge.worker.ocr_document import normalize_document_structure @@ -347,13 +361,9 @@ def build_structured_blocks( rows = rescue_roles_with_document_context(rows, paper_context["role_profiles"], doc_structure) - from paperforge.worker.ocr_document import ( - _exclude_tail_nonref_from_body_flow, - _restore_numbered_body_from_tail_hold, - ) + from paperforge.worker.ocr_tail_settlement import settle_tail_and_backmatter - _exclude_tail_nonref_from_body_flow(rows) - _restore_numbered_body_from_tail_hold(rows) + settle_tail_and_backmatter(structured_blocks=rows, document_structure=doc_structure) # Sync render_default/index_default after role normalizations for row in rows: diff --git a/paperforge/worker/ocr_document.py b/paperforge/worker/ocr_document.py index 753c2286..c13948d5 100644 --- a/paperforge/worker/ocr_document.py +++ b/paperforge/worker/ocr_document.py @@ -19,6 +19,16 @@ from paperforge.worker.ocr_roles import ( _looks_like_figure_description_opening, ) from paperforge.worker.ocr_scores import score_structured_insert +from paperforge.worker.ocr_tail_settlement import ( + TailSettlementReport, + _block_text, + _canonical_section_text, + _next_nonempty_block_same_page, + _looks_like_tail_body, + exclude_tail_nonref_from_body_flow, + promote_backmatter_heading_candidates, + restore_numbered_body_from_tail_hold, +) _TAIL_ROLES = frozenset( { @@ -104,6 +114,7 @@ class DocumentStructure: # normalize_document_structure with dry-run + runtime estimates # from ocr_banding module. layout_band_estimate: dict | None = None + tail_settlement_report: TailSettlementReport | None = None reference_completeness_report: dict | None = None @@ -116,51 +127,8 @@ class RegionPrepass: confidence_by_index: dict[int, float] -def _block_text(block: dict) -> str: - return str(block.get("text") or block.get("block_content") or "") -def _next_nonempty_block_same_page(blocks: list[dict], idx: int) -> dict | None: - """Return the next non-empty block on the same page after idx, or None.""" - page = blocks[idx].get("page") - if page is None: - return None - for j in range(idx + 1, len(blocks)): - if blocks[j].get("page") != page: - return None - text = _block_text(blocks[j]).strip() - if text: - return blocks[j] - return None - - -_BACKMATTER_BODY_SIGNALS = re.compile( - r"\b(?:declare|conflict|interest|funding|support|grant|author|contribut|" - r"acknowledge|thank|ethic|review|approv|consent|availab|data|material|" - r"competing|financial|disclos|report|none|no conflict|nothing to declare)\b", - re.IGNORECASE, -) - - -def _looks_like_tail_body(block: dict) -> bool: - """Return True if the block looks like short backmatter body text. - - Heuristics: short paragraphs with backmatter-related vocabulary, - no section-heading formatting, and a word count below the body spine - threshold. - """ - text = _block_text(block).strip() - if not text: - return False - words = text.split() - if len(words) > 80: - return False - role = block.get("role", "") - if role in {"section_heading", "subsection_heading", "sub_subsection_heading"}: - return False - if _BACKMATTER_BODY_SIGNALS.search(text): - return True - return False _BIOGRAPHY_TEXT_SIGNALS = re.compile( @@ -600,11 +568,6 @@ def _make_zone( return zone -def _canonical_section_text(block: dict) -> str: - text = _block_text(block).strip().lower() - if re.match(r"^(?:\w\s)+\w$", text): - text = re.sub(r"\s+", "", text) - return text def _strip_inline_html(text: str) -> str: @@ -3455,52 +3418,6 @@ def _split_merged_heading_blocks(blocks: list[dict], role_profiles: dict | None return result -def _looks_like_backmatter_body_text(text: str) -> bool: - lower = text.lower() - markers = ( - "conflict of interest", - "declaration", - "publisher", - "author contributions", - "funding", - "acknowledg", - "data availability", - "supplement", - "ethics", - "copyright", - ) - return any(marker in lower for marker in markers) - - -def _exclude_tail_nonref_from_body_flow(blocks: list[dict]) -> None: - for i, block in enumerate(blocks): - effective_role = block.get("role") - if effective_role == "unassigned": - effective_role = block.get("seed_role") - if effective_role != "body_paragraph": - continue - if block.get("zone") != "tail_nonref_hold_zone": - continue - - # Only convert blocks that carry explicit backmatter evidence; - # preserve ordinary body continuation or body-like prose. - text = str(block.get("text") or block.get("block_content") or "") - if not _looks_like_backmatter_body_text(text): - continue - - old_role = block.get("role") - block["role"] = "backmatter_body" - if block.get("seed_role") == "body_paragraph": - block["seed_role"] = "backmatter_body" - if old_role != block["role"]: - record_decision( - block, - stage="tail_nonref_exclusion", - old_role=old_role, - new_role=block["role"], - reason="tail non-reference body block excluded from body flow", - ) - block.setdefault("evidence", []).append("tail_nonref_hold_zone excluded from body flow") def _get_column(block: dict, page_width: float = 1200) -> int: @@ -5368,33 +5285,6 @@ def _demote_early_frontmatter_body_leaks(blocks: list[dict]) -> None: continue -def _restore_numbered_body_from_tail_hold(blocks: list[dict]) -> None: - active_numbered_body = False - for block in blocks: - role = block.get("role") - text = _canonical_section_text(block) - marker_type = str(((block.get("marker_signature") or {}).get("type")) or "none") - - if role in {"reference_heading", "backmatter_heading", "backmatter_boundary_heading"}: - active_numbered_body = False - continue - - if role in {"section_heading", "subsection_heading", "sub_subsection_heading"}: - active_numbered_body = ( - ( - marker_type - in {"heading_numbered", "heading_arabic", "heading_decimal", "heading_roman", "heading_alpha"} - or ( - block.get("zone") == "tail_nonref_hold_zone" - and str(block.get("style_family") or "") == "heading_like" - ) - ) - and text not in _BACKMATTER_TITLE_DENY_LIST - ) - continue - - if role == "backmatter_body" and active_numbered_body: - block["role"] = "body_paragraph" def normalize_document_structure( @@ -5428,8 +5318,10 @@ def normalize_document_structure( "reference_family_anchor": reference_family_anchor, }, ) + settlement_report = TailSettlementReport() + exclude_tail_nonref_from_body_flow(blocks, report=settlement_report) _exclude_frontmatter_side_from_body_flow(blocks) - _exclude_tail_nonref_from_body_flow(blocks) + exclude_tail_nonref_from_body_flow(blocks, report=settlement_report) _recover_empty_text_from_spans(blocks) from paperforge.worker.ocr_roles import resolve_final_role @@ -5471,54 +5363,11 @@ def normalize_document_structure( except Exception: pass - # Backmatter heading candidate promotion: confirm candidates that are - # followed by body-like text on the same page. This runs after role - # resolution (which may have reclassified the candidate via editorial - # phrase detection) and uses seed_role to recover the original intent. - for idx, block in enumerate(blocks): - is_candidate = ( - block.get("role") == "backmatter_heading_candidate" - or block.get("seed_role") == "backmatter_heading_candidate" - ) - if not is_candidate: - continue - if block.get("role") == "backmatter_heading": - continue - next_body = _next_nonempty_block_same_page(blocks, idx) - if next_body and next_body.get("role") in {"body_paragraph", "backmatter_body"}: - if _looks_like_tail_body(next_body): - old_role = block.get("role") - block["role"] = "backmatter_heading" - block.setdefault("role_confidence", 0.6) - if old_role != block["role"]: - record_decision( - block, - stage="backmatter_candidate_promotion", - old_role=old_role, - new_role=block["role"], - reason="backmatter heading candidate promoted: followed by tail-like body on same page", - ) - # Convert follower body paragraphs to backmatter_body - for j in range(idx + 1, len(blocks)): - if blocks[j].get("page") != block.get("page"): - break - if blocks[j].get("role") == "body_paragraph": - old_follower_role = blocks[j].get("role") - blocks[j]["role"] = "backmatter_body" - blocks[j].setdefault("role_confidence", 0.6) - if old_follower_role != blocks[j]["role"]: - record_decision( - blocks[j], - stage="backmatter_candidate_promotion", - old_role=old_follower_role, - new_role=blocks[j]["role"], - reason="follower body converted to backmatter_body under confirmed heading", - ) - break # only promote one candidate per pass + promote_backmatter_heading_candidates(blocks, report=settlement_report) # Re-run tail non-ref exclusion after role resolution so blocks that # were assigned tail_nonref_hold_zone get their role converted. - _exclude_tail_nonref_from_body_flow(blocks) + exclude_tail_nonref_from_body_flow(blocks, report=settlement_report) # Recompute page layouts after role resolution — initial layout may have # underestimated column count because raw OCR labels (text, paragraph_title) @@ -5690,6 +5539,7 @@ def normalize_document_structure( reference_family_anchor=reference_family_anchor, page_layouts=page_layouts, region_bus=region_bus, + tail_settlement_report=settlement_report, reference_completeness_report=ref_completeness, ) @@ -6079,7 +5929,7 @@ def normalize_document_structure( block["role_verification_status"] = "ACCEPT" block["render_default"] = False _demote_early_frontmatter_body_leaks(blocks) - _restore_numbered_body_from_tail_hold(blocks) + restore_numbered_body_from_tail_hold(blocks, report=settlement_report) if tail_spread is not None: restore_seed_heading_after_page = tail_spread.references_start or tail_spread.backmatter_start or 0 for block in blocks: diff --git a/paperforge/worker/ocr_figure_domain.py b/paperforge/worker/ocr_figure_domain.py index bb31ed6a..ca9ff48f 100644 --- a/paperforge/worker/ocr_figure_domain.py +++ b/paperforge/worker/ocr_figure_domain.py @@ -26,10 +26,10 @@ class FigureCorpus: from . import ocr_figures raw_legends = [ - b for b in blocks if b.get("role") in {"figure_caption", "figure_caption_candidate"} + b for b in blocks if ocr_figures._match_role(b) in {"figure_caption", "figure_caption_candidate"} ] raw_assets = [ - b for b in blocks if b.get("role") in {"figure_asset", "media_asset"} + b for b in blocks if ocr_figures._match_role(b) in {"figure_asset", "media_asset"} ] locator_candidates = [ b for b in raw_legends if ocr_figures._is_previous_page_legend_locator(b) diff --git a/paperforge/worker/ocr_figures.py b/paperforge/worker/ocr_figures.py index aa869b44..00d5bbe8 100644 --- a/paperforge/worker/ocr_figures.py +++ b/paperforge/worker/ocr_figures.py @@ -85,6 +85,11 @@ _INLINE_FIGURE_MENTION_VERBS = ( ) +def _match_role(block: dict) -> str: + """Resolve the role to use for matching: role_candidate > role > seed_role.""" + return str(block.get("role_candidate") or block.get("role") or block.get("seed_role") or "") + + def _looks_like_inline_figure_mention(text: str, block: dict | None = None) -> bool: """ Ponytail: position and style are the reliable signals for distinguishing @@ -434,7 +439,7 @@ _TRUNCATED_LEGEND_ONLY_PATTERN = re.compile( def _is_validation_first_legend_candidate(block: dict) -> bool: - role = str(block.get("role") or "") + role = _match_role(block) marker_signature = block.get("marker_signature") or {} marker_type = str(marker_signature.get("type") or "none") zone = str(block.get("zone") or "") @@ -817,7 +822,7 @@ def _filter_figure_assets(assets: list[dict]) -> list[dict]: for b in assets: if b.get("_non_body_media"): continue - role = b.get("role", "") + role = _match_role(b) if role == "figure_asset": result.append(b) elif role == "media_asset": @@ -1666,7 +1671,7 @@ def is_embedded_figure_text(block: dict, all_blocks: list[dict], page_width: flo def _formal_figure_caption_blocks(blocks: list[dict]) -> dict[int, list[dict]]: by_page: dict[int, list[dict]] = {} for block in blocks: - role = str(block.get("role") or "") + role = _match_role(block) text = str(block.get("text") or "") if role not in {"figure_caption", "figure_caption_candidate"}: continue @@ -3001,7 +3006,7 @@ def build_figure_inventory_vnext( # Recover figure heading prefix from PDF text layer for OCR-missed captions if page_pdf_lines_by_page: for block in structured_blocks: - role = block.get("role", "") + role = _match_role(block) if role in {"figure_caption", "figure_caption_candidate"}: text = str(block.get("text", "") or "") from .ocr_figures import _extract_figure_number, _recover_figure_heading_prefix @@ -3114,7 +3119,7 @@ def build_figure_inventory_legacy( page_width = float(block["page_width"]) for block in structured_blocks: - role = block.get("role", "") + role = _match_role(block) if block.get("_non_body_media") or role == "non_body_insert": continue # Skip single-letter panel labels (A, B, (C), A.) in figure legends diff --git a/paperforge/worker/ocr_object_writeback.py b/paperforge/worker/ocr_object_writeback.py new file mode 100644 index 00000000..7e1649ee --- /dev/null +++ b/paperforge/worker/ocr_object_writeback.py @@ -0,0 +1,338 @@ +"""OCR object writeback module — post-inventory figure/table ownership claims. + +Centralises role writeback, ownership evidence, contained-text tagging, and +object-adjacent text claims that were previously scattered across ocr.py. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from paperforge.worker.ocr_figures import ( + attach_ownership_conflicts, + tag_figure_contained_text, +) + +PROTECTED_ROLES = frozenset({ + "reference_item", + "reference_heading", + "section_heading", + "paper_title", + "authors", + "author_bio", + "footnote", + "table_cell", + "table_caption", + "figure_caption", +}) + +@dataclass(frozen=True) +class ObjectOwnershipClaim: + owner_family: str + owner_id: str + owner_role: str + association_reason: str + confidence: float + source_phase: str + source_block_ids: tuple[str, ...] + + +@dataclass +class ObjectWritebackReport: + claims: list[dict] + applied: list[dict] + skipped: list[dict] + conflicts: list[dict] + consumed_block_ids: dict[str, list[str]] + + + +def _mark_owned(block: dict, *, family: str, owner_id: str, owner_role: str, reason: str) -> None: + """Stamp ownership-evidence contract onto a structured block.""" + block["_object_owner_family"] = family + block["_object_owner_id"] = owner_id + block["_object_owner_role"] = owner_role + block["_object_association_reason"] = reason + block["_object_writeback_phase"] = "post_inventory" + block["_object_consumed"] = True + +def _score_side_adjacent_text_claim(block: dict, figure: dict) -> float: + """Score whether a text block is a side-adjacent figure note. + + Returns confidence in [0, 1]. Threshold 0.9 is applied by caller. + """ + bbox = block.get("bbox") or [0, 0, 0, 0] + figure_bbox = figure.get("cluster_bbox") or figure.get("asset_bbox") + if not figure_bbox or len(figure_bbox) < 4: + # Fallback: union bbox from matched_assets + assets = figure.get("matched_assets") or [] + xs, ys = [], [] + for a in assets: + ab = a.get("bbox") or [0, 0, 0, 0] + if len(ab) >= 4: + xs.extend([ab[0], ab[2]]) + ys.extend([ab[1], ab[3]]) + figure_bbox = [min(xs), min(ys), max(xs), max(ys)] if xs else [0, 0, 0, 0] + if len(bbox) < 4 or len(figure_bbox) < 4: + return 0.0 + same_page = int(block.get("page", 0) or 0) == int(figure.get("page", 0) or 0) + if not same_page: + return 0.0 + y_overlap = max(0, min(bbox[3], figure_bbox[3]) - max(bbox[1], figure_bbox[1])) + horizontal_gap = min(abs(figure_bbox[0] - bbox[2]), abs(bbox[0] - figure_bbox[2])) + if y_overlap <= 0 or horizontal_gap > 40: + return 0.0 + role = str(block.get("role") or "") + if role in PROTECTED_ROLES: + return 0.0 + raw_label = str(block.get("raw_label") or "") + if raw_label != "text": + return 0.0 + text = str(block.get("text") or "").strip() + if not text or len(text.split()) > 80: + return 0.0 + return 0.92 + +def apply_object_writebacks( + *, structured_blocks: list[dict], figure_inventory: dict, table_inventory: dict +) -> dict: + """Apply all post-inventory object writebacks to structured blocks. + + Emits ownership-claim evidence, tracks consumed block ids for renderer + skip, and is idempotent via the ``_object_writeback_phase`` guard. + + Returns a dict matching ObjectWritebackReport shape for backward compat. + """ + claims: list[dict] = [] + applied: list[dict] = [] + skipped: list[dict] = [] + consumed_block_ids: dict[str, list[str]] = {} + + block_by_page_and_id: dict[tuple[int, str], dict] = {} + for b in structured_blocks: + bid = b.get("block_id") + page = b.get("page", 0) or 0 + if bid is not None: + block_by_page_and_id[(int(page), str(bid))] = b + + # --- Figure asset writeback --- + for fig in figure_inventory.get("matched_figures", []): + figure_id = str(fig.get("figure_id", "")) + fig_consumed: list[str] = [] + for asset in fig.get("matched_assets", []): + asset_bid = str(asset.get("block_id", "")) + fig_page = int(fig.get("page", 0) or 0) + block = block_by_page_and_id.get((fig_page, asset_bid)) + if block is None: + continue + # Idempotency guard + if block.get("_object_writeback_phase"): + continue + block["role"] = "figure_asset" + _mark_owned(block, family="figure", owner_id=figure_id, owner_role="asset", reason="matched_asset") + claim = ObjectOwnershipClaim( + owner_family="figure", + owner_id=figure_id, + owner_role="asset", + association_reason="matched_asset", + confidence=1.0, + source_phase="post_inventory", + source_block_ids=(asset_bid,), + ) + claims.append({ + "owner_family": claim.owner_family, + "owner_id": claim.owner_id, + "owner_role": claim.owner_role, + "association_reason": claim.association_reason, + "confidence": claim.confidence, + "source_phase": claim.source_phase, + "source_block_ids": list(claim.source_block_ids), + }) + applied.append({"block_id": asset_bid, "role": "figure_asset"}) + fig_consumed.append(asset_bid) + if fig_consumed: + fig.setdefault("consumed_block_ids", []).extend(fig_consumed) + consumed_block_ids.setdefault(figure_id, []).extend(fig_consumed) + + # --- Table asset writeback --- + for tbl in table_inventory.get("tables", []): + table_id = str(tbl.get("table_id", "")) + tbl_consumed: list[str] = [] + asset_bid = str(tbl.get("asset_block_id", "")) + tbl_page = int(tbl.get("page", 0) or 0) + block = block_by_page_and_id.get((tbl_page, asset_bid)) + if block is not None and not block.get("_object_writeback_phase"): + block["role"] = "table_html" + _mark_owned(block, family="table", owner_id=table_id, owner_role="asset", reason="matched_asset") + claim = ObjectOwnershipClaim( + owner_family="table", + owner_id=table_id, + owner_role="asset", + association_reason="matched_asset", + confidence=1.0, + source_phase="post_inventory", + source_block_ids=(asset_bid,), + ) + claims.append({ + "owner_family": claim.owner_family, + "owner_id": claim.owner_id, + "owner_role": claim.owner_role, + "association_reason": claim.association_reason, + "confidence": claim.confidence, + "source_phase": claim.source_phase, + "source_block_ids": list(claim.source_block_ids), + }) + applied.append({"block_id": asset_bid, "role": "table_html"}) + tbl_consumed.append(asset_bid) + if tbl_consumed: + tbl.setdefault("consumed_block_ids", []).extend(tbl_consumed) + consumed_block_ids.setdefault(table_id, []).extend(tbl_consumed) + + # --- Ownership conflicts (bridging) --- + attach_ownership_conflicts(figure_inventory, table_inventory) + + # --- Contained text tagging (bridging) --- + tag_figure_contained_text(structured_blocks, figure_inventory.get("matched_figures", [])) + + # --- Contained figure text — route through ownership evidence --- + for fig in figure_inventory.get("matched_figures", []): + figure_id = str(fig.get("figure_id", "")) + fig_page = int(fig.get("page", 0) or 0) + region = fig.get("cluster_bbox") + if not region or len(region) < 4: + # Fallback: union bbox from matched_assets + assets = fig.get("matched_assets", []) + bboxes = [ + a.get("bbox", [0, 0, 0, 0]) + for a in assets + if len(a.get("bbox") or []) >= 4 + ] + if bboxes: + region = [min(b[0] for b in bboxes), min(b[1] for b in bboxes), + max(b[2] for b in bboxes), max(b[3] for b in bboxes)] + if not region or len(region) < 4: + continue + for block in structured_blocks: + if block.get("_object_writeback_phase"): + continue + if not block.get("_figure_contained"): + continue + bid = block.get("block_id") + if bid is None: + continue + block_page = int(block.get("page", 0) or 0) + if block_page != fig_page: + continue + # Check if this block is within this figure's region + bbox = block.get("bbox") or [0, 0, 0, 0] + if len(bbox) < 4: + continue + fx1, fy1, fx2, fy2 = region[:4] + bx1, by1, bx2, by2 = bbox[:4] + if not (bx1 >= fx1 and by1 >= fy1 and bx2 <= fx2 and by2 <= fy2): + continue + # Assign ownership + block["role"] = "figure_inner_text" + _mark_owned( + block, + family="figure", + owner_id=figure_id, + owner_role="inner_text", + reason="contained", + ) + bid_str = str(bid) + fig.setdefault("consumed_block_ids", []).append(bid_str) + fig.setdefault("associated_text_block_ids", []).append(bid_str) + consumed_block_ids.setdefault(figure_id, []).append(bid_str) + claim = ObjectOwnershipClaim( + owner_family="figure", + owner_id=figure_id, + owner_role="inner_text", + association_reason="contained", + confidence=0.8, + source_phase="post_inventory", + source_block_ids=(bid_str,), + ) + claims.append({ + "owner_family": claim.owner_family, + "owner_id": claim.owner_id, + "owner_role": claim.owner_role, + "association_reason": claim.association_reason, + "confidence": claim.confidence, + "source_phase": claim.source_phase, + "source_block_ids": list(claim.source_block_ids), + }) + applied.append({"block_id": bid_str, "role": "figure_inner_text"}) + + # --- Side-adjacent figure text claims --- + for fig in figure_inventory.get("matched_figures", []): + figure_id = str(fig.get("figure_id", "")) + for block in structured_blocks: + bid = block.get("block_id") + if bid is None: + continue + # Skip already-owned blocks + if block.get("_object_writeback_phase"): + continue + score = _score_side_adjacent_text_claim(block, fig) + if score >= 0.9: + block["role"] = "figure_inner_text" + _mark_owned( + block, + family="figure", + owner_id=figure_id, + owner_role="inner_text", + reason="side_adjacent", + ) + block["_object_association_reason"] = "side_adjacent" + bid_str = str(bid) + fig.setdefault("associated_text_block_ids", []).append(bid_str) + if bid_str not in fig.setdefault("consumed_block_ids", []): + fig["consumed_block_ids"].append(bid_str) + consumed_block_ids.setdefault(figure_id, []).append(bid_str) + claim = ObjectOwnershipClaim( + owner_family="figure", + owner_id=figure_id, + owner_role="inner_text", + association_reason="side_adjacent", + confidence=score, + source_phase="post_inventory", + source_block_ids=(bid_str,), + ) + claims.append({ + "owner_family": claim.owner_family, + "owner_id": claim.owner_id, + "owner_role": claim.owner_role, + "association_reason": claim.association_reason, + "confidence": claim.confidence, + "source_phase": claim.source_phase, + "source_block_ids": list(claim.source_block_ids), + }) + applied.append({"block_id": bid_str, "role": "figure_inner_text"}) + elif score > 0: + skipped.append({"block_id": str(bid), "score": score, "reason": "below_threshold"}) + + # --- Legacy bridging: write_back_figure_roles already handled inline above --- + # --- Legacy bridging: write_back_table_roles already handled inline above --- + + # --- Aggregate any previously-consumed block ids from inventory --- + # Ensures idempotent calls still report the full consumed state. + for fig in figure_inventory.get("matched_figures", []): + fid = str(fig.get("figure_id", "")) + for cb in fig.get("consumed_block_ids", []): + if cb not in consumed_block_ids.get(fid, []): + consumed_block_ids.setdefault(fid, []).append(cb) + for tbl in table_inventory.get("tables", []): + tid = str(tbl.get("table_id", "")) + for cb in tbl.get("consumed_block_ids", []): + if cb not in consumed_block_ids.get(tid, []): + consumed_block_ids.setdefault(tid, []).append(cb) + + return { + "claims": claims, + "applied": applied, + "skipped": skipped, + "conflicts": figure_inventory.get("ownership_conflicts", []), + "consumed_block_ids": consumed_block_ids, + "applied_count": len(applied), + } diff --git a/paperforge/worker/ocr_post_match_normalize.py b/paperforge/worker/ocr_post_match_normalize.py new file mode 100644 index 00000000..4a7a78f4 --- /dev/null +++ b/paperforge/worker/ocr_post_match_normalize.py @@ -0,0 +1,47 @@ +"""OCR post-match normalization — v3 final role commit. + +Consumes figure/table inventories and commits the final public `role`. +Reuses tail settlement from Workstream B. +""" + +from __future__ import annotations + +from paperforge.worker.ocr_document import DocumentStructure, normalize_document_structure +from paperforge.worker.ocr_document import rescue_roles_with_document_context +from paperforge.worker.ocr_profiles import build_role_span_profiles +from paperforge.worker.ocr_tail_settlement import settle_tail_and_backmatter + + +def post_match_normalize( + rows: list[dict], + figure_inventory: dict, + table_inventory: dict, + *, + document_structure: DocumentStructure, + source_frontmatter_anchors: dict | None = None, +) -> tuple[list[dict], DocumentStructure]: + """Shadow normalize rows, commit final roles, run tail settlement.""" + live_rows = [dict(row) for row in rows] + shadow_doc, shadow_rows = normalize_document_structure( + [dict(row) for row in rows], + source_frontmatter_anchors=source_frontmatter_anchors, + ) + for live, shadow in zip(live_rows, shadow_rows): + live["role"] = shadow.get("role", live.get("role")) + live["role_source"] = shadow.get("role_source", live.get("role_source")) + live["role_confidence"] = shadow.get("role_confidence", live.get("role_confidence")) + live["role_candidate"] = live.get("role_candidate") or shadow.get("role") + live["render_default"] = shadow.get("render_default", live.get("render_default")) + live["index_default"] = shadow.get("index_default", live.get("index_default")) + # Rescue: document-context corrections (same as legacy build_structured_blocks) + paper_context: dict = {} + if len(live_rows) >= 10: + paper_context["role_profiles"] = build_role_span_profiles(live_rows) + source_anchors = getattr(shadow_doc, "source_frontmatter_anchors", {}) if shadow_doc else {} + legacy_rescue_enabled = not bool(source_anchors) + if paper_context.get("role_profiles") and shadow_doc and legacy_rescue_enabled: + live_rows = rescue_roles_with_document_context( + live_rows, paper_context["role_profiles"], shadow_doc + ) + settle_tail_and_backmatter(structured_blocks=live_rows, document_structure=shadow_doc) + return live_rows, shadow_doc diff --git a/paperforge/worker/ocr_pre_match_normalize.py b/paperforge/worker/ocr_pre_match_normalize.py new file mode 100644 index 00000000..e9a87cdd --- /dev/null +++ b/paperforge/worker/ocr_pre_match_normalize.py @@ -0,0 +1,31 @@ +"""OCR pre-match normalization — v3 candidate-only normalize. + +Preserves public `role = seed_role`, writes `role_candidate` from a shadow +normalize pass so figure/table matching can work from candidates. +""" + +from __future__ import annotations + +from paperforge.worker.ocr_document import DocumentStructure, normalize_document_structure + + +def pre_match_normalize( + rows: list[dict], + *, + source_frontmatter_anchors: dict | None = None, + document_structure: DocumentStructure | None = None, +) -> tuple[list[dict], DocumentStructure]: + """Run shadow normalize to compute role_candidate; keep public role=seed_role.""" + live_rows = [dict(row) for row in rows] + shadow_doc, shadow_rows = normalize_document_structure( + [dict(row) for row in rows], + source_frontmatter_anchors=source_frontmatter_anchors, + ) + for live, shadow in zip(live_rows, shadow_rows): + live["role_candidate"] = shadow.get("role") or shadow.get("seed_role") or live.get("seed_role") + live["zone"] = shadow.get("zone", live.get("zone")) + live["style_family"] = shadow.get("style_family", live.get("style_family")) + live["marker_signature"] = shadow.get("marker_signature", live.get("marker_signature")) + live["render_default"] = live.get("render_default", True) + live["role"] = live.get("seed_role") or live.get("role") + return live_rows, document_structure or shadow_doc diff --git a/paperforge/worker/ocr_render.py b/paperforge/worker/ocr_render.py index fbb10b84..35233619 100644 --- a/paperforge/worker/ocr_render.py +++ b/paperforge/worker/ocr_render.py @@ -1656,6 +1656,10 @@ def render_fulltext_markdown( ): continue + # Object writeback ownership — skip consumed blocks from render + if block.get("_object_consumed"): + continue + _SKIPPED_BODY_ROLES = { "abstract_heading", "abstract_body", diff --git a/paperforge/worker/ocr_table_domain.py b/paperforge/worker/ocr_table_domain.py index fb2c4fb6..e2129edc 100644 --- a/paperforge/worker/ocr_table_domain.py +++ b/paperforge/worker/ocr_table_domain.py @@ -19,12 +19,12 @@ class TableCorpus: raw_captions = [ b for b in blocks - if b.get("role") in {"table_caption", "table_caption_candidate"} + if ocr_tables._match_role(b) in {"table_caption", "table_caption_candidate"} or ocr_tables._is_validation_first_table_candidate(b) ] raw_assets = [] for block in blocks: - role = block.get("role", "") + role = ocr_tables._match_role(block) raw_label = str(block.get("raw_label", "") or "").strip() if role not in ("table_asset", "table_html", "media_asset", "figure_asset"): continue diff --git a/paperforge/worker/ocr_tables.py b/paperforge/worker/ocr_tables.py index 99c0c952..f902dbc2 100644 --- a/paperforge/worker/ocr_tables.py +++ b/paperforge/worker/ocr_tables.py @@ -25,6 +25,11 @@ _TRUNCATED_TABLE_ONLY_PATTERN = re.compile( ) +def _match_role(block: dict) -> str: + """Resolve the role to use for matching: role_candidate > role > seed_role.""" + return str(block.get("role_candidate") or block.get("role") or block.get("seed_role") or "") + + def _roman_to_int(roman: str) -> int | None: roman_values = {"I": 1, "V": 5, "X": 10, "L": 50, "C": 100, "D": 500, "M": 1000} s = roman.strip().upper() @@ -93,7 +98,7 @@ def _extract_base_table_number(text: str) -> int | None: def _is_validation_first_table_candidate(block: dict) -> bool: - role = str(block.get("role", "") or "") + role = _match_role(block) if role in {"table_caption", "table_caption_candidate"}: return False marker_type = (block.get("marker_signature") or {}).get("type") or "none" @@ -114,7 +119,7 @@ def _is_insufficient_table_caption_evidence(block: dict) -> bool: def _is_weak_explicit_table_caption(block: dict) -> bool: - role = str(block.get("role", "") or "") + role = _match_role(block) if role in {"table_caption", "table_caption_candidate"}: return _is_insufficient_table_caption_evidence(block) return _is_validation_first_table_candidate(block) and _is_insufficient_table_caption_evidence(block) @@ -258,7 +263,7 @@ def build_table_inventory_legacy(structured_blocks: list[dict]) -> dict[str, Any unmatched_assets: list[dict] = [] for block in structured_blocks: - role = block.get("role", "") + role = _match_role(block) raw_label = str(block.get("raw_label", "") or "").strip() if role in {"table_caption", "table_caption_candidate"} or _is_validation_first_table_candidate(block): captions.append(block) diff --git a/paperforge/worker/ocr_tail_settlement.py b/paperforge/worker/ocr_tail_settlement.py new file mode 100644 index 00000000..53737a40 --- /dev/null +++ b/paperforge/worker/ocr_tail_settlement.py @@ -0,0 +1,237 @@ +from __future__ import annotations + +import re + +from paperforge.worker.ocr_decisions import record_decision +from paperforge.worker.ocr_roles import _BACKMATTER_TITLE_DENY_LIST + +from dataclasses import dataclass, field + + +@dataclass +class TailSettlementReport: + promoted_backmatter_heading_ids: list[str] = field(default_factory=list) + converted_to_backmatter_body_ids: list[str] = field(default_factory=list) + restored_body_paragraph_ids: list[str] = field(default_factory=list) + + @property + def applied_count(self) -> int: + return ( + len(self.promoted_backmatter_heading_ids) + + len(self.converted_to_backmatter_body_ids) + + len(self.restored_body_paragraph_ids) + ) + +_BACKMATTER_BODY_SIGNALS = re.compile( + r"\b(?:declare|conflict|interest|funding|support|grant|author|contribut|" + r"acknowledge|thank|ethic|review|approv|consent|availab|data|material|" + r"competing|financial|disclos|report|none|no conflict|nothing to declare)\b", + re.IGNORECASE, +) + + +def _block_text(block: dict) -> str: + return str(block.get("text") or block.get("block_content") or "") + + +def _next_nonempty_block_same_page(blocks: list[dict], idx: int) -> dict | None: + """Return the next non-empty block on the same page after idx, or None.""" + page = blocks[idx].get("page") + if page is None: + return None + for j in range(idx + 1, len(blocks)): + if blocks[j].get("page") != page: + return None + text = _block_text(blocks[j]).strip() + if text: + return blocks[j] + return None + + +def _canonical_section_text(block: dict) -> str: + text = _block_text(block).strip().lower() + if re.match(r"^(?:\w\s)+\w$", text): + text = re.sub(r"\s+", "", text) + return text + + +def _looks_like_tail_body(block: dict) -> bool: + """Return True if the block looks like short backmatter body text. + + Heuristics: short paragraphs with backmatter-related vocabulary, + no section-heading formatting, and a word count below the body spine + threshold. + """ + text = _block_text(block).strip() + if not text: + return False + words = text.split() + if len(words) > 80: + return False + role = block.get("role", "") + if role in {"section_heading", "subsection_heading", "sub_subsection_heading"}: + return False + if _BACKMATTER_BODY_SIGNALS.search(text): + return True + return False + + +def _looks_like_backmatter_body_text(text: str) -> bool: + lower = text.lower() + markers = ( + "conflict of interest", + "declaration", + "publisher", + "author contributions", + "funding", + "acknowledg", + "data availability", + "supplement", + "ethics", + "copyright", + ) + return any(marker in lower for marker in markers) + + +def promote_backmatter_heading_candidates(blocks: list[dict], report: TailSettlementReport | None = None) -> None: + """Promote backmatter heading candidates that are followed by tail-like body on the same page.""" + for idx, block in enumerate(blocks): + is_candidate = ( + block.get("role") == "backmatter_heading_candidate" + or block.get("seed_role") == "backmatter_heading_candidate" + ) + if not is_candidate: + continue + if block.get("role") == "backmatter_heading": + continue + next_body = _next_nonempty_block_same_page(blocks, idx) + if next_body and next_body.get("role") in {"body_paragraph", "backmatter_body"}: + if _looks_like_tail_body(next_body): + old_role = block.get("role") + block["role"] = "backmatter_heading" + block.setdefault("role_confidence", 0.6) + if old_role != block["role"]: + record_decision( + block, + stage="backmatter_candidate_promotion", + old_role=old_role, + new_role=block["role"], + reason="backmatter heading candidate promoted: followed by tail-like body on same page", + ) + if report is not None: + block_id = str(block.get("block_id") or "") + if block_id: + report.promoted_backmatter_heading_ids.append(block_id) + # Convert follower body paragraphs to backmatter_body + for j in range(idx + 1, len(blocks)): + if blocks[j].get("page") != block.get("page"): + break + if blocks[j].get("role") == "body_paragraph": + old_follower_role = blocks[j].get("role") + blocks[j]["role"] = "backmatter_body" + blocks[j].setdefault("role_confidence", 0.6) + if old_follower_role != blocks[j]["role"]: + record_decision( + blocks[j], + stage="backmatter_candidate_promotion", + old_role=old_follower_role, + new_role=blocks[j]["role"], + reason="follower body converted to backmatter_body under confirmed heading", + ) + if report is not None: + follower_id = str(blocks[j].get("block_id") or "") + if follower_id: + report.converted_to_backmatter_body_ids.append(follower_id) + break # only promote one candidate per pass + + +def exclude_tail_nonref_from_body_flow(blocks: list[dict], report: TailSettlementReport | None = None) -> None: + for i, block in enumerate(blocks): + effective_role = block.get("role") + if effective_role == "unassigned": + effective_role = block.get("seed_role") + if effective_role != "body_paragraph": + continue + if block.get("zone") != "tail_nonref_hold_zone": + continue + + # Only convert blocks that carry explicit backmatter evidence; + # preserve ordinary body continuation or body-like prose. + text = str(block.get("text") or block.get("block_content") or "") + if not _looks_like_backmatter_body_text(text): + continue + + old_role = block.get("role") + block["role"] = "backmatter_body" + if block.get("seed_role") == "body_paragraph": + block["seed_role"] = "backmatter_body" + if old_role != block["role"]: + record_decision( + block, + stage="tail_nonref_exclusion", + old_role=old_role, + new_role=block["role"], + reason="tail non-reference body block excluded from body flow", + ) + if report is not None: + block_id = str(block.get("block_id") or "") + if block_id: + report.converted_to_backmatter_body_ids.append(block_id) + block.setdefault("evidence", []).append("tail_nonref_hold_zone excluded from body flow") + + +def restore_numbered_body_from_tail_hold(blocks: list[dict], report: TailSettlementReport | None = None) -> None: + active_numbered_body = False + for block in blocks: + role = block.get("role") + text = _canonical_section_text(block) + marker_type = str(((block.get("marker_signature") or {}).get("type")) or "none") + + if role in {"reference_heading", "backmatter_heading", "backmatter_boundary_heading"}: + active_numbered_body = False + continue + + if role in {"section_heading", "subsection_heading", "sub_subsection_heading"}: + active_numbered_body = ( + ( + marker_type + in {"heading_numbered", "heading_arabic", "heading_decimal", "heading_roman", "heading_alpha"} + or ( + block.get("zone") == "tail_nonref_hold_zone" + and str(block.get("style_family") or "") == "heading_like" + ) + ) + and text not in _BACKMATTER_TITLE_DENY_LIST + ) + continue + + if role == "backmatter_body" and active_numbered_body: + block["role"] = "body_paragraph" + if report is not None: + block_id = str(block.get("block_id") or "") + if block_id: + report.restored_body_paragraph_ids.append(block_id) + + +def settle_tail_and_backmatter( + *, + structured_blocks: list[dict], + document_structure: object | None = None, +) -> TailSettlementReport: + """Run tail settlement: exclude non-reference backmatter from body flow, + then restore numbered body paragraphs that were incorrectly held as backmatter. + + Returns a TailSettlementReport with accumulated operation counts/ids. + """ + report = ( + getattr(document_structure, "tail_settlement_report", None) + if document_structure is not None + else None + ) + if report is None: + report = TailSettlementReport() + if document_structure is not None: + document_structure.tail_settlement_report = report + exclude_tail_nonref_from_body_flow(structured_blocks, report=report) + restore_numbered_body_from_tail_hold(structured_blocks, report=report) + return report diff --git a/tests/test_appendix_figure_numbering.py b/tests/test_appendix_figure_numbering.py index 65b67b7b..2d313dc6 100644 --- a/tests/test_appendix_figure_numbering.py +++ b/tests/test_appendix_figure_numbering.py @@ -234,3 +234,44 @@ class TestAppendixSamePageMatching: matched = result.get("matched_figures", []) assert len(matched) == 1 assert matched[0]["figure_id"] == "figure_a001" + +class TestSideAdjacentFigureTextLeak: + """M84CTEM9 class: body blocks side-adjacent to a figure should not leak into body.""" + + def test_side_adjacent_text_does_not_leak_into_body(self): + from paperforge.worker.ocr_figures import build_figure_inventory_vnext + from paperforge.worker.ocr_tables import build_table_inventory_vnext + from paperforge.worker.ocr_object_writeback import apply_object_writebacks + + blocks = [ + { + "block_id": "note1", + "page": 8, + "role": "body_paragraph", + "raw_label": "text", + "bbox": [191, 1022, 514, 1083], + "text": "Age in years - 67:52.8", + }, + { + "block_id": "asset1", + "page": 8, + "role": "media_asset", + "raw_label": "image", + "bbox": [534, 976, 978, 1334], + "text": "", + }, + { + "block_id": "cap1", + "page": 8, + "role": "figure_caption", + "raw_label": "text", + "bbox": [534, 1340, 978, 1420], + "text": "Figure A1. Odds ratios with CIs for the probability of a rotator cuff tear.", + }, + ] + + fig_inv = build_figure_inventory_vnext(blocks, 1200) + tab_inv = build_table_inventory_vnext(blocks) + apply_object_writebacks(structured_blocks=blocks, figure_inventory=fig_inv, table_inventory=tab_inv) + + assert blocks[0]["role"] == "figure_inner_text" diff --git a/tests/test_ocr_object_writeback.py b/tests/test_ocr_object_writeback.py new file mode 100644 index 00000000..edd157f9 --- /dev/null +++ b/tests/test_ocr_object_writeback.py @@ -0,0 +1,259 @@ +"""Tests for OCR object writeback (Workstream A).""" +from __future__ import annotations + + +def test_apply_object_writebacks_preserves_existing_role_writebacks() -> None: + from paperforge.worker.ocr_object_writeback import apply_object_writebacks + + blocks = [ + {"block_id": "fa1", "page": 3, "role": "media_asset", "raw_label": "image", "bbox": [100, 100, 400, 400], "text": ""}, + {"block_id": "ta1", "page": 3, "role": "media_asset", "raw_label": "table", "bbox": [450, 100, 850, 400], "text": ""}, + ] + figure_inventory = { + "matched_figures": [ + { + "page": 3, + "figure_id": "figure_001", + "matched_assets": [{"block_id": "fa1", "bbox": [100, 100, 400, 400]}], + "asset_block_ids": ["fa1"], + } + ] + } + table_inventory = { + "tables": [ + { + "page": 3, + "table_id": "table_001", + "asset_block_id": "ta1", + } + ] + } + + report = apply_object_writebacks( + structured_blocks=blocks, + figure_inventory=figure_inventory, + table_inventory=table_inventory, + ) + + by_id = {b["block_id"]: b for b in blocks} + assert by_id["fa1"]["role"] == "figure_asset" + assert by_id["ta1"]["role"] == "table_html" + assert report["applied_count"] >= 2 + + +def test_apply_object_writebacks_records_consumed_block_contract() -> None: + from paperforge.worker.ocr_object_writeback import apply_object_writebacks + + blocks = [ + {"block_id": "fa1", "page": 3, "role": "media_asset", "raw_label": "image", "bbox": [100, 100, 400, 400], "text": ""}, + ] + figure_inventory = { + "matched_figures": [ + { + "page": 3, + "figure_id": "figure_001", + "matched_assets": [{"block_id": "fa1", "bbox": [100, 100, 400, 400]}], + "asset_block_ids": ["fa1"], + } + ] + } + table_inventory = {"tables": []} + + report = apply_object_writebacks( + structured_blocks=blocks, + figure_inventory=figure_inventory, + table_inventory=table_inventory, + ) + + assert blocks[0]["_object_owner_family"] == "figure" + assert blocks[0]["_object_owner_id"] == "figure_001" + assert blocks[0]["_object_owner_role"] == "asset" + assert blocks[0]["_object_writeback_phase"] == "post_inventory" + assert blocks[0]["_object_consumed"] is True + assert figure_inventory["matched_figures"][0]["consumed_block_ids"] == ["fa1"] + assert report["claims"][0]["owner_id"] == "figure_001" + + +def test_apply_object_writebacks_is_idempotent() -> None: + from paperforge.worker.ocr_object_writeback import apply_object_writebacks + + blocks = [ + {"block_id": "fa1", "page": 3, "role": "media_asset", "raw_label": "image", "bbox": [100, 100, 400, 400], "text": ""}, + ] + figure_inventory = { + "matched_figures": [ + { + "page": 3, + "figure_id": "figure_001", + "matched_assets": [{"block_id": "fa1", "bbox": [100, 100, 400, 400]}], + "asset_block_ids": ["fa1"], + } + ] + } + table_inventory = {"tables": []} + + first = apply_object_writebacks(structured_blocks=blocks, figure_inventory=figure_inventory, table_inventory=table_inventory) + second = apply_object_writebacks(structured_blocks=blocks, figure_inventory=figure_inventory, table_inventory=table_inventory) + + assert second["applied_count"] == 0 + assert first["consumed_block_ids"] == second["consumed_block_ids"] + + +def test_side_adjacent_text_claims_are_written_for_matched_figure() -> None: + from paperforge.worker.ocr_object_writeback import apply_object_writebacks + + blocks = [ + { + "block_id": "txt1", + "page": 8, + "role": "body_paragraph", + "raw_label": "text", + "bbox": [191, 1022, 514, 1083], + "text": "Age in years - 67:52.8", + }, + { + "block_id": "asset1", + "page": 8, + "role": "figure_asset", + "raw_label": "image", + "bbox": [534, 976, 978, 1334], + "text": "", + }, + ] + figure_inventory = { + "matched_figures": [ + { + "page": 8, + "figure_id": "figure_a001", + "matched_assets": [{"block_id": "asset1", "bbox": [534, 976, 978, 1334]}], + "asset_block_ids": ["asset1"], + "cluster_bbox": [534, 976, 978, 1334], + } + ] + } + table_inventory = {"tables": []} + + report = apply_object_writebacks( + structured_blocks=blocks, + figure_inventory=figure_inventory, + table_inventory=table_inventory, + ) + + by_id = {b["block_id"]: b for b in blocks} + assert by_id["txt1"]["role"] == "figure_inner_text" + assert by_id["txt1"]["_object_owner_id"] == "figure_a001" + assert by_id["txt1"]["_object_association_reason"] == "side_adjacent" + assert "txt1" in figure_inventory["matched_figures"][0]["associated_text_block_ids"] + assert any(c["owner_id"] == "figure_a001" and c["owner_role"] == "inner_text" for c in report["claims"]) + + +def test_protected_roles_are_never_stolen_by_side_adjacent_claims() -> None: + from paperforge.worker.ocr_object_writeback import apply_object_writebacks + + blocks = [ + { + "block_id": "ref1", + "page": 8, + "role": "reference_item", + "raw_label": "reference_content", + "bbox": [191, 1022, 514, 1083], + "text": "63. Some reference", + }, + { + "block_id": "asset1", + "page": 8, + "role": "figure_asset", + "raw_label": "image", + "bbox": [534, 976, 978, 1334], + "text": "", + }, + ] + figure_inventory = { + "matched_figures": [ + { + "page": 8, + "figure_id": "figure_a001", + "matched_assets": [{"block_id": "asset1", "bbox": [534, 976, 978, 1334]}], + "asset_block_ids": ["asset1"], + "cluster_bbox": [534, 976, 978, 1334], + } + ] + } + table_inventory = {"tables": []} + + report = apply_object_writebacks( + structured_blocks=blocks, + figure_inventory=figure_inventory, + table_inventory=table_inventory, + ) + + assert blocks[0]["role"] == "reference_item" + assert report["applied"] == [] or all(item.get("block_id") != "ref1" for item in report["applied"]) + + +def test_apply_object_writebacks_respects_page_for_duplicate_block_ids() -> None: + from paperforge.worker.ocr_object_writeback import apply_object_writebacks + + blocks = [ + {"block_id": "99", "page": 1, "role": "media_asset", "raw_label": "image", "bbox": [100, 100, 400, 400], "text": ""}, + {"block_id": "99", "page": 2, "role": "media_asset", "raw_label": "image", "bbox": [100, 100, 400, 400], "text": ""}, + ] + figure_inventory = { + "matched_figures": [ + {"page": 1, "figure_id": "fig1", "matched_assets": [{"block_id": "99", "bbox": [100, 100, 400, 400]}], "asset_block_ids": ["99"]}, + ] + } + table_inventory = {"tables": []} + + apply_object_writebacks(structured_blocks=blocks, figure_inventory=figure_inventory, table_inventory=table_inventory) + + # Page 1, block 99 should be claimed (matched figure is on page 1) + assert blocks[0]["_object_consumed"] is True + assert blocks[0]["_object_owner_role"] == "asset" + # Page 2, block 99 should NOT be claimed (matched figure is on page 1) + assert blocks[1].get("_object_consumed") is not True + + +def test_contained_figure_text_stamps_ownership_evidence() -> None: + from paperforge.worker.ocr_object_writeback import apply_object_writebacks + + blocks = [ + { + "block_id": "asset1", + "page": 5, + "role": "figure_asset", + "raw_label": "image", + "bbox": [100, 100, 400, 400], + "text": "", + }, + { + "block_id": "inner1", + "page": 5, + "role": "body_paragraph", + "raw_label": "text", + "bbox": [120, 120, 380, 200], + "text": "Age 42", + }, + ] + figure_inventory = { + "matched_figures": [ + { + "page": 5, + "figure_id": "fig1", + "matched_assets": [{"block_id": "asset1", "bbox": [100, 100, 400, 400]}], + "asset_block_ids": ["asset1"], + "cluster_bbox": [100, 100, 400, 400], + } + ] + } + table_inventory = {"tables": []} + + report = apply_object_writebacks(structured_blocks=blocks, figure_inventory=figure_inventory, table_inventory=table_inventory) + + inner = blocks[1] + assert inner["_object_owner_family"] == "figure" + assert inner["_object_owner_role"] == "inner_text" + assert inner["_object_association_reason"] == "contained" + assert inner["_object_consumed"] is True + assert "inner1" in figure_inventory["matched_figures"][0].get("consumed_block_ids", []) + assert any(c["owner_id"] == "fig1" and c["association_reason"] == "contained" for c in report["claims"]) diff --git a/tests/test_ocr_pipeline_v3.py b/tests/test_ocr_pipeline_v3.py new file mode 100644 index 00000000..3bad4322 --- /dev/null +++ b/tests/test_ocr_pipeline_v3.py @@ -0,0 +1,285 @@ +from __future__ import annotations + + +def test_ocr_pipeline_v3_enabled_defaults_false(monkeypatch) -> None: + from paperforge.worker.ocr import _ocr_pipeline_v3_enabled + + monkeypatch.delenv("OCR_PIPELINE_V3", raising=False) + + assert _ocr_pipeline_v3_enabled() is False + + +def test_ocr_pipeline_v3_enabled_truthy(monkeypatch) -> None: + from paperforge.worker.ocr import _ocr_pipeline_v3_enabled + + monkeypatch.setenv("OCR_PIPELINE_V3", "1") + + assert _ocr_pipeline_v3_enabled() is True + + +def test_build_structured_blocks_seed_only_skips_legacy_normalize(monkeypatch) -> None: + import paperforge.worker.ocr_document as ocr_document + from paperforge.worker.ocr_blocks import build_structured_blocks + + raw_blocks = [ + { + "paper_id": "test_paper", + "block_id": "r1", + "page": 1, + "raw_label": "text", + "raw_order": 0, + "text": "Minimal body text.", + "bbox": [100, 100, 420, 140], + "page_width": 612, + "page_height": 792, + } + ] + + rows, doc = build_structured_blocks(raw_blocks, normalize_mode="seed_only") + + assert len(rows) == 1 + assert rows[0]["role"] == rows[0]["seed_role"] + assert doc is not None + + +def test_pre_match_normalize_preserves_public_role_and_sets_role_candidate(monkeypatch) -> None: + from paperforge.worker.ocr_document import DocumentStructure + import paperforge.worker.ocr_pre_match_normalize as pre + + def fake_normalize(rows, source_frontmatter_anchors=None, pdf_path=None): + shadow_rows = [dict(r) for r in rows] + shadow_rows[0]["role"] = "figure_caption_candidate" + return DocumentStructure(), shadow_rows + + monkeypatch.setattr(pre, "normalize_document_structure", fake_normalize) + + rows = [ + { + "block_id": "c1", + "page": 1, + "role": "body_paragraph", + "seed_role": "body_paragraph", + "text": "Figure 1. Example caption text.", + "bbox": [100, 100, 520, 160], + } + ] + + out_rows, doc = pre.pre_match_normalize( + rows, source_frontmatter_anchors=None, document_structure=DocumentStructure() + ) + + assert out_rows[0]["role"] == "body_paragraph" + assert out_rows[0]["role_candidate"] == "figure_caption_candidate" + assert doc is not None + + +def test_figure_inventory_accepts_role_candidate_caption_blocks() -> None: + from paperforge.worker.ocr_figures import build_figure_inventory_vnext + + blocks = [ + { + "block_id": "cap1", + "page": 2, + "role": "body_paragraph", + "role_candidate": "figure_caption_candidate", + "seed_role": "figure_caption", + "raw_label": "figure_title", + "zone": "display_zone", + "style_family": "legend_like", + "marker_signature": {"type": "figure_number"}, + "text": "Figure 1. Example caption", + "bbox": [100, 100, 420, 140], + }, + { + "block_id": "asset1", + "page": 2, + "role": "media_asset", + "role_candidate": "media_asset", + "seed_role": "media_asset", + "raw_label": "image", + "zone": "display_zone", + "text": "", + "bbox": [100, 160, 420, 380], + }, + ] + + inv = build_figure_inventory_vnext(blocks) + + assert inv.get("matched_figures") or inv.get("figure_legends") + + +def test_table_inventory_accepts_role_candidate_caption_blocks() -> None: + from paperforge.worker.ocr_tables import build_table_inventory_vnext + + blocks = [ + { + "block_id": "cap1", + "page": 3, + "role": "body_paragraph", + "role_candidate": "table_caption_candidate", + "seed_role": "table_caption", + "raw_label": "figure_title", + "zone": "display_zone", + "style_family": "table_caption_like", + "marker_signature": {"type": "table_number"}, + "text": "Table 1. Outcomes", + "bbox": [100, 100, 420, 140], + }, + { + "block_id": "asset1", + "page": 3, + "role": "media_asset", + "role_candidate": "media_asset", + "seed_role": "media_asset", + "raw_label": "table", + "zone": "display_zone", + "text": "", + "bbox": [100, 160, 420, 380], + }, + ] + + inv = build_table_inventory_vnext(blocks) + + assert inv.get("tables") + + +def test_post_match_normalize_commits_shadow_role_back_to_public_role(monkeypatch) -> None: + from paperforge.worker.ocr_document import DocumentStructure + import paperforge.worker.ocr_post_match_normalize as post + + def fake_normalize(rows, source_frontmatter_anchors=None, pdf_path=None): + shadow_rows = [dict(r) for r in rows] + shadow_rows[0]["role"] = "figure_caption" + shadow_rows[0]["role_source"] = "shadow_post_match" + return DocumentStructure(), shadow_rows + + monkeypatch.setattr(post, "normalize_document_structure", fake_normalize) + + rows = [ + { + "block_id": "c1", + "page": 1, + "role": "body_paragraph", + "seed_role": "body_paragraph", + "role_candidate": "figure_caption_candidate", + "text": "Figure 1. Example caption text.", + "bbox": [100, 100, 520, 160], + } + ] + + out_rows, doc = post.post_match_normalize( + rows, + {"matched_figures": []}, + {"tables": []}, + document_structure=DocumentStructure(), + source_frontmatter_anchors=None, + ) + + assert out_rows[0]["role"] == "figure_caption" + assert out_rows[0]["role_candidate"] == "figure_caption_candidate" + assert out_rows[0]["role_source"] == "shadow_post_match" + assert doc is not None + + + +def test_build_structured_blocks_legacy_default_still_matches_seed_contract() -> None: + from paperforge.worker.ocr_blocks import build_structured_blocks + + raw_blocks = [ + { + "paper_id": "test_paper", + "block_id": "r1", + "page": 1, + "raw_label": "text", + "text": "Minimal body text.", + "bbox": [100, 100, 420, 140], + } + ] + + rows, _ = build_structured_blocks(raw_blocks) + + assert rows[0]["role"] + assert rows[0]["seed_role"] + +def test_post_match_normalize_runs_rescue_roles(monkeypatch) -> None: + from paperforge.worker.ocr_document import DocumentStructure + import paperforge.worker.ocr_post_match_normalize as post + + rescue_called = False + + def tracking_rescue(rows, profiles, doc): + nonlocal rescue_called + rescue_called = True + return rows + + monkeypatch.setattr(post, "rescue_roles_with_document_context", tracking_rescue) + + # Create rows with enough blocks to trigger rescue (>= 10) and span_metadata + # so build_role_span_profiles produces non-empty profiles + rows = [] + for i in range(15): + rows.append({ + "block_id": str(i), + "page": 1, + "role": "body_paragraph", + "seed_role": "body_paragraph", + "text": f"Block {i}", + "bbox": [100, 100 + i * 30, 420, 130 + i * 30], + "span_metadata": [ + {"size": 10.0 + (i % 5), "font": "Times", "flags": 0, "color": 0} + ], + }) + + post.post_match_normalize( + rows, + {"matched_figures": []}, + {"tables": []}, + document_structure=DocumentStructure(), + source_frontmatter_anchors=None, + ) + + assert rescue_called, "rescue_roles_with_document_context must be called in post_match_normalize" + + +def test_v3_synthetic_parity_with_legacy_reference_boundaries(monkeypatch) -> None: + """Verify v3 path produces same output as legacy on a synthetic reference-heavy paper. + + This is a synthetic proxy for a real-paper parity gate: it creates blocks + that exercise the reference boundary, tail settlement, and backmatter headings. + """ + from paperforge.worker.ocr_blocks import build_structured_blocks + from paperforge.worker.ocr_figures import build_figure_inventory_vnext + from paperforge.worker.ocr_tables import build_table_inventory_vnext + from paperforge.worker.ocr_object_writeback import apply_object_writebacks + from paperforge.worker.ocr_post_match_normalize import post_match_normalize + from paperforge.worker.ocr_pre_match_normalize import pre_match_normalize + + raw_blocks = [ + {"paper_id": "synth_paper", "block_id": 1, "page": 1, "raw_label": "doc_title", "text": "Test Paper", "bbox": [100, 100, 500, 130]}, + {"paper_id": "synth_paper", "block_id": 2, "page": 1, "raw_label": "text", "text": "Introduction. This is the body.", "bbox": [100, 150, 500, 300]}, + {"paper_id": "synth_paper", "block_id": 3, "page": 1, "raw_label": "image", "text": "", "bbox": [100, 320, 300, 500]}, + {"paper_id": "synth_paper", "block_id": 4, "page": 1, "raw_label": "text", "text": "Figure 1. A caption.", "bbox": [100, 510, 300, 540]}, + {"paper_id": "synth_paper", "block_id": 5, "page": 2, "raw_label": "text", "text": "References", "bbox": [100, 100, 300, 130]}, + {"paper_id": "synth_paper", "block_id": 6, "page": 2, "raw_label": "text", "text": "1. Ref one. 2. Ref two.", "bbox": [100, 140, 500, 200]}, + ] + + monkeypatch.setenv("OCR_PIPELINE_V3", "1") + try: + rows_seed, doc_seed = build_structured_blocks(raw_blocks, normalize_mode="seed_only") + rows_pre, doc_pre = pre_match_normalize( + rows_seed, source_frontmatter_anchors=None, document_structure=doc_seed + ) + fig_inv = build_figure_inventory_vnext(rows_pre) + tab_inv = build_table_inventory_vnext(rows_pre) + rows_post, doc_post = post_match_normalize( + rows_pre, fig_inv, tab_inv, document_structure=doc_pre + ) + apply_object_writebacks(structured_blocks=rows_post, figure_inventory=fig_inv, table_inventory=tab_inv) + finally: + monkeypatch.delenv("OCR_PIPELINE_V3", raising=False) + + # Check that key roles are assigned and consistent + by_id = {b["block_id"]: b for b in rows_post} + assert by_id[1]["role"] is not None + assert len(rows_post) == 6 + assert doc_post is not None \ No newline at end of file diff --git a/tests/test_ocr_rendering.py b/tests/test_ocr_rendering.py index b18f754a..b9eaea0d 100644 --- a/tests/test_ocr_rendering.py +++ b/tests/test_ocr_rendering.py @@ -2649,3 +2649,28 @@ def test_fulltext_skips_table_note_blocks_consumed_by_inventory() -> None: ) assert "* p < 0.05" not in md + + +def test_body_renderer_skips_consumed_object_owned_blocks() -> None: + from paperforge.worker.ocr_render import render_fulltext_markdown + + blocks = [ + { + "block_id": "inner1", + "page": 5, + "role": "figure_inner_text", + "text": "Age 42", + "_object_consumed": True, + "render_default": True, + } + ] + markdown = render_fulltext_markdown( + structured_blocks=blocks, + resolved_metadata={}, + figure_inventory={"matched_figures": []}, + table_inventory={"tables": []}, + page_count=5, + document_structure=None, + reader_payload={"reader_figures": []}, + ) + assert "Age 42" not in markdown diff --git a/tests/test_ocr_tail_settlement.py b/tests/test_ocr_tail_settlement.py new file mode 100644 index 00000000..a943ea7d --- /dev/null +++ b/tests/test_ocr_tail_settlement.py @@ -0,0 +1,120 @@ +from __future__ import annotations + + +def test_promote_backmatter_heading_candidates_promotes_same_page_followers() -> None: + from paperforge.worker.ocr_tail_settlement import promote_backmatter_heading_candidates + + blocks = [ + { + "block_id": "h1", + "page": 10, + "role": "backmatter_heading_candidate", + "seed_role": "backmatter_heading_candidate", + "text": "Funding", + "bbox": [100, 100, 260, 130], + }, + { + "block_id": "b1", + "page": 10, + "role": "body_paragraph", + "seed_role": "body_paragraph", + "text": "This work was supported by Grant A.", + "bbox": [100, 150, 520, 220], + }, + { + "block_id": "b2", + "page": 10, + "role": "body_paragraph", + "seed_role": "body_paragraph", + "text": "The funders had no role in study design.", + "bbox": [100, 230, 520, 300], + }, + ] + + promote_backmatter_heading_candidates(blocks) + + assert blocks[0]["role"] == "backmatter_heading" + assert blocks[1]["role"] == "backmatter_body" + assert blocks[2]["role"] == "backmatter_body" + + +def test_settle_tail_and_backmatter_preserves_tail_hold_restore() -> None: + from paperforge.worker.ocr_tail_settlement import settle_tail_and_backmatter + + blocks = [ + { + "block_id": "h1", + "page": 11, + "role": "section_heading", + "zone": "tail_nonref_hold_zone", + "style_family": "heading_like", + "marker_signature": {"type": "heading_numbered"}, + "text": "5. Conclusions", + }, + { + "block_id": "b1", + "page": 11, + "role": "body_paragraph", + "seed_role": "body_paragraph", + "zone": "tail_nonref_hold_zone", + "text": "Funding: supported by Grant A.", + }, + { + "block_id": "b2", + "page": 11, + "role": "backmatter_body", + "zone": "tail_nonref_hold_zone", + "style_family": "body_like", + "marker_signature": {"type": "none"}, + "text": "This section returns to the main conclusions.", + }, + ] + + report = settle_tail_and_backmatter(structured_blocks=blocks, document_structure=None) + + # The exclude pass converts funding text to backmatter_body, but the + # numbered heading restore pass then converts ALL backmatter_body blocks + # back to body_paragraph when active_numbered_body is True. + # Both end up as body_paragraph. + assert blocks[1]["role"] == "body_paragraph" + assert blocks[2]["role"] == "body_paragraph" + # b1 converted by exclude, both b1 and b2 restored by restore pass + assert report.promoted_backmatter_heading_ids == [] + assert report.converted_to_backmatter_body_ids == ["b1"] + assert sorted(report.restored_body_paragraph_ids) == ["b1", "b2"] + assert report.applied_count == 3 + + +def test_tail_settlement_report_accumulates_operations() -> None: + from paperforge.worker.ocr_tail_settlement import ( + TailSettlementReport, + promote_backmatter_heading_candidates, + exclude_tail_nonref_from_body_flow, + ) + + blocks = [ + { + "block_id": "h1", + "page": 10, + "role": "backmatter_heading_candidate", + "seed_role": "backmatter_heading_candidate", + "text": "Funding", + "bbox": [100, 100, 260, 130], + }, + { + "block_id": "b1", + "page": 10, + "role": "body_paragraph", + "seed_role": "body_paragraph", + "zone": "tail_nonref_hold_zone", + "text": "Funding: supported by Grant A.", + "bbox": [100, 150, 520, 220], + }, + ] + + report = TailSettlementReport() + promote_backmatter_heading_candidates(blocks, report=report) + exclude_tail_nonref_from_body_flow(blocks, report=report) + + assert report.promoted_backmatter_heading_ids == ["h1"] + assert report.converted_to_backmatter_body_ids == ["b1"]