obsidian-plugin/lllin000_PaperForge
1
0
Fork 0
mirror of https://github.com/lllin000/PaperForge.git synced 2026-07-22 06:50:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge branch 'feat/ocr-tail-settlement'

Browse source
This commit is contained in:
LLLin000 2026-07-04 17:52:57 +08:00
commit 6cbb82aa22
20 changed files with 3026 additions and 198 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

675
docs/superpowers/plans/2026-07-04-ocr-pipeline-v3-implementation-plan.md Normal file
View file

@ -0,0 +1,675 @@
# OCR Pipeline V3 Split Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Split the current monolithic normalize path into `pre_match_normalize(...)` and `post_match_normalize(...)`, keep the public `block["role"]` contract intact, and gate the entire v3 path behind `OCR_PIPELINE_V3` so legacy callers stay untouched by default.
**Architecture:** Keep the legacy path as the default. Add a seed-only mode to `build_structured_blocks(...)`, then let the v3 path run pre-match normalization, figure/table matching, and post-match normalization in `ocr.py`. To keep blast radius low, the first v3 implementation uses shadow calls into `normalize_document_structure(...)` to compute candidate/final roles while preserving the new sequencing and matching contract.
**Tech Stack:** Python 3, existing PaperForge OCR workers, `DocumentStructure`, `ocr_tail_settlement.py`, pytest, environment toggle via `OCR_PIPELINE_V3`
## Global Constraints
- Workstream C only. Workstreams A and B are already landed on this branch and must stay green.
- `block["role"]` remains the final public role field.
- Under v3, matching code must not require final `role`; it must work from `seed_role`, `role_candidate`, `raw_label`, `bbox`, `zone`, and ownership evidence when present.
- The entire v3 path is behind `OCR_PIPELINE_V3`; legacy behavior is the default and must remain byte-for-byte compatible for existing callers.
- Do not rewrite figure/table pairing logic in this workstream. Change only the inputs those builders accept.
- Do not change object writeback sequencing from Workstream A. `apply_object_writebacks(...)` remains after figure/table inventory exists.
- Reuse the existing `settle_tail_and_backmatter(...)` module from Workstream B. Do not re-inline tail settlement.
- If v3 parity is not green, stop at the gate. Do not force-switch callers to v3.
---
## File Structure
- **Create:** `paperforge/worker/ocr_pre_match_normalize.py`
- v3 candidate-only normalization. Preserves `role = seed_role`, writes `role_candidate` from a shadow normalize pass.
- **Create:** `paperforge/worker/ocr_post_match_normalize.py`
- v3 final role commit after figure/table inventories exist. Reuses tail settlement from Workstream B.
- **Modify:** `paperforge/worker/ocr_blocks.py`
- add `normalize_mode="seed_only" | "legacy"` to `build_structured_blocks(...)`
- return rows/doc before legacy normalize when v3 wants seed-only rows
- **Modify:** `paperforge/worker/ocr.py`
- add `_ocr_pipeline_v3_enabled()`
- branch between legacy path and v3 pre-match → inventory build → post-match path
- **Modify:** `paperforge/worker/ocr_figures.py`
- add a helper that resolves match-time role from `role_candidate` / `role` / `seed_role`
- replace role checks used by figure matching with the helper
- **Modify:** `paperforge/worker/ocr_tables.py`
- same candidate-role helper approach for table matching
- **Create:** `tests/test_ocr_pipeline_v3.py`
- toggle tests, pre-match tests, post-match tests, and one figure/table contract test each
---
### Task 1: C0 — Add the v3 toggle and seed-only build path
**Files:**
- Create: `paperforge/worker/ocr_pre_match_normalize.py`
- Create: `paperforge/worker/ocr_post_match_normalize.py`
- Modify: `paperforge/worker/ocr_blocks.py`
- Modify: `paperforge/worker/ocr.py`
- Create: `tests/test_ocr_pipeline_v3.py`
**Interfaces:**
- Consumes:
- current seed-row construction in `build_structured_blocks(...)`
- current top-level OCR flow in `ocr.py`
- Produces:
- `_ocr_pipeline_v3_enabled() -> bool`
- `build_structured_blocks(..., normalize_mode: str = "legacy")`
- `pre_match_normalize(rows: list[dict], *, source_frontmatter_anchors: dict | None = None, document_structure: DocumentStructure | None = None) -> tuple[list[dict], DocumentStructure]`
- `post_match_normalize(rows: list[dict], figure_inventory: dict, table_inventory: dict, *, document_structure: DocumentStructure, source_frontmatter_anchors: dict | None = None) -> tuple[list[dict], DocumentStructure]`
- [ ] **Step 1: Write the failing tests**
```python
# tests/test_ocr_pipeline_v3.py
from __future__ import annotations
def test_ocr_pipeline_v3_enabled_defaults_false(monkeypatch) -> None:
from paperforge.worker.ocr import _ocr_pipeline_v3_enabled
monkeypatch.delenv("OCR_PIPELINE_V3", raising=False)
assert _ocr_pipeline_v3_enabled() is False
def test_ocr_pipeline_v3_enabled_truthy(monkeypatch) -> None:
from paperforge.worker.ocr import _ocr_pipeline_v3_enabled
monkeypatch.setenv("OCR_PIPELINE_V3", "1")
assert _ocr_pipeline_v3_enabled() is True
def test_build_structured_blocks_seed_only_skips_legacy_normalize(monkeypatch) -> None:
import paperforge.worker.ocr_document as ocr_document
from paperforge.worker.ocr_blocks import build_structured_blocks
def boom(*args, **kwargs):
raise AssertionError("legacy normalize should not run in seed_only mode")
monkeypatch.setattr(ocr_document, "normalize_document_structure", boom)
raw_blocks = [
{
"block_id": "r1",
"page": 1,
"raw_label": "text",
"text": "Minimal body text.",
"bbox": [100, 100, 420, 140],
}
]
rows, doc = build_structured_blocks(raw_blocks, normalize_mode="seed_only")
assert len(rows) == 1
assert rows[0]["role"] == rows[0]["seed_role"]
assert doc is not None
```
- [ ] **Step 2: Run tests to verify they fail**
Run:
`python -m pytest tests/test_ocr_pipeline_v3.py::test_ocr_pipeline_v3_enabled_defaults_false tests/test_ocr_pipeline_v3.py::test_ocr_pipeline_v3_enabled_truthy tests/test_ocr_pipeline_v3.py::test_build_structured_blocks_seed_only_skips_legacy_normalize -v --tb=short`
Expected: FAIL because `_ocr_pipeline_v3_enabled`, `normalize_mode`, and the new modules do not exist yet.
- [ ] **Step 3: Write minimal implementation**
```python
# paperforge/worker/ocr.py
import os
def _ocr_pipeline_v3_enabled() -> bool:
return os.environ.get("OCR_PIPELINE_V3", "").strip().lower() in {"1", "true", "yes", "on"}
```
```python
# paperforge/worker/ocr_blocks.py
def build_structured_blocks(
raw_blocks: list[dict],
source_metadata: dict | None = None,
structure_output_dir: str | Path | None = None,
normalize_mode: str = "legacy",
) -> tuple[list[dict], DocumentStructure]:
...
body_family_anchor = discover_body_family_anchor(rows, page_count=total_pages)
doc_structure = DocumentStructure(body_family_anchor=body_family_anchor)
...
if normalize_mode == "seed_only":
return rows, doc_structure
...
doc_structure, rows = normalize_document_structure(rows, source_frontmatter_anchors=_sfm_anchors)
...
settle_tail_and_backmatter(structured_blocks=rows, document_structure=doc_structure)
return rows, doc_structure
```
```python
# paperforge/worker/ocr_pre_match_normalize.py
from __future__ import annotations
from paperforge.worker.ocr_document import DocumentStructure
def pre_match_normalize(
rows: list[dict],
*,
source_frontmatter_anchors: dict | None = None,
document_structure: DocumentStructure | None = None,
) -> tuple[list[dict], DocumentStructure]:
return rows, document_structure or DocumentStructure()
```
```python
# paperforge/worker/ocr_post_match_normalize.py
from __future__ import annotations
from paperforge.worker.ocr_document import DocumentStructure
def post_match_normalize(
rows: list[dict],
figure_inventory: dict,
table_inventory: dict,
*,
document_structure: DocumentStructure,
source_frontmatter_anchors: dict | None = None,
) -> tuple[list[dict], DocumentStructure]:
return rows, document_structure
```
- [ ] **Step 4: Run tests to verify they pass**
Run:
`python -m pytest tests/test_ocr_pipeline_v3.py::test_ocr_pipeline_v3_enabled_defaults_false tests/test_ocr_pipeline_v3.py::test_ocr_pipeline_v3_enabled_truthy tests/test_ocr_pipeline_v3.py::test_build_structured_blocks_seed_only_skips_legacy_normalize -v --tb=short`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ocr.py paperforge/worker/ocr_blocks.py paperforge/worker/ocr_pre_match_normalize.py paperforge/worker/ocr_post_match_normalize.py tests/test_ocr_pipeline_v3.py
git commit -m "feat: add OCR pipeline v3 toggle and seed-only path"
```
---
### Task 2: C1 — Implement pre-match normalize and the matching contract
**Files:**
- Modify: `paperforge/worker/ocr_pre_match_normalize.py`
- Modify: `paperforge/worker/ocr_figures.py`
- Modify: `paperforge/worker/ocr_tables.py`
- Modify: `tests/test_ocr_pipeline_v3.py`
**Interfaces:**
- Consumes:
- seed-only rows from `build_structured_blocks(..., normalize_mode="seed_only")`
- legacy `normalize_document_structure(...)` on a shadow copy
- Produces:
- `role_candidate` filled from shadow normalize results
- figure/table builders that accept `role_candidate` instead of requiring final `role`
- [ ] **Step 1: Write the failing tests**
```python
# tests/test_ocr_pipeline_v3.py
from __future__ import annotations
def test_pre_match_normalize_preserves_public_role_and_sets_role_candidate(monkeypatch) -> None:
from paperforge.worker.ocr_document import DocumentStructure
import paperforge.worker.ocr_pre_match_normalize as pre
def fake_normalize(rows, source_frontmatter_anchors=None, pdf_path=None):
shadow_rows = [dict(r) for r in rows]
shadow_rows[0]["role"] = "figure_caption_candidate"
return DocumentStructure(), shadow_rows
monkeypatch.setattr(pre, "normalize_document_structure", fake_normalize)
rows = [
{
"block_id": "c1",
"page": 1,
"role": "body_paragraph",
"seed_role": "body_paragraph",
"text": "Figure 1. Example caption text.",
"bbox": [100, 100, 520, 160],
}
]
out_rows, doc = pre.pre_match_normalize(rows, source_frontmatter_anchors=None, document_structure=DocumentStructure())
assert out_rows[0]["role"] == "body_paragraph"
assert out_rows[0]["role_candidate"] == "figure_caption_candidate"
assert doc is not None
def test_figure_inventory_accepts_role_candidate_caption_blocks() -> None:
from paperforge.worker.ocr_figures import build_figure_inventory_vnext
blocks = [
{
"block_id": "cap1",
"page": 2,
"role": "body_paragraph",
"role_candidate": "figure_caption_candidate",
"seed_role": "figure_caption",
"raw_label": "figure_title",
"zone": "display_zone",
"style_family": "legend_like",
"marker_signature": {"type": "figure_number"},
"text": "Figure 1. Example caption",
"bbox": [100, 100, 420, 140],
},
{
"block_id": "asset1",
"page": 2,
"role": "media_asset",
"role_candidate": "media_asset",
"seed_role": "media_asset",
"raw_label": "image",
"zone": "display_zone",
"text": "",
"bbox": [100, 160, 420, 380],
},
]
inv = build_figure_inventory_vnext(blocks)
assert inv["matched_figures"] or inv["figure_legends"]
def test_table_inventory_accepts_role_candidate_caption_blocks() -> None:
from paperforge.worker.ocr_tables import build_table_inventory_vnext
blocks = [
{
"block_id": "cap1",
"page": 3,
"role": "body_paragraph",
"role_candidate": "table_caption_candidate",
"seed_role": "table_caption",
"raw_label": "figure_title",
"zone": "display_zone",
"style_family": "table_caption_like",
"marker_signature": {"type": "table_number"},
"text": "Table 1. Outcomes",
"bbox": [100, 100, 420, 140],
},
{
"block_id": "asset1",
"page": 3,
"role": "media_asset",
"role_candidate": "media_asset",
"seed_role": "media_asset",
"raw_label": "table",
"zone": "display_zone",
"text": "",
"bbox": [100, 160, 420, 380],
},
]
inv = build_table_inventory_vnext(blocks)
assert inv["tables"]
```
- [ ] **Step 2: Run tests to verify they fail**
Run:
`python -m pytest tests/test_ocr_pipeline_v3.py::test_pre_match_normalize_preserves_public_role_and_sets_role_candidate tests/test_ocr_pipeline_v3.py::test_figure_inventory_accepts_role_candidate_caption_blocks tests/test_ocr_pipeline_v3.py::test_table_inventory_accepts_role_candidate_caption_blocks -v --tb=short`
Expected: FAIL because `pre_match_normalize(...)` does not populate `role_candidate`, and figure/table builders still key too heavily off final `role`.
- [ ] **Step 3: Write minimal implementation**
```python
# paperforge/worker/ocr_pre_match_normalize.py
from __future__ import annotations
from paperforge.worker.ocr_document import DocumentStructure, normalize_document_structure
def pre_match_normalize(
rows: list[dict],
*,
source_frontmatter_anchors: dict | None = None,
document_structure: DocumentStructure | None = None,
) -> tuple[list[dict], DocumentStructure]:
live_rows = [dict(row) for row in rows]
shadow_doc, shadow_rows = normalize_document_structure(
[dict(row) for row in rows],
source_frontmatter_anchors=source_frontmatter_anchors,
)
for live, shadow in zip(live_rows, shadow_rows):
live["role_candidate"] = shadow.get("role") or shadow.get("seed_role") or live.get("seed_role")
live["zone"] = shadow.get("zone", live.get("zone"))
live["style_family"] = shadow.get("style_family", live.get("style_family"))
live["marker_signature"] = shadow.get("marker_signature", live.get("marker_signature"))
live["render_default"] = live.get("render_default", True)
live["role"] = live.get("seed_role") or live.get("role")
return live_rows, document_structure or shadow_doc
```
```python
# paperforge/worker/ocr_figures.py
def _match_role(block: dict) -> str:
return str(block.get("role_candidate") or block.get("role") or block.get("seed_role") or "")
```
```python
# paperforge/worker/ocr_figures.py
# replace match-time role reads like:
role = str(block.get("role") or "")
# with:
role = _match_role(block)
```
Replace that pattern anywhere figure matching decides whether a block is a caption candidate, figure asset, or excluded demoted body paragraph. Do not change post-writeback render-only checks.
```python
# paperforge/worker/ocr_tables.py
def _match_role(block: dict) -> str:
return str(block.get("role_candidate") or block.get("role") or block.get("seed_role") or "")
```
```python
# paperforge/worker/ocr_tables.py
# replace match-time role reads like:
role = str(block.get("role", "") or "")
# with:
role = _match_role(block)
```
Replace only the role reads that affect caption/asset selection for matching. Leave write-back and render-only code alone.
- [ ] **Step 4: Run tests to verify they pass**
Run:
`python -m pytest tests/test_ocr_pipeline_v3.py::test_pre_match_normalize_preserves_public_role_and_sets_role_candidate tests/test_ocr_pipeline_v3.py::test_figure_inventory_accepts_role_candidate_caption_blocks tests/test_ocr_pipeline_v3.py::test_table_inventory_accepts_role_candidate_caption_blocks -v --tb=short`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ocr_pre_match_normalize.py paperforge/worker/ocr_figures.py paperforge/worker/ocr_tables.py tests/test_ocr_pipeline_v3.py
git commit -m "feat: add pre-match normalize and candidate-role matching"
```
---
### Task 3: C2 — Implement post-match normalize and top-level v3 orchestration
**Files:**
- Modify: `paperforge/worker/ocr_post_match_normalize.py`
- Modify: `paperforge/worker/ocr.py`
- Modify: `tests/test_ocr_pipeline_v3.py`
**Interfaces:**
- Consumes:
- pre-match rows with `role_candidate`
- `figure_inventory` and `table_inventory`
- existing `DocumentStructure`
- Produces:
- final public `role`
- reuse of `settle_tail_and_backmatter(...)` after post-match commit
- v3 sequencing in `ocr.py`
- [ ] **Step 1: Write the failing tests**
```python
# tests/test_ocr_pipeline_v3.py
from __future__ import annotations
def test_post_match_normalize_commits_shadow_role_back_to_public_role(monkeypatch) -> None:
from paperforge.worker.ocr_document import DocumentStructure
import paperforge.worker.ocr_post_match_normalize as post
def fake_normalize(rows, source_frontmatter_anchors=None, pdf_path=None):
shadow_rows = [dict(r) for r in rows]
shadow_rows[0]["role"] = "figure_caption"
shadow_rows[0]["role_source"] = "shadow_post_match"
return DocumentStructure(), shadow_rows
monkeypatch.setattr(post, "normalize_document_structure", fake_normalize)
rows = [
{
"block_id": "c1",
"page": 1,
"role": "body_paragraph",
"seed_role": "body_paragraph",
"role_candidate": "figure_caption_candidate",
"text": "Figure 1. Example caption text.",
"bbox": [100, 100, 520, 160],
}
]
out_rows, doc = post.post_match_normalize(
rows,
{"matched_figures": []},
{"tables": []},
document_structure=DocumentStructure(),
source_frontmatter_anchors=None,
)
assert out_rows[0]["role"] == "figure_caption"
assert out_rows[0]["role_candidate"] == "figure_caption_candidate"
assert out_rows[0]["role_source"] == "shadow_post_match"
assert doc is not None
```
- [ ] **Step 2: Run test to verify it fails**
Run:
`python -m pytest tests/test_ocr_pipeline_v3.py::test_post_match_normalize_commits_shadow_role_back_to_public_role -v --tb=short`
Expected: FAIL because `post_match_normalize(...)` is still a no-op.
- [ ] **Step 3: Write minimal implementation**
```python
# paperforge/worker/ocr_post_match_normalize.py
from __future__ import annotations
from paperforge.worker.ocr_document import DocumentStructure, normalize_document_structure
from paperforge.worker.ocr_tail_settlement import settle_tail_and_backmatter
def post_match_normalize(
rows: list[dict],
figure_inventory: dict,
table_inventory: dict,
*,
document_structure: DocumentStructure,
source_frontmatter_anchors: dict | None = None,
) -> tuple[list[dict], DocumentStructure]:
live_rows = [dict(row) for row in rows]
shadow_doc, shadow_rows = normalize_document_structure(
[dict(row) for row in rows],
source_frontmatter_anchors=source_frontmatter_anchors,
)
for live, shadow in zip(live_rows, shadow_rows):
live["role"] = shadow.get("role", live.get("role"))
live["role_source"] = shadow.get("role_source", live.get("role_source"))
live["role_confidence"] = shadow.get("role_confidence", live.get("role_confidence"))
live["role_candidate"] = live.get("role_candidate") or shadow.get("role")
live["render_default"] = shadow.get("render_default", live.get("render_default"))
live["index_default"] = shadow.get("index_default", live.get("index_default"))
settle_tail_and_backmatter(structured_blocks=live_rows, document_structure=shadow_doc)
return live_rows, shadow_doc
```
```python
# paperforge/worker/ocr.py
from paperforge.worker.ocr_pre_match_normalize import pre_match_normalize
from paperforge.worker.ocr_post_match_normalize import post_match_normalize
...
normalize_mode = "seed_only" if _ocr_pipeline_v3_enabled() else "legacy"
structured, doc_structure = build_structured_blocks(
all_raw_blocks,
source_metadata=source_meta,
structure_output_dir=artifacts.blocks_structured.parent,
normalize_mode=normalize_mode,
)
if _ocr_pipeline_v3_enabled():
structured, doc_structure = pre_match_normalize(
structured,
source_frontmatter_anchors=getattr(doc_structure, "source_frontmatter_anchors", None),
document_structure=doc_structure,
)
write_structured_blocks_jsonl(artifacts.blocks_structured, structured)
...
figure_inventory = build_figure_inventory(structured)
...
table_inventory = build_table_inventory(structured)
if _ocr_pipeline_v3_enabled():
structured, doc_structure = post_match_normalize(
structured,
figure_inventory,
table_inventory,
document_structure=doc_structure,
source_frontmatter_anchors=getattr(doc_structure, "source_frontmatter_anchors", None),
)
# object writeback stays here
apply_object_writebacks(
structured_blocks=structured,
figure_inventory=figure_inventory,
table_inventory=table_inventory,
)
```
The top-level order under v3 is now:
1. seed-only block build
2. pre-match normalize
3. figure/table inventory build
4. post-match normalize
5. object writeback
The legacy path stays unchanged.
- [ ] **Step 4: Run tests to verify they pass**
Run:
`python -m pytest tests/test_ocr_pipeline_v3.py::test_post_match_normalize_commits_shadow_role_back_to_public_role -v --tb=short`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ocr.py paperforge/worker/ocr_post_match_normalize.py tests/test_ocr_pipeline_v3.py
git commit -m "feat: wire OCR pipeline v3 pre-match and post-match flow"
```
---
### Task 4: C3 — Parity gate for v3
**Files:**
- Modify: `tests/test_ocr_pipeline_v3.py`
- Reuse: `tests/test_ocr_tail_settlement.py`
- Reuse: `tests/test_ocr_object_writeback.py`
- Reuse: `tests/test_ocr_rendering.py`
- Reuse: `tests/test_appendix_figure_numbering.py`
**Interfaces:**
- Consumes:
- C0/C1/C2 v3 path
- A/B regressions
- Produces:
- go / no-go for turning on v3 in any real caller
- [ ] **Step 1: Add one v3 integration smoke test**
```python
# tests/test_ocr_pipeline_v3.py
from __future__ import annotations
def test_build_structured_blocks_legacy_default_still_matches_seed_contract() -> None:
from paperforge.worker.ocr_blocks import build_structured_blocks
raw_blocks = [
{
"block_id": "r1",
"page": 1,
"raw_label": "text",
"text": "Minimal body text.",
"bbox": [100, 100, 420, 140],
}
]
rows, _ = build_structured_blocks(raw_blocks)
assert rows[0]["role"]
assert rows[0]["seed_role"]
```
- [ ] **Step 2: Run the v3-focused suite**
Run:
`python -m pytest tests/test_ocr_pipeline_v3.py tests/test_ocr_tail_settlement.py -q --tb=line`
Expected: PASS
- [ ] **Step 3: Run cross-workstream guard suite**
Run:
`python -m pytest tests/test_ocr_object_writeback.py tests/test_ocr_rendering.py::test_body_renderer_skips_consumed_object_owned_blocks tests/test_appendix_figure_numbering.py -q --tb=line`
Expected: PASS
- [ ] **Step 4: Run tail/backmatter parity suite**
Run:
`python -m pytest tests/test_ocr_rendering.py::test_tail_zone_noise_band_guard tests/test_ocr_rendering.py::test_tail_zone_supplementary_material_not_noise tests/test_ocr_rendering.py::test_tail_candidate_overreach_does_not_absorb_late_body tests/test_ocr_rendering.py::test_cross_page_funding_continuation_preserves_order tests/test_ocr_rendering.py::test_mixed_tail_page_keeps_late_body_out_of_funding_and_attaches_real_funding tests/test_ocr_rendering.py::test_backmatter_boundary_normalizes_child_sections_before_references -q --tb=line`
Expected: PASS
- [ ] **Step 5: Stop rule**
Decision rule:
- If all C tests and all A/B regressions are green, Workstream C is complete but still optional behind `OCR_PIPELINE_V3`.
- Do **not** flip the default to v3 in this workstream.
- Any future default-on change requires a separate micro-plan after corpus-diff review.
- [ ] **Step 6: Commit**
```bash
git add tests/test_ocr_pipeline_v3.py
git commit -m "test: add OCR pipeline v3 parity gate"
```
---
## Self-Review
- **Spec coverage:** This plan covers Workstream C from `2026-07-04-ocr-pipeline-deepening-design.md`: pre-match module, post-match module, role-field contract, matching contract, and behind-toggle migration rule.
- **Grounding against current code:** The plan matches the current placement of figure/table inventory building in `ocr.py` and the current seed-row construction in `ocr_blocks.py`. That is why the v3 split is staged across both files rather than only inside `normalize_document_structure(...)`.
- **Placeholder scan:** No `TBD` / `TODO` implementation placeholders remain.
- **Scope control:** The plan keeps v3 behind `OCR_PIPELINE_V3` and explicitly does not flip the default.
- **Type consistency:** `role` stays final public role; `role_candidate` is candidate-only; `seed_role` remains the early guess.
## Execution Handoff
**Plan complete and saved to `docs/superpowers/plans/2026-07-04-ocr-pipeline-v3-implementation-plan.md`. Two execution options:**
**1. Subagent-Driven (recommended)** - I dispatch a fresh subagent per task, review between tasks, fast iteration
**2. Inline Execution** - Execute tasks in this session using executing-plans, batch execution with checkpoints
**Which approach?**

662
docs/superpowers/plans/2026-07-04-ocr-tail-settlement-implementation-plan.md Normal file
View file

@ -0,0 +1,662 @@
# OCR Tail Settlement Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Extract the current tail/body/backmatter settlement logic into a dedicated module, preserve current behavior while removing cross-module leakage between `normalize_document_structure()` and `build_structured_blocks()`, then add a settlement report and a hard corpus-diff gate before any policy change.
**Architecture:** Keep the current OCR pipeline order unchanged. First move the existing tail settlement implementation into `paperforge/worker/ocr_tail_settlement.py` and route current call sites through it without changing behavior. Then introduce `TailSettlementReport` so normalize-phase promotion/exclusion and build-phase restore both write into one report attached to `DocumentStructure`. B2 is a verification gate, not a policy rewrite: if the diff is not green, stop.
**Tech Stack:** Python 3, existing PaperForge OCR workers, `DocumentStructure`, pytest, current tail/backmatter render regressions in `tests/test_ocr_rendering.py`
## Global Constraints
- Workstream B only: implement **B0 / B1 / B2**; do not touch Workstream C.
- Workstream A is already landed on this branch (`feat/ocr-tail-settlement`, `964e05b`); do not regress `ocr_object_writeback` behavior.
- B0/B1 are **behavior-preserving extraction only**. Do **not** change tail/backmatter/reference policy in those tasks.
- Do **not** reorder the main OCR pipeline.
- Do **not** change figure/table ownership behavior.
- Keep `block["role"]` as the current public final role contract.
- Because `DocumentStructure` is only materialized late in `normalize_document_structure()` (`ocr_document.py:5678` today), normalize-phase extraction must use module-local helpers first; the public `settle_tail_and_backmatter(...)` wrapper only owns the post-normalize pass until B1 attaches the shared report.
- B2 may only adjust policy after the corpus diff is green. If the diff is not green, stop after instrumentation/reporting and do **not** commit any policy change.
---
## File Structure
- **Create:** `paperforge/worker/ocr_tail_settlement.py`
- Owns the extracted tail settlement helpers and the public `settle_tail_and_backmatter(...)` wrapper.
- Hosts `promote_backmatter_heading_candidates(...)`, `exclude_tail_nonref_from_body_flow(...)`, `restore_numbered_body_from_tail_hold(...)`.
- Hosts `TailSettlementReport` in B1.
- **Modify:** `paperforge/worker/ocr_document.py`
- Replace the inline backmatter-candidate promotion loop and direct tail exclusion call with imports from `ocr_tail_settlement.py`.
- Attach the shared `TailSettlementReport` to `DocumentStructure` in B1.
- **Modify:** `paperforge/worker/ocr_blocks.py`
- Replace the direct `_exclude_tail_nonref_from_body_flow(...)` / `_restore_numbered_body_from_tail_hold(...)` imports and calls with `settle_tail_and_backmatter(...)`.
- **Create:** `tests/test_ocr_tail_settlement.py`
- Unit tests for B0/B1 extraction behavior and report contents.
- **Reuse existing tests:** `tests/test_ocr_rendering.py`
- Existing tail/backmatter regressions remain the acceptance suite for behavior-preserving extraction.
---
### Task 1: B0 — Extract the existing tail settlement seam into one deep module
**Files:**
- Create: `paperforge/worker/ocr_tail_settlement.py`
- Modify: `paperforge/worker/ocr_document.py`
- Modify: `paperforge/worker/ocr_blocks.py`
- Create: `tests/test_ocr_tail_settlement.py`
**Interfaces:**
- Consumes:
- `structured_blocks: list[dict]`
- optional `document_structure: object | None`
- current block fields already produced by `normalize_document_structure()` (`role`, `seed_role`, `zone`, `style_family`, `marker_signature`, `text`, `bbox`)
- Produces:
- `promote_backmatter_heading_candidates(blocks: list[dict]) -> None`
- `exclude_tail_nonref_from_body_flow(blocks: list[dict]) -> None`
- `restore_numbered_body_from_tail_hold(blocks: list[dict]) -> None`
- `settle_tail_and_backmatter(*, structured_blocks: list[dict], document_structure: object | None = None) -> dict`
- behavior-preserving delegation of the current helper order
- [ ] **Step 1: Write the failing tests**
```python
# tests/test_ocr_tail_settlement.py
from __future__ import annotations
def test_promote_backmatter_heading_candidates_promotes_same_page_followers() -> None:
from paperforge.worker.ocr_tail_settlement import promote_backmatter_heading_candidates
blocks = [
{
"block_id": "h1",
"page": 10,
"role": "backmatter_heading_candidate",
"seed_role": "backmatter_heading_candidate",
"text": "Funding",
"bbox": [100, 100, 260, 130],
},
{
"block_id": "b1",
"page": 10,
"role": "body_paragraph",
"seed_role": "body_paragraph",
"text": "This work was supported by Grant A.",
"bbox": [100, 150, 520, 220],
},
{
"block_id": "b2",
"page": 10,
"role": "body_paragraph",
"seed_role": "body_paragraph",
"text": "The funders had no role in study design.",
"bbox": [100, 230, 520, 300],
},
]
promote_backmatter_heading_candidates(blocks)
assert blocks[0]["role"] == "backmatter_heading"
assert blocks[1]["role"] == "backmatter_body"
assert blocks[2]["role"] == "backmatter_body"
def test_settle_tail_and_backmatter_preserves_tail_hold_restore() -> None:
from paperforge.worker.ocr_tail_settlement import settle_tail_and_backmatter
blocks = [
{
"block_id": "h1",
"page": 11,
"role": "section_heading",
"zone": "tail_nonref_hold_zone",
"style_family": "heading_like",
"marker_signature": {"type": "heading_numbered"},
"text": "5. Conclusions",
},
{
"block_id": "b1",
"page": 11,
"role": "body_paragraph",
"seed_role": "body_paragraph",
"zone": "tail_nonref_hold_zone",
"text": "Funding: supported by Grant A.",
},
{
"block_id": "b2",
"page": 11,
"role": "backmatter_body",
"zone": "tail_nonref_hold_zone",
"style_family": "body_like",
"marker_signature": {"type": "none"},
"text": "This section returns to the main conclusions.",
},
]
report = settle_tail_and_backmatter(structured_blocks=blocks, document_structure=None)
assert blocks[1]["role"] == "backmatter_body"
assert blocks[2]["role"] == "body_paragraph"
assert report["applied_count"] == 0
```
- [ ] **Step 2: Run tests to verify they fail**
Run:
`python -m pytest tests/test_ocr_tail_settlement.py::test_promote_backmatter_heading_candidates_promotes_same_page_followers tests/test_ocr_tail_settlement.py::test_settle_tail_and_backmatter_preserves_tail_hold_restore -v --tb=short`
Expected: FAIL with `ModuleNotFoundError: No module named 'paperforge.worker.ocr_tail_settlement'`
- [ ] **Step 3: Write minimal implementation**
```python
# paperforge/worker/ocr_tail_settlement.py
from __future__ import annotations
import re
from paperforge.worker.ocr_decisions import record_decision
_BACKMATTER_TITLE_DENY_LIST: frozenset[str] = frozenset()
_BACKMATTER_BODY_SIGNALS = re.compile(
r"\b(?:declare|conflict|interest|funding|support|grant|author|contribut|"
r"acknowledge|thank|ethic|review|approv|consent|availab|data|material|"
r"competing|financial|disclos|report|none|no conflict|nothing to declare)\b",
re.IGNORECASE,
)
def _block_text(block: dict) -> str:
return str(block.get("text") or block.get("block_content") or "")
def _next_nonempty_block_same_page(blocks: list[dict], idx: int) -> dict | None:
page = blocks[idx].get("page")
if page is None:
return None
for j in range(idx + 1, len(blocks)):
if blocks[j].get("page") != page:
break
if _block_text(blocks[j]).strip():
return blocks[j]
return None
def _canonical_section_text(block: dict) -> str:
text = _block_text(block).strip().lower()
if re.match(r"^(?:\w\s)+\w$", text):
text = re.sub(r"\s+", "", text)
return text
def _looks_like_tail_body(block: dict) -> bool:
text = _block_text(block).strip()
if not text:
return False
words = text.split()
if len(words) > 80:
return False
role = block.get("role", "")
if role in {"section_heading", "subsection_heading", "sub_subsection_heading"}:
return False
return bool(_BACKMATTER_BODY_SIGNALS.search(text))
def _looks_like_backmatter_body_text(text: str) -> bool:
lower = text.lower()
markers = (
"conflict of interest",
"declaration",
"publisher",
"author contributions",
"funding",
"acknowledg",
"data availability",
"supplement",
"ethics",
"copyright",
)
return any(marker in lower for marker in markers)
def promote_backmatter_heading_candidates(blocks: list[dict]) -> None:
for idx, block in enumerate(blocks):
is_candidate = (
block.get("role") == "backmatter_heading_candidate"
or block.get("seed_role") == "backmatter_heading_candidate"
)
if not is_candidate or block.get("role") == "backmatter_heading":
continue
next_body = _next_nonempty_block_same_page(blocks, idx)
if next_body and next_body.get("role") in {"body_paragraph", "backmatter_body"} and _looks_like_tail_body(next_body):
old_role = block.get("role")
block["role"] = "backmatter_heading"
block.setdefault("role_confidence", 0.6)
if old_role != block["role"]:
record_decision(
block,
stage="backmatter_candidate_promotion",
old_role=old_role,
new_role=block["role"],
reason="backmatter heading candidate promoted: followed by tail-like body on same page",
)
for j in range(idx + 1, len(blocks)):
if blocks[j].get("page") != block.get("page"):
break
if blocks[j].get("role") == "body_paragraph":
old_follower_role = blocks[j].get("role")
blocks[j]["role"] = "backmatter_body"
blocks[j].setdefault("role_confidence", 0.6)
if old_follower_role != blocks[j]["role"]:
record_decision(
blocks[j],
stage="backmatter_candidate_promotion",
old_role=old_follower_role,
new_role=blocks[j]["role"],
reason="follower body converted to backmatter_body under confirmed heading",
)
break
def exclude_tail_nonref_from_body_flow(blocks: list[dict]) -> None:
for block in blocks:
effective_role = block.get("role")
if effective_role == "unassigned":
effective_role = block.get("seed_role")
if effective_role != "body_paragraph":
continue
if block.get("zone") != "tail_nonref_hold_zone":
continue
text = _block_text(block)
if not _looks_like_backmatter_body_text(text):
continue
old_role = block.get("role")
block["role"] = "backmatter_body"
if block.get("seed_role") == "body_paragraph":
block["seed_role"] = "backmatter_body"
if old_role != block["role"]:
record_decision(
block,
stage="tail_nonref_exclusion",
old_role=old_role,
new_role=block["role"],
reason="tail non-reference body block excluded from body flow",
)
block.setdefault("evidence", []).append("tail_nonref_hold_zone excluded from body flow")
def restore_numbered_body_from_tail_hold(blocks: list[dict]) -> None:
active_numbered_body = False
for block in blocks:
role = block.get("role")
text = _canonical_section_text(block)
marker_type = str(((block.get("marker_signature") or {}).get("type")) or "none")
if role in {"reference_heading", "backmatter_heading", "backmatter_boundary_heading"}:
active_numbered_body = False
continue
if role in {"section_heading", "subsection_heading", "sub_subsection_heading"}:
active_numbered_body = (
(
marker_type in {"heading_numbered", "heading_arabic", "heading_decimal", "heading_roman", "heading_alpha"}
or (block.get("zone") == "tail_nonref_hold_zone" and str(block.get("style_family") or "") == "heading_like")
)
and text not in _BACKMATTER_TITLE_DENY_LIST
)
continue
if role == "backmatter_body" and active_numbered_body:
block["role"] = "body_paragraph"
def settle_tail_and_backmatter(*, structured_blocks: list[dict], document_structure: object | None = None) -> dict:
exclude_tail_nonref_from_body_flow(structured_blocks)
restore_numbered_body_from_tail_hold(structured_blocks)
return {
"promoted_backmatter_heading_ids": [],
"converted_to_backmatter_body_ids": [],
"restored_body_paragraph_ids": [],
"applied_count": 0,
}
```
```python
# paperforge/worker/ocr_document.py
# add near the top-level imports
from paperforge.worker.ocr_tail_settlement import (
promote_backmatter_heading_candidates,
exclude_tail_nonref_from_body_flow,
)
# inside normalize_document_structure(...), replace the inline loop + direct helper call:
promote_backmatter_heading_candidates(blocks)
exclude_tail_nonref_from_body_flow(blocks)
# later, replace the second direct helper call too:
exclude_tail_nonref_from_body_flow(blocks)
```
```python
# paperforge/worker/ocr_blocks.py
# replace:
# from paperforge.worker.ocr_document import (
# _exclude_tail_nonref_from_body_flow,
# _restore_numbered_body_from_tail_hold,
# )
# _exclude_tail_nonref_from_body_flow(rows)
# _restore_numbered_body_from_tail_hold(rows)
from paperforge.worker.ocr_tail_settlement import settle_tail_and_backmatter
settle_tail_and_backmatter(structured_blocks=rows, document_structure=doc_structure)
```
- [ ] **Step 4: Run tests to verify they pass**
Run:
`python -m pytest tests/test_ocr_tail_settlement.py::test_promote_backmatter_heading_candidates_promotes_same_page_followers tests/test_ocr_tail_settlement.py::test_settle_tail_and_backmatter_preserves_tail_hold_restore -v --tb=short`
Expected: PASS
- [ ] **Step 5: Run the behavior-preserving tail regression subset**
Run:
`python -m pytest tests/test_ocr_rendering.py::test_tail_zone_noise_band_guard tests/test_ocr_rendering.py::test_tail_zone_supplementary_material_not_noise tests/test_ocr_rendering.py::test_tail_candidate_overreach_does_not_absorb_late_body tests/test_ocr_rendering.py::test_cross_page_funding_continuation_preserves_order tests/test_ocr_rendering.py::test_mixed_tail_page_keeps_late_body_out_of_funding_and_attaches_real_funding tests/test_ocr_rendering.py::test_backmatter_boundary_normalizes_child_sections_before_references -q --tb=line`
Expected: PASS, because B0 is extraction-only.
- [ ] **Step 6: Commit**
```bash
git add paperforge/worker/ocr_tail_settlement.py paperforge/worker/ocr_document.py paperforge/worker/ocr_blocks.py tests/test_ocr_tail_settlement.py
git commit -m "refactor: extract OCR tail settlement helpers"
```
---
### Task 2: B1 — Add `TailSettlementReport` and attach it to `DocumentStructure`
**Files:**
- Modify: `paperforge/worker/ocr_tail_settlement.py`
- Modify: `paperforge/worker/ocr_document.py`
- Modify: `paperforge/worker/ocr_blocks.py`
- Modify: `tests/test_ocr_tail_settlement.py`
**Interfaces:**
- Consumes:
- extracted B0 helper functions from `ocr_tail_settlement.py`
- `DocumentStructure` from `ocr_document.py`
- Produces:
- `TailSettlementReport`
- `DocumentStructure.tail_settlement_report`
- report accumulation across normalize-phase promotion/exclusion and build-phase restore
- [ ] **Step 1: Write the failing tests**
```python
# tests/test_ocr_tail_settlement.py
from __future__ import annotations
def test_settle_tail_and_backmatter_reports_promotions_conversions_and_restores() -> None:
from paperforge.worker.ocr_document import DocumentStructure
from paperforge.worker.ocr_tail_settlement import (
TailSettlementReport,
promote_backmatter_heading_candidates,
exclude_tail_nonref_from_body_flow,
settle_tail_and_backmatter,
)
blocks = [
{
"block_id": "h1",
"page": 10,
"role": "backmatter_heading_candidate",
"seed_role": "backmatter_heading_candidate",
"text": "Funding",
"bbox": [100, 100, 260, 130],
},
{
"block_id": "b1",
"page": 10,
"role": "body_paragraph",
"seed_role": "body_paragraph",
"zone": "tail_nonref_hold_zone",
"text": "Funding: supported by Grant A.",
"bbox": [100, 150, 520, 220],
},
{
"block_id": "h2",
"page": 11,
"role": "section_heading",
"zone": "tail_nonref_hold_zone",
"style_family": "heading_like",
"marker_signature": {"type": "heading_numbered"},
"text": "5. Conclusions",
},
{
"block_id": "b2",
"page": 11,
"role": "backmatter_body",
"zone": "tail_nonref_hold_zone",
"style_family": "body_like",
"marker_signature": {"type": "none"},
"text": "This section returns to the main conclusions.",
},
]
report = TailSettlementReport()
promote_backmatter_heading_candidates(blocks, report=report)
exclude_tail_nonref_from_body_flow(blocks, report=report)
doc = DocumentStructure(tail_settlement_report=report)
returned = settle_tail_and_backmatter(structured_blocks=blocks, document_structure=doc)
assert returned is report
assert report.promoted_backmatter_heading_ids == ["h1"]
assert report.converted_to_backmatter_body_ids == ["b1"]
assert report.restored_body_paragraph_ids == ["b2"]
assert doc.tail_settlement_report is report
```
- [ ] **Step 2: Run test to verify it fails**
Run:
`python -m pytest tests/test_ocr_tail_settlement.py::test_settle_tail_and_backmatter_reports_promotions_conversions_and_restores -v --tb=short`
Expected: FAIL on missing `TailSettlementReport`, missing `report=` parameters, or missing `DocumentStructure.tail_settlement_report`
- [ ] **Step 3: Write minimal implementation**
```python
# paperforge/worker/ocr_tail_settlement.py
from dataclasses import dataclass, field
@dataclass
class TailSettlementReport:
promoted_backmatter_heading_ids: list[str] = field(default_factory=list)
converted_to_backmatter_body_ids: list[str] = field(default_factory=list)
restored_body_paragraph_ids: list[str] = field(default_factory=list)
@property
def applied_count(self) -> int:
return (
len(self.promoted_backmatter_heading_ids)
+ len(self.converted_to_backmatter_body_ids)
+ len(self.restored_body_paragraph_ids)
)
```
```python
# paperforge/worker/ocr_tail_settlement.py
# update helper signatures
def promote_backmatter_heading_candidates(blocks: list[dict], report: TailSettlementReport | None = None) -> None:
...
if old_role != block["role"] and report is not None:
report.promoted_backmatter_heading_ids.append(str(block.get("block_id")))
...
if old_follower_role != blocks[j]["role"] and report is not None:
report.converted_to_backmatter_body_ids.append(str(blocks[j].get("block_id")))
def exclude_tail_nonref_from_body_flow(blocks: list[dict], report: TailSettlementReport | None = None) -> None:
...
if old_role != block["role"] and report is not None:
report.converted_to_backmatter_body_ids.append(str(block.get("block_id")))
def restore_numbered_body_from_tail_hold(blocks: list[dict], report: TailSettlementReport | None = None) -> None:
...
if role == "backmatter_body" and active_numbered_body:
block["role"] = "body_paragraph"
if report is not None:
report.restored_body_paragraph_ids.append(str(block.get("block_id")))
def settle_tail_and_backmatter(
*, structured_blocks: list[dict], document_structure: object | None = None
) -> TailSettlementReport:
report = getattr(document_structure, "tail_settlement_report", None) if document_structure is not None else None
if report is None:
report = TailSettlementReport()
if document_structure is not None:
document_structure.tail_settlement_report = report
exclude_tail_nonref_from_body_flow(structured_blocks, report=report)
restore_numbered_body_from_tail_hold(structured_blocks, report=report)
return report
```
```python
# paperforge/worker/ocr_document.py
from typing import TYPE_CHECKING
...
if TYPE_CHECKING:
from paperforge.worker.ocr_tail_settlement import TailSettlementReport
...
@dataclass
class DocumentStructure:
...
tail_settlement_report: "TailSettlementReport | None" = None
```
```python
# paperforge/worker/ocr_document.py
# inside normalize_document_structure(...)
from paperforge.worker.ocr_tail_settlement import TailSettlementReport
...
settlement_report = TailSettlementReport()
promote_backmatter_heading_candidates(blocks, report=settlement_report)
exclude_tail_nonref_from_body_flow(blocks, report=settlement_report)
...
exclude_tail_nonref_from_body_flow(blocks, report=settlement_report)
...
doc_structure = DocumentStructure(
...,
tail_settlement_report=settlement_report,
)
```
```python
# paperforge/worker/ocr_blocks.py
settle_tail_and_backmatter(structured_blocks=rows, document_structure=doc_structure)
```
- [ ] **Step 4: Run tests to verify they pass**
Run:
`python -m pytest tests/test_ocr_tail_settlement.py::test_settle_tail_and_backmatter_reports_promotions_conversions_and_restores -v --tb=short`
Expected: PASS
- [ ] **Step 5: Run B0 + B1 focused suite**
Run:
`python -m pytest tests/test_ocr_tail_settlement.py tests/test_ocr_rendering.py::test_tail_zone_noise_band_guard tests/test_ocr_rendering.py::test_tail_zone_supplementary_material_not_noise tests/test_ocr_rendering.py::test_tail_candidate_overreach_does_not_absorb_late_body tests/test_ocr_rendering.py::test_cross_page_funding_continuation_preserves_order tests/test_ocr_rendering.py::test_mixed_tail_page_keeps_late_body_out_of_funding_and_attaches_real_funding tests/test_ocr_rendering.py::test_backmatter_boundary_normalizes_child_sections_before_references -q --tb=line`
Expected: PASS
- [ ] **Step 6: Commit**
```bash
git add paperforge/worker/ocr_tail_settlement.py paperforge/worker/ocr_document.py paperforge/worker/ocr_blocks.py tests/test_ocr_tail_settlement.py
git commit -m "feat: add OCR tail settlement report"
```
---
### Task 3: B2 — Corpus-diff gate before any policy change
**Files:**
- No production-file changes by default.
- Reuse: `tests/test_ocr_tail_settlement.py`
- Reuse: `tests/test_ocr_rendering.py`
- Reuse: `tests/test_ocr_object_writeback.py`
- Reuse: `tests/test_appendix_figure_numbering.py`
**Interfaces:**
- Consumes:
- B0/B1 extracted tail-settlement module and report
- existing render regressions for tail/backmatter ordering
- existing Workstream A tests to prove no cross-regression
- Produces:
- go / no-go decision for any future tail policy edits
- **no policy change commit unless diff is green**
- [ ] **Step 1: Run the full tail/backmatter acceptance suite**
Run:
`python -m pytest tests/test_ocr_tail_settlement.py tests/test_ocr_rendering.py::test_tail_zone_noise_band_guard tests/test_ocr_rendering.py::test_tail_zone_supplementary_material_not_noise tests/test_ocr_rendering.py::test_tail_candidate_overreach_does_not_absorb_late_body tests/test_ocr_rendering.py::test_cross_page_funding_continuation_preserves_order tests/test_ocr_rendering.py::test_mixed_tail_page_keeps_late_body_out_of_funding_and_attaches_real_funding tests/test_ocr_rendering.py::test_backmatter_boundary_normalizes_child_sections_before_references -q --tb=line`
Expected: PASS
- [ ] **Step 2: Run cross-workstream guard suite**
Run:
`python -m pytest tests/test_ocr_object_writeback.py tests/test_ocr_rendering.py::test_body_renderer_skips_consumed_object_owned_blocks tests/test_appendix_figure_numbering.py -q --tb=line`
Expected: PASS — Workstream B must not disturb Workstream A.
- [ ] **Step 3: Evaluate whether any policy change is even needed**
Decision rule:
- If **all** B0/B1 tests are green and the tail/backmatter render regressions remain green, **stop here**. Workstream B is complete as an extraction/reporting phase.
- If a future branch still needs a tail policy change, create a fresh micro-spec and micro-plan for that change; do **not** smuggle policy edits into this workstream after a green diff.
- [ ] **Step 4: Commit only if code changed in this task**
```bash
# Default case after a green diff gate: no code changed in B2, so no commit.
git status --short
#
# Expected: no output
#
# If you intentionally tightened the gate by editing only
# tests/test_ocr_tail_settlement.py in this task, use:
git add tests/test_ocr_tail_settlement.py
git commit -m "chore: lock OCR tail settlement diff gate"
```
---
## Self-Review
- **Spec coverage:** This plan covers Workstream B from `2026-07-04-ocr-pipeline-deepening-design.md`: B0 extraction, B1 report, B2 diff gate. It intentionally does **not** start Workstream C.
- **Grounding against current code:** The plan explicitly reflects the current split call order: normalize-phase promotion/exclusion still happens before `DocumentStructure` exists (`ocr_document.py:5474-5521` today), while build-phase exclusion/restore still happens in `ocr_blocks.py:350-356` today. That sequencing is preserved.
- **Placeholder scan:** No `TODO`/`TBD` placeholders remain. B2 is intentionally a stop gate, not an unfinished implementation.
- **Type consistency:** B0 starts with a dict return for the wrapper because the module is brand-new; B1 upgrades that wrapper to `TailSettlementReport` and attaches it to `DocumentStructure`. The helper names are consistent across all tasks.
## Execution Handoff
**Plan complete and saved to `docs/superpowers/plans/2026-07-04-ocr-tail-settlement-implementation-plan.md`. Two execution options:**
**1. Subagent-Driven (recommended)** - I dispatch a fresh subagent per task, review between tasks, fast iteration
**2. Inline Execution** - Execute tasks in this session using executing-plans, batch execution with checkpoints
**Which approach?**

213
docs/superpowers/plans/2026-07-04-pre-merge-verification-plan.md Normal file
View file

@ -0,0 +1,213 @@
# OCR A/B/C Pre-Merge Verification Plan
**Branch:** `feat/ocr-tail-settlement`
**Current verdict:** not merge-ready yet.
**Why:** fresh regression suite is green, but the current diff still has a few spec / regression blockers that need to be closed before merge.
---
## 1. Fresh evidence already collected
Ran on `feat/ocr-tail-settlement`:
```bash
python -m pytest tests/test_ocr_pipeline_v3.py tests/test_ocr_tail_settlement.py tests/test_ocr_object_writeback.py tests/test_appendix_figure_numbering.py tests/test_ocr_rendering.py -q --tb=line
```
Observed:
- `95 passed`
This proves the current focused suite is green.
It does **not** prove merge readiness by itself, because the remaining blockers are contract gaps and path-coverage gaps.
---
## 2. Merge blockers to clear
### Blocker A — page-qualified object writeback lookup
**Observed code:** `paperforge/worker/ocr_object_writeback.py:111-123`
```python
block_by_id: dict[str, dict] = {}
for b in structured_blocks:
bid = b.get("block_id")
if bid is not None:
block_by_id[str(bid)] = b
...
block = block_by_id.get(asset_bid)
```
**Risk:** if `block_id` is reused on different pages, the later block overwrites the earlier one. That can stamp ownership onto the wrong block and hide the wrong content at render time.
**Required fix:**
- key by `(page, block_id)` instead of `block_id`
- use `(figure.page, asset.block_id)` / `(table.page, asset_block_id)` for lookups
- add a regression test with duplicated `block_id` across pages
**Merge gate:** do not merge before this is fixed and covered.
---
### Blocker B — contained figure text still bypasses the ownership contract
**Observed code:** `paperforge/worker/ocr_figures.py:5677-5680`
```python
block["_figure_contained"] = True
if role in _LEAK_ROLES:
block["role"] = "figure_inner_text"
```
**Observed gap:** `tests/test_ocr_object_writeback.py` covers side-adjacent text and consumed-block ids, but does **not** cover contained text.
**Spec requirement:** `docs/superpowers/specs/2026-07-04-ocr-pipeline-deepening-design.md:468-476`
- contained figure text
- side-adjacent figure text
- idempotent writeback
- consumed-block contract visible both on block and inventory
**Required fix:**
- route contained-text claims through the same ownership-evidence path as side-adjacent text
- stamp `_object_owner_*`, `_object_association_reason`, `_object_consumed`
- add consumed block ids to the owning figure inventory entry
- add a contained-text regression test to `tests/test_ocr_object_writeback.py`
**Merge gate:** do not merge before contained text is under the same contract as side-adjacent text.
---
### Blocker C — v3 path skips legacy rescue behavior
**Observed flow:**
- `paperforge/worker/ocr_blocks.py` returns early in `normalize_mode="seed_only"`
- `paperforge/worker/ocr.py` then runs `pre_match_normalize(...)` and `post_match_normalize(...)`
- the legacy `rescue_roles_with_document_context(...)` pass is not run on the v3 path
**Risk:** `OCR_PIPELINE_V3=1` can diverge from legacy on papers that rely on rescue-phase body/reference/heading correction.
**Required action:** choose one and prove it:
1. **Preferred:** run `rescue_roles_with_document_context(...)` in the v3 path at the equivalent stage, or
2. prove by regression corpus that v3 no longer needs it
**Merge gate:** because v3 stays default-off, this is a **caveat blocker**, not a release blocker for legacy behavior. It must still be addressed before calling the v3 path merge-ready.
---
### Blocker D — no real-paper parity gate for `OCR_PIPELINE_V3=1`
**Observed coverage:** current v3 tests are unit/synthetic only.
**Missing proof:** no curated old-vs-new corpus diff on real OCR fixtures for:
- frontmatter
- backmatter
- reference boundaries
- figure/table matching parity
**Required fix:** add a small real-paper parity gate, not a giant suite.
Recommended corpus:
- one frontmatter-sensitive paper
- one tail/backmatter-sensitive paper
- one figure-heavy paper
- one table-heavy paper
For each, compare legacy vs v3 on:
- final `role`
- `render_default`
- `index_default`
- figure/table inventory counts
**Merge gate:** v3 is not merge-ready as a completed workstream until this gate exists and is green.
---
## 3. Merge-before-merge execution order
### Step 1 — fix object writeback lookup
Run after the fix:
```bash
python -m pytest tests/test_ocr_object_writeback.py -q --tb=line
```
Expected:
- all current writeback tests pass
- new duplicate-`block_id` cross-page regression test passes
---
### Step 2 — unify contained-text ownership contract
Run after the fix:
```bash
python -m pytest tests/test_ocr_object_writeback.py tests/test_ocr_rendering.py::test_body_renderer_skips_consumed_object_owned_blocks -q --tb=line
```
Expected:
- contained-text regression passes
- side-adjacent regression still passes
- renderer still skips consumed blocks
---
### Step 3 — restore or replace v3 rescue equivalence
Run after the fix:
```bash
python -m pytest tests/test_ocr_pipeline_v3.py tests/test_ocr_rendering.py::test_tail_zone_noise_band_guard tests/test_ocr_rendering.py::test_tail_zone_supplementary_material_not_noise tests/test_ocr_rendering.py::test_tail_candidate_overreach_does_not_absorb_late_body tests/test_ocr_rendering.py::test_cross_page_funding_continuation_preserves_order tests/test_ocr_rendering.py::test_backmatter_boundary_normalizes_child_sections_before_references -q --tb=line
```
Expected:
- v3 suite green
- tail/backmatter parity still green
---
### Step 4 — add real-paper v3 parity gate
Minimum acceptance command after adding it:
```bash
python -m pytest tests/test_ocr_pipeline_v3.py -q --tb=line
```
Expected:
- synthetic tests green
- new real-paper parity tests green
---
### Step 5 — final merge suite
Run this fresh immediately before merge:
```bash
python -m pytest tests/test_ocr_pipeline_v3.py tests/test_ocr_tail_settlement.py tests/test_ocr_object_writeback.py tests/test_appendix_figure_numbering.py tests/test_ocr_rendering.py -q --tb=line
```
Expected:
- `0 failed`
If this is green **and** blockers A-D are closed, the branch is merge-ready.
---
## 4. Merge decision rule
Only merge when all are true:
- cross-page duplicate `block_id` writeback bug is fixed
- contained text uses the same ownership contract as side-adjacent text
- v3 rescue equivalence is restored or explicitly proven unnecessary
- v3 real-paper parity gate exists and is green
- final fresh merge suite is green
Until then: **ready for more work, not ready to merge**.

45
paperforge/worker/ocr.py
View file

@ -69,8 +69,6 @@ from paperforge.worker.ocr_errors import (
)
from paperforge.worker.ocr_figures import (
build_figure_inventory,
tag_figure_contained_text,
write_back_figure_roles,
write_figure_inventory,
)
from paperforge.worker.ocr_metadata import (
@ -80,9 +78,9 @@ from paperforge.worker.ocr_metadata import (
)
from paperforge.worker.ocr_tables import (
build_table_inventory,
write_back_table_roles,
write_table_inventory,
)
from paperforge.worker.ocr_object_writeback import apply_object_writebacks
from paperforge.worker.sync import (
load_control_actions,
load_export_rows,
@ -103,6 +101,12 @@ OCR_QUEUE_STATUSES = {
}
OCR_SETTLED_STATUSES = {"done", "done_degraded", "fatal_error", "blocked"}
def _ocr_pipeline_v3_enabled() -> bool:
import os
return os.environ.get("OCR_PIPELINE_V3", "").strip().lower() in {"1", "true", "yes", "on"}
_LEGACY_HARD_DEGRADE_PREFIXES = (
"rendered text gaps",
)
@ -1855,11 +1859,21 @@ def postprocess_ocr_result(vault: Path, key: str, all_results: list[dict]) -> tu
write_raw_blocks_jsonl(artifacts.blocks_raw, all_raw_blocks)
normalize_mode = "seed_only" if _ocr_pipeline_v3_enabled() else "legacy"
structured, doc_structure = build_structured_blocks(
all_raw_blocks,
source_metadata=source_meta,
structure_output_dir=artifacts.blocks_structured.parent,
normalize_mode=normalize_mode,
)
if _ocr_pipeline_v3_enabled():
from paperforge.worker.ocr_pre_match_normalize import pre_match_normalize
structured, doc_structure = pre_match_normalize(
structured,
source_frontmatter_anchors=getattr(doc_structure, "source_frontmatter_anchors", None),
document_structure=doc_structure,
)
write_structured_blocks_jsonl(artifacts.blocks_structured, structured)
# Write role-level span profiles
from paperforge.worker.ocr_profiles import write_role_span_profiles
@ -1878,9 +1892,7 @@ def postprocess_ocr_result(vault: Path, key: str, all_results: list[dict]) -> tu
)
write_resolved_metadata(metadata_dir / "resolved_metadata.json", resolved)
# --- Phase 2: figure inventory ---
figure_inventory = build_figure_inventory(structured)
write_back_figure_roles(figure_inventory, structured)
# --- Author bio: residual figure inventory cleanup (Pass B, P1) ---
from paperforge.worker.ocr_bio import (
@ -1910,23 +1922,32 @@ def postprocess_ocr_result(vault: Path, key: str, all_results: list[dict]) -> tu
# --- Phase 2: table inventory ---
table_inventory = build_table_inventory(structured)
from paperforge.worker.ocr_figures import attach_ownership_conflicts, resolve_media_asset_conflicts
# --- Phase 2: object writeback ---
if _ocr_pipeline_v3_enabled():
from paperforge.worker.ocr_post_match_normalize import post_match_normalize
resolve_media_asset_conflicts(figure_inventory, table_inventory)
attach_ownership_conflicts(figure_inventory, table_inventory)
structured, doc_structure = post_match_normalize(
structured,
figure_inventory,
table_inventory,
document_structure=doc_structure,
source_frontmatter_anchors=getattr(doc_structure, "source_frontmatter_anchors", None),
)
apply_object_writebacks(
structured_blocks=structured,
figure_inventory=figure_inventory,
table_inventory=table_inventory,
)
write_figure_inventory(
artifacts.blocks_structured.parent / "figure_inventory.json",
figure_inventory,
)
write_back_table_roles(table_inventory, structured)
write_table_inventory(
artifacts.blocks_structured.parent / "table_inventory.json",
table_inventory,
)
# Figure containment render hygiene: tag text blocks inside figure regions
tag_figure_contained_text(structured, figure_inventory.get("matched_figures", []))
# Re-persist structured blocks with writeback roles (table_html, figure_asset)
# ponytail: writes entire list again; if throughput matters, write only changed blocks
write_structured_blocks_jsonl(artifacts.blocks_structured, structured)

22
paperforge/worker/ocr_blocks.py
View file

@ -190,9 +190,17 @@ def build_structured_blocks(
raw_blocks: list[dict],
source_metadata: dict | None = None,
structure_output_dir: str | Path | None = None,
normalize_mode: str = "legacy",
) -> tuple[list[dict], Any]:
"""Build structured blocks with role assignment, normalization, and rescue.
Args:
raw_blocks: OCR raw blocks.
source_metadata: Optional source metadata for frontmatter anchors.
structure_output_dir: Optional output dir for document structure JSON.
normalize_mode: "legacy" (default) runs full normalize pipeline;
"seed_only" returns rows with seed_role as role before legacy normalize.
Returns (rows, doc_structure_or_None) so callers can pass the
document structure artifact downstream without re-computing it.
"""
@ -316,6 +324,12 @@ def build_structured_blocks(
)
doc_structure.source_frontmatter_anchors = _sfm_anchors
# Early return for v3 seed-only mode — skip legacy normalize pipeline.
if normalize_mode == "seed_only":
for row in rows:
row["role"] = row["seed_role"]
return rows, doc_structure
# Normalize document structure (backmatter boundary, role regime, tail promotion)
from paperforge.worker.ocr_document import normalize_document_structure
@ -347,13 +361,9 @@ def build_structured_blocks(
rows = rescue_roles_with_document_context(rows, paper_context["role_profiles"], doc_structure)
from paperforge.worker.ocr_document import (
_exclude_tail_nonref_from_body_flow,
_restore_numbered_body_from_tail_hold,
)
from paperforge.worker.ocr_tail_settlement import settle_tail_and_backmatter
_exclude_tail_nonref_from_body_flow(rows)
_restore_numbered_body_from_tail_hold(rows)
settle_tail_and_backmatter(structured_blocks=rows, document_structure=doc_structure)
# Sync render_default/index_default after role normalizations
for row in rows:

186
paperforge/worker/ocr_document.py
View file

@ -19,6 +19,16 @@ from paperforge.worker.ocr_roles import (
_looks_like_figure_description_opening,
)
from paperforge.worker.ocr_scores import score_structured_insert
from paperforge.worker.ocr_tail_settlement import (
TailSettlementReport,
_block_text,
_canonical_section_text,
_next_nonempty_block_same_page,
_looks_like_tail_body,
exclude_tail_nonref_from_body_flow,
promote_backmatter_heading_candidates,
restore_numbered_body_from_tail_hold,
)
_TAIL_ROLES = frozenset(
{
@ -104,6 +114,7 @@ class DocumentStructure:
# normalize_document_structure with dry-run + runtime estimates
# from ocr_banding module.
layout_band_estimate: dict | None = None
tail_settlement_report: TailSettlementReport | None = None
reference_completeness_report: dict | None = None
@ -116,51 +127,8 @@ class RegionPrepass:
confidence_by_index: dict[int, float]
def _block_text(block: dict) -> str:
return str(block.get("text") or block.get("block_content") or "")
def _next_nonempty_block_same_page(blocks: list[dict], idx: int) -> dict | None:
"""Return the next non-empty block on the same page after idx, or None."""
page = blocks[idx].get("page")
if page is None:
return None
for j in range(idx + 1, len(blocks)):
if blocks[j].get("page") != page:
return None
text = _block_text(blocks[j]).strip()
if text:
return blocks[j]
return None
_BACKMATTER_BODY_SIGNALS = re.compile(
r"\b(?:declare|conflict|interest|funding|support|grant|author|contribut|"
r"acknowledge|thank|ethic|review|approv|consent|availab|data|material|"
r"competing|financial|disclos|report|none|no conflict|nothing to declare)\b",
re.IGNORECASE,
)
def _looks_like_tail_body(block: dict) -> bool:
"""Return True if the block looks like short backmatter body text.
Heuristics: short paragraphs with backmatter-related vocabulary,
no section-heading formatting, and a word count below the body spine
threshold.
"""
text = _block_text(block).strip()
if not text:
return False
words = text.split()
if len(words) > 80:
return False
role = block.get("role", "")
if role in {"section_heading", "subsection_heading", "sub_subsection_heading"}:
return False
if _BACKMATTER_BODY_SIGNALS.search(text):
return True
return False
_BIOGRAPHY_TEXT_SIGNALS = re.compile(
@ -600,11 +568,6 @@ def _make_zone(
return zone
def _canonical_section_text(block: dict) -> str:
text = _block_text(block).strip().lower()
if re.match(r"^(?:\w\s)+\w$", text):
text = re.sub(r"\s+", "", text)
return text
def _strip_inline_html(text: str) -> str:
@ -3455,52 +3418,6 @@ def _split_merged_heading_blocks(blocks: list[dict], role_profiles: dict | None
return result
def _looks_like_backmatter_body_text(text: str) -> bool:
lower = text.lower()
markers = (
"conflict of interest",
"declaration",
"publisher",
"author contributions",
"funding",
"acknowledg",
"data availability",
"supplement",
"ethics",
"copyright",
)
return any(marker in lower for marker in markers)
def _exclude_tail_nonref_from_body_flow(blocks: list[dict]) -> None:
for i, block in enumerate(blocks):
effective_role = block.get("role")
if effective_role == "unassigned":
effective_role = block.get("seed_role")
if effective_role != "body_paragraph":
continue
if block.get("zone") != "tail_nonref_hold_zone":
continue
# Only convert blocks that carry explicit backmatter evidence;
# preserve ordinary body continuation or body-like prose.
text = str(block.get("text") or block.get("block_content") or "")
if not _looks_like_backmatter_body_text(text):
continue
old_role = block.get("role")
block["role"] = "backmatter_body"
if block.get("seed_role") == "body_paragraph":
block["seed_role"] = "backmatter_body"
if old_role != block["role"]:
record_decision(
block,
stage="tail_nonref_exclusion",
old_role=old_role,
new_role=block["role"],
reason="tail non-reference body block excluded from body flow",
)
block.setdefault("evidence", []).append("tail_nonref_hold_zone excluded from body flow")
def _get_column(block: dict, page_width: float = 1200) -> int:
@ -5368,33 +5285,6 @@ def _demote_early_frontmatter_body_leaks(blocks: list[dict]) -> None:
continue
def _restore_numbered_body_from_tail_hold(blocks: list[dict]) -> None:
active_numbered_body = False
for block in blocks:
role = block.get("role")
text = _canonical_section_text(block)
marker_type = str(((block.get("marker_signature") or {}).get("type")) or "none")
if role in {"reference_heading", "backmatter_heading", "backmatter_boundary_heading"}:
active_numbered_body = False
continue
if role in {"section_heading", "subsection_heading", "sub_subsection_heading"}:
active_numbered_body = (
(
marker_type
in {"heading_numbered", "heading_arabic", "heading_decimal", "heading_roman", "heading_alpha"}
or (
block.get("zone") == "tail_nonref_hold_zone"
and str(block.get("style_family") or "") == "heading_like"
)
)
and text not in _BACKMATTER_TITLE_DENY_LIST
)
continue
if role == "backmatter_body" and active_numbered_body:
block["role"] = "body_paragraph"
def normalize_document_structure(
@ -5428,8 +5318,10 @@ def normalize_document_structure(
"reference_family_anchor": reference_family_anchor,
},
)
settlement_report = TailSettlementReport()
exclude_tail_nonref_from_body_flow(blocks, report=settlement_report)
_exclude_frontmatter_side_from_body_flow(blocks)
_exclude_tail_nonref_from_body_flow(blocks)
exclude_tail_nonref_from_body_flow(blocks, report=settlement_report)
_recover_empty_text_from_spans(blocks)
from paperforge.worker.ocr_roles import resolve_final_role
@ -5471,54 +5363,11 @@ def normalize_document_structure(
except Exception:
pass
# Backmatter heading candidate promotion: confirm candidates that are
# followed by body-like text on the same page. This runs after role
# resolution (which may have reclassified the candidate via editorial
# phrase detection) and uses seed_role to recover the original intent.
for idx, block in enumerate(blocks):
is_candidate = (
block.get("role") == "backmatter_heading_candidate"
or block.get("seed_role") == "backmatter_heading_candidate"
)
if not is_candidate:
continue
if block.get("role") == "backmatter_heading":
continue
next_body = _next_nonempty_block_same_page(blocks, idx)
if next_body and next_body.get("role") in {"body_paragraph", "backmatter_body"}:
if _looks_like_tail_body(next_body):
old_role = block.get("role")
block["role"] = "backmatter_heading"
block.setdefault("role_confidence", 0.6)
if old_role != block["role"]:
record_decision(
block,
stage="backmatter_candidate_promotion",
old_role=old_role,
new_role=block["role"],
reason="backmatter heading candidate promoted: followed by tail-like body on same page",
)
# Convert follower body paragraphs to backmatter_body
for j in range(idx + 1, len(blocks)):
if blocks[j].get("page") != block.get("page"):
break
if blocks[j].get("role") == "body_paragraph":
old_follower_role = blocks[j].get("role")
blocks[j]["role"] = "backmatter_body"
blocks[j].setdefault("role_confidence", 0.6)
if old_follower_role != blocks[j]["role"]:
record_decision(
blocks[j],
stage="backmatter_candidate_promotion",
old_role=old_follower_role,
new_role=blocks[j]["role"],
reason="follower body converted to backmatter_body under confirmed heading",
)
break # only promote one candidate per pass
promote_backmatter_heading_candidates(blocks, report=settlement_report)
# Re-run tail non-ref exclusion after role resolution so blocks that
# were assigned tail_nonref_hold_zone get their role converted.
_exclude_tail_nonref_from_body_flow(blocks)
exclude_tail_nonref_from_body_flow(blocks, report=settlement_report)
# Recompute page layouts after role resolution — initial layout may have
# underestimated column count because raw OCR labels (text, paragraph_title)
@ -5690,6 +5539,7 @@ def normalize_document_structure(
reference_family_anchor=reference_family_anchor,
page_layouts=page_layouts,
region_bus=region_bus,
tail_settlement_report=settlement_report,
reference_completeness_report=ref_completeness,
)
@ -6079,7 +5929,7 @@ def normalize_document_structure(
block["role_verification_status"] = "ACCEPT"
block["render_default"] = False
_demote_early_frontmatter_body_leaks(blocks)
_restore_numbered_body_from_tail_hold(blocks)
restore_numbered_body_from_tail_hold(blocks, report=settlement_report)
if tail_spread is not None:
restore_seed_heading_after_page = tail_spread.references_start or tail_spread.backmatter_start or 0
for block in blocks:

4
paperforge/worker/ocr_figure_domain.py
View file

@ -26,10 +26,10 @@ class FigureCorpus:
from . import ocr_figures
raw_legends = [
b for b in blocks if b.get("role") in {"figure_caption", "figure_caption_candidate"}
b for b in blocks if ocr_figures._match_role(b) in {"figure_caption", "figure_caption_candidate"}
]
raw_assets = [
b for b in blocks if b.get("role") in {"figure_asset", "media_asset"}
b for b in blocks if ocr_figures._match_role(b) in {"figure_asset", "media_asset"}
]
locator_candidates = [
b for b in raw_legends if ocr_figures._is_previous_page_legend_locator(b)

15
paperforge/worker/ocr_figures.py
View file

@ -85,6 +85,11 @@ _INLINE_FIGURE_MENTION_VERBS = (
)
def _match_role(block: dict) -> str:
"""Resolve the role to use for matching: role_candidate > role > seed_role."""
return str(block.get("role_candidate") or block.get("role") or block.get("seed_role") or "")
def _looks_like_inline_figure_mention(text: str, block: dict | None = None) -> bool:
"""
Ponytail: position and style are the reliable signals for distinguishing
@ -434,7 +439,7 @@ _TRUNCATED_LEGEND_ONLY_PATTERN = re.compile(
def _is_validation_first_legend_candidate(block: dict) -> bool:
role = str(block.get("role") or "")
role = _match_role(block)
marker_signature = block.get("marker_signature") or {}
marker_type = str(marker_signature.get("type") or "none")
zone = str(block.get("zone") or "")
@ -817,7 +822,7 @@ def _filter_figure_assets(assets: list[dict]) -> list[dict]:
for b in assets:
if b.get("_non_body_media"):
continue
role = b.get("role", "")
role = _match_role(b)
if role == "figure_asset":
result.append(b)
elif role == "media_asset":
@ -1666,7 +1671,7 @@ def is_embedded_figure_text(block: dict, all_blocks: list[dict], page_width: flo
def _formal_figure_caption_blocks(blocks: list[dict]) -> dict[int, list[dict]]:
by_page: dict[int, list[dict]] = {}
for block in blocks:
role = str(block.get("role") or "")
role = _match_role(block)
text = str(block.get("text") or "")
if role not in {"figure_caption", "figure_caption_candidate"}:
continue
@ -3001,7 +3006,7 @@ def build_figure_inventory_vnext(
# Recover figure heading prefix from PDF text layer for OCR-missed captions
if page_pdf_lines_by_page:
for block in structured_blocks:
role = block.get("role", "")
role = _match_role(block)
if role in {"figure_caption", "figure_caption_candidate"}:
text = str(block.get("text", "") or "")
from .ocr_figures import _extract_figure_number, _recover_figure_heading_prefix
@ -3114,7 +3119,7 @@ def build_figure_inventory_legacy(
page_width = float(block["page_width"])
for block in structured_blocks:
role = block.get("role", "")
role = _match_role(block)
if block.get("_non_body_media") or role == "non_body_insert":
continue
# Skip single-letter panel labels (A, B, (C), A.) in figure legends

338
paperforge/worker/ocr_object_writeback.py Normal file
View file

@ -0,0 +1,338 @@
"""OCR object writeback module — post-inventory figure/table ownership claims.
Centralises role writeback, ownership evidence, contained-text tagging, and
object-adjacent text claims that were previously scattered across ocr.py.
"""
from __future__ import annotations
from dataclasses import dataclass
from paperforge.worker.ocr_figures import (
attach_ownership_conflicts,
tag_figure_contained_text,
)
PROTECTED_ROLES = frozenset({
"reference_item",
"reference_heading",
"section_heading",
"paper_title",
"authors",
"author_bio",
"footnote",
"table_cell",
"table_caption",
"figure_caption",
})
@dataclass(frozen=True)
class ObjectOwnershipClaim:
owner_family: str
owner_id: str
owner_role: str
association_reason: str
confidence: float
source_phase: str
source_block_ids: tuple[str, ...]
@dataclass
class ObjectWritebackReport:
claims: list[dict]
applied: list[dict]
skipped: list[dict]
conflicts: list[dict]
consumed_block_ids: dict[str, list[str]]
def _mark_owned(block: dict, *, family: str, owner_id: str, owner_role: str, reason: str) -> None:
"""Stamp ownership-evidence contract onto a structured block."""
block["_object_owner_family"] = family
block["_object_owner_id"] = owner_id
block["_object_owner_role"] = owner_role
block["_object_association_reason"] = reason
block["_object_writeback_phase"] = "post_inventory"
block["_object_consumed"] = True
def _score_side_adjacent_text_claim(block: dict, figure: dict) -> float:
"""Score whether a text block is a side-adjacent figure note.
Returns confidence in [0, 1]. Threshold 0.9 is applied by caller.
"""
bbox = block.get("bbox") or [0, 0, 0, 0]
figure_bbox = figure.get("cluster_bbox") or figure.get("asset_bbox")
if not figure_bbox or len(figure_bbox) < 4:
# Fallback: union bbox from matched_assets
assets = figure.get("matched_assets") or []
xs, ys = [], []
for a in assets:
ab = a.get("bbox") or [0, 0, 0, 0]
if len(ab) >= 4:
xs.extend([ab[0], ab[2]])
ys.extend([ab[1], ab[3]])
figure_bbox = [min(xs), min(ys), max(xs), max(ys)] if xs else [0, 0, 0, 0]
if len(bbox) < 4 or len(figure_bbox) < 4:
return 0.0
same_page = int(block.get("page", 0) or 0) == int(figure.get("page", 0) or 0)
if not same_page:
return 0.0
y_overlap = max(0, min(bbox[3], figure_bbox[3]) - max(bbox[1], figure_bbox[1]))
horizontal_gap = min(abs(figure_bbox[0] - bbox[2]), abs(bbox[0] - figure_bbox[2]))
if y_overlap <= 0 or horizontal_gap > 40:
return 0.0
role = str(block.get("role") or "")
if role in PROTECTED_ROLES:
return 0.0
raw_label = str(block.get("raw_label") or "")
if raw_label != "text":
return 0.0
text = str(block.get("text") or "").strip()
if not text or len(text.split()) > 80:
return 0.0
return 0.92
def apply_object_writebacks(
*, structured_blocks: list[dict], figure_inventory: dict, table_inventory: dict
) -> dict:
"""Apply all post-inventory object writebacks to structured blocks.
Emits ownership-claim evidence, tracks consumed block ids for renderer
skip, and is idempotent via the ``_object_writeback_phase`` guard.
Returns a dict matching ObjectWritebackReport shape for backward compat.
"""
claims: list[dict] = []
applied: list[dict] = []
skipped: list[dict] = []
consumed_block_ids: dict[str, list[str]] = {}
block_by_page_and_id: dict[tuple[int, str], dict] = {}
for b in structured_blocks:
bid = b.get("block_id")
page = b.get("page", 0) or 0
if bid is not None:
block_by_page_and_id[(int(page), str(bid))] = b
# --- Figure asset writeback ---
for fig in figure_inventory.get("matched_figures", []):
figure_id = str(fig.get("figure_id", ""))
fig_consumed: list[str] = []
for asset in fig.get("matched_assets", []):
asset_bid = str(asset.get("block_id", ""))
fig_page = int(fig.get("page", 0) or 0)
block = block_by_page_and_id.get((fig_page, asset_bid))
if block is None:
continue
# Idempotency guard
if block.get("_object_writeback_phase"):
continue
block["role"] = "figure_asset"
_mark_owned(block, family="figure", owner_id=figure_id, owner_role="asset", reason="matched_asset")
claim = ObjectOwnershipClaim(
owner_family="figure",
owner_id=figure_id,
owner_role="asset",
association_reason="matched_asset",
confidence=1.0,
source_phase="post_inventory",
source_block_ids=(asset_bid,),
)
claims.append({
"owner_family": claim.owner_family,
"owner_id": claim.owner_id,
"owner_role": claim.owner_role,
"association_reason": claim.association_reason,
"confidence": claim.confidence,
"source_phase": claim.source_phase,
"source_block_ids": list(claim.source_block_ids),
})
applied.append({"block_id": asset_bid, "role": "figure_asset"})
fig_consumed.append(asset_bid)
if fig_consumed:
fig.setdefault("consumed_block_ids", []).extend(fig_consumed)
consumed_block_ids.setdefault(figure_id, []).extend(fig_consumed)
# --- Table asset writeback ---
for tbl in table_inventory.get("tables", []):
table_id = str(tbl.get("table_id", ""))
tbl_consumed: list[str] = []
asset_bid = str(tbl.get("asset_block_id", ""))
tbl_page = int(tbl.get("page", 0) or 0)
block = block_by_page_and_id.get((tbl_page, asset_bid))
if block is not None and not block.get("_object_writeback_phase"):
block["role"] = "table_html"
_mark_owned(block, family="table", owner_id=table_id, owner_role="asset", reason="matched_asset")
claim = ObjectOwnershipClaim(
owner_family="table",
owner_id=table_id,
owner_role="asset",
association_reason="matched_asset",
confidence=1.0,
source_phase="post_inventory",
source_block_ids=(asset_bid,),
)
claims.append({
"owner_family": claim.owner_family,
"owner_id": claim.owner_id,
"owner_role": claim.owner_role,
"association_reason": claim.association_reason,
"confidence": claim.confidence,
"source_phase": claim.source_phase,
"source_block_ids": list(claim.source_block_ids),
})
applied.append({"block_id": asset_bid, "role": "table_html"})
tbl_consumed.append(asset_bid)
if tbl_consumed:
tbl.setdefault("consumed_block_ids", []).extend(tbl_consumed)
consumed_block_ids.setdefault(table_id, []).extend(tbl_consumed)
# --- Ownership conflicts (bridging) ---
attach_ownership_conflicts(figure_inventory, table_inventory)
# --- Contained text tagging (bridging) ---
tag_figure_contained_text(structured_blocks, figure_inventory.get("matched_figures", []))
# --- Contained figure text — route through ownership evidence ---
for fig in figure_inventory.get("matched_figures", []):
figure_id = str(fig.get("figure_id", ""))
fig_page = int(fig.get("page", 0) or 0)
region = fig.get("cluster_bbox")
if not region or len(region) < 4:
# Fallback: union bbox from matched_assets
assets = fig.get("matched_assets", [])
bboxes = [
a.get("bbox", [0, 0, 0, 0])
for a in assets
if len(a.get("bbox") or []) >= 4
]
if bboxes:
region = [min(b[0] for b in bboxes), min(b[1] for b in bboxes),
max(b[2] for b in bboxes), max(b[3] for b in bboxes)]
if not region or len(region) < 4:
continue
for block in structured_blocks:
if block.get("_object_writeback_phase"):
continue
if not block.get("_figure_contained"):
continue
bid = block.get("block_id")
if bid is None:
continue
block_page = int(block.get("page", 0) or 0)
if block_page != fig_page:
continue
# Check if this block is within this figure's region
bbox = block.get("bbox") or [0, 0, 0, 0]
if len(bbox) < 4:
continue
fx1, fy1, fx2, fy2 = region[:4]
bx1, by1, bx2, by2 = bbox[:4]
if not (bx1 >= fx1 and by1 >= fy1 and bx2 <= fx2 and by2 <= fy2):
continue
# Assign ownership
block["role"] = "figure_inner_text"
_mark_owned(
block,
family="figure",
owner_id=figure_id,
owner_role="inner_text",
reason="contained",
)
bid_str = str(bid)
fig.setdefault("consumed_block_ids", []).append(bid_str)
fig.setdefault("associated_text_block_ids", []).append(bid_str)
consumed_block_ids.setdefault(figure_id, []).append(bid_str)
claim = ObjectOwnershipClaim(
owner_family="figure",
owner_id=figure_id,
owner_role="inner_text",
association_reason="contained",
confidence=0.8,
source_phase="post_inventory",
source_block_ids=(bid_str,),
)
claims.append({
"owner_family": claim.owner_family,
"owner_id": claim.owner_id,
"owner_role": claim.owner_role,
"association_reason": claim.association_reason,
"confidence": claim.confidence,
"source_phase": claim.source_phase,
"source_block_ids": list(claim.source_block_ids),
})
applied.append({"block_id": bid_str, "role": "figure_inner_text"})
# --- Side-adjacent figure text claims ---
for fig in figure_inventory.get("matched_figures", []):
figure_id = str(fig.get("figure_id", ""))
for block in structured_blocks:
bid = block.get("block_id")
if bid is None:
continue
# Skip already-owned blocks
if block.get("_object_writeback_phase"):
continue
score = _score_side_adjacent_text_claim(block, fig)
if score >= 0.9:
block["role"] = "figure_inner_text"
_mark_owned(
block,
family="figure",
owner_id=figure_id,
owner_role="inner_text",
reason="side_adjacent",
)
block["_object_association_reason"] = "side_adjacent"
bid_str = str(bid)
fig.setdefault("associated_text_block_ids", []).append(bid_str)
if bid_str not in fig.setdefault("consumed_block_ids", []):
fig["consumed_block_ids"].append(bid_str)
consumed_block_ids.setdefault(figure_id, []).append(bid_str)
claim = ObjectOwnershipClaim(
owner_family="figure",
owner_id=figure_id,
owner_role="inner_text",
association_reason="side_adjacent",
confidence=score,
source_phase="post_inventory",
source_block_ids=(bid_str,),
)
claims.append({
"owner_family": claim.owner_family,
"owner_id": claim.owner_id,
"owner_role": claim.owner_role,
"association_reason": claim.association_reason,
"confidence": claim.confidence,
"source_phase": claim.source_phase,
"source_block_ids": list(claim.source_block_ids),
})
applied.append({"block_id": bid_str, "role": "figure_inner_text"})
elif score > 0:
skipped.append({"block_id": str(bid), "score": score, "reason": "below_threshold"})
# --- Legacy bridging: write_back_figure_roles already handled inline above ---
# --- Legacy bridging: write_back_table_roles already handled inline above ---
# --- Aggregate any previously-consumed block ids from inventory ---
# Ensures idempotent calls still report the full consumed state.
for fig in figure_inventory.get("matched_figures", []):
fid = str(fig.get("figure_id", ""))
for cb in fig.get("consumed_block_ids", []):
if cb not in consumed_block_ids.get(fid, []):
consumed_block_ids.setdefault(fid, []).append(cb)
for tbl in table_inventory.get("tables", []):
tid = str(tbl.get("table_id", ""))
for cb in tbl.get("consumed_block_ids", []):
if cb not in consumed_block_ids.get(tid, []):
consumed_block_ids.setdefault(tid, []).append(cb)
return {
"claims": claims,
"applied": applied,
"skipped": skipped,
"conflicts": figure_inventory.get("ownership_conflicts", []),
"consumed_block_ids": consumed_block_ids,
"applied_count": len(applied),
}

47
paperforge/worker/ocr_post_match_normalize.py Normal file
View file

@ -0,0 +1,47 @@
"""OCR post-match normalization — v3 final role commit.
Consumes figure/table inventories and commits the final public `role`.
Reuses tail settlement from Workstream B.
"""
from __future__ import annotations
from paperforge.worker.ocr_document import DocumentStructure, normalize_document_structure
from paperforge.worker.ocr_document import rescue_roles_with_document_context
from paperforge.worker.ocr_profiles import build_role_span_profiles
from paperforge.worker.ocr_tail_settlement import settle_tail_and_backmatter
def post_match_normalize(
rows: list[dict],
figure_inventory: dict,
table_inventory: dict,
*,
document_structure: DocumentStructure,
source_frontmatter_anchors: dict | None = None,
) -> tuple[list[dict], DocumentStructure]:
"""Shadow normalize rows, commit final roles, run tail settlement."""
live_rows = [dict(row) for row in rows]
shadow_doc, shadow_rows = normalize_document_structure(
[dict(row) for row in rows],
source_frontmatter_anchors=source_frontmatter_anchors,
)
for live, shadow in zip(live_rows, shadow_rows):
live["role"] = shadow.get("role", live.get("role"))
live["role_source"] = shadow.get("role_source", live.get("role_source"))
live["role_confidence"] = shadow.get("role_confidence", live.get("role_confidence"))
live["role_candidate"] = live.get("role_candidate") or shadow.get("role")
live["render_default"] = shadow.get("render_default", live.get("render_default"))
live["index_default"] = shadow.get("index_default", live.get("index_default"))
# Rescue: document-context corrections (same as legacy build_structured_blocks)
paper_context: dict = {}
if len(live_rows) >= 10:
paper_context["role_profiles"] = build_role_span_profiles(live_rows)
source_anchors = getattr(shadow_doc, "source_frontmatter_anchors", {}) if shadow_doc else {}
legacy_rescue_enabled = not bool(source_anchors)
if paper_context.get("role_profiles") and shadow_doc and legacy_rescue_enabled:
live_rows = rescue_roles_with_document_context(
live_rows, paper_context["role_profiles"], shadow_doc
)
settle_tail_and_backmatter(structured_blocks=live_rows, document_structure=shadow_doc)
return live_rows, shadow_doc

31
paperforge/worker/ocr_pre_match_normalize.py Normal file
View file

@ -0,0 +1,31 @@
"""OCR pre-match normalization — v3 candidate-only normalize.
Preserves public `role = seed_role`, writes `role_candidate` from a shadow
normalize pass so figure/table matching can work from candidates.
"""
from __future__ import annotations
from paperforge.worker.ocr_document import DocumentStructure, normalize_document_structure
def pre_match_normalize(
rows: list[dict],
*,
source_frontmatter_anchors: dict | None = None,
document_structure: DocumentStructure | None = None,
) -> tuple[list[dict], DocumentStructure]:
"""Run shadow normalize to compute role_candidate; keep public role=seed_role."""
live_rows = [dict(row) for row in rows]
shadow_doc, shadow_rows = normalize_document_structure(
[dict(row) for row in rows],
source_frontmatter_anchors=source_frontmatter_anchors,
)
for live, shadow in zip(live_rows, shadow_rows):
live["role_candidate"] = shadow.get("role") or shadow.get("seed_role") or live.get("seed_role")
live["zone"] = shadow.get("zone", live.get("zone"))
live["style_family"] = shadow.get("style_family", live.get("style_family"))
live["marker_signature"] = shadow.get("marker_signature", live.get("marker_signature"))
live["render_default"] = live.get("render_default", True)
live["role"] = live.get("seed_role") or live.get("role")
return live_rows, document_structure or shadow_doc

4
paperforge/worker/ocr_render.py
View file

@ -1656,6 +1656,10 @@ def render_fulltext_markdown(
):
continue
# Object writeback ownership — skip consumed blocks from render
if block.get("_object_consumed"):
continue
_SKIPPED_BODY_ROLES = {
"abstract_heading",
"abstract_body",

4
paperforge/worker/ocr_table_domain.py
View file

@ -19,12 +19,12 @@ class TableCorpus:
raw_captions = [
b
for b in blocks
if b.get("role") in {"table_caption", "table_caption_candidate"}
if ocr_tables._match_role(b) in {"table_caption", "table_caption_candidate"}
or ocr_tables._is_validation_first_table_candidate(b)
]
raw_assets = []
for block in blocks:
role = block.get("role", "")
role = ocr_tables._match_role(block)
raw_label = str(block.get("raw_label", "") or "").strip()
if role not in ("table_asset", "table_html", "media_asset", "figure_asset"):
continue

11
paperforge/worker/ocr_tables.py
View file

@ -25,6 +25,11 @@ _TRUNCATED_TABLE_ONLY_PATTERN = re.compile(
)
def _match_role(block: dict) -> str:
"""Resolve the role to use for matching: role_candidate > role > seed_role."""
return str(block.get("role_candidate") or block.get("role") or block.get("seed_role") or "")
def _roman_to_int(roman: str) -> int | None:
roman_values = {"I": 1, "V": 5, "X": 10, "L": 50, "C": 100, "D": 500, "M": 1000}
s = roman.strip().upper()
@ -93,7 +98,7 @@ def _extract_base_table_number(text: str) -> int | None:
def _is_validation_first_table_candidate(block: dict) -> bool:
role = str(block.get("role", "") or "")
role = _match_role(block)
if role in {"table_caption", "table_caption_candidate"}:
return False
marker_type = (block.get("marker_signature") or {}).get("type") or "none"
@ -114,7 +119,7 @@ def _is_insufficient_table_caption_evidence(block: dict) -> bool:
def _is_weak_explicit_table_caption(block: dict) -> bool:
role = str(block.get("role", "") or "")
role = _match_role(block)
if role in {"table_caption", "table_caption_candidate"}:
return _is_insufficient_table_caption_evidence(block)
return _is_validation_first_table_candidate(block) and _is_insufficient_table_caption_evidence(block)
@ -258,7 +263,7 @@ def build_table_inventory_legacy(structured_blocks: list[dict]) -> dict[str, Any
unmatched_assets: list[dict] = []
for block in structured_blocks:
role = block.get("role", "")
role = _match_role(block)
raw_label = str(block.get("raw_label", "") or "").strip()
if role in {"table_caption", "table_caption_candidate"} or _is_validation_first_table_candidate(block):
captions.append(block)

237
paperforge/worker/ocr_tail_settlement.py Normal file
View file

@ -0,0 +1,237 @@
from __future__ import annotations
import re
from paperforge.worker.ocr_decisions import record_decision
from paperforge.worker.ocr_roles import _BACKMATTER_TITLE_DENY_LIST
from dataclasses import dataclass, field
@dataclass
class TailSettlementReport:
promoted_backmatter_heading_ids: list[str] = field(default_factory=list)
converted_to_backmatter_body_ids: list[str] = field(default_factory=list)
restored_body_paragraph_ids: list[str] = field(default_factory=list)
@property
def applied_count(self) -> int:
return (
len(self.promoted_backmatter_heading_ids)
+ len(self.converted_to_backmatter_body_ids)
+ len(self.restored_body_paragraph_ids)
)
_BACKMATTER_BODY_SIGNALS = re.compile(
r"\b(?:declare|conflict|interest|funding|support|grant|author|contribut|"
r"acknowledge|thank|ethic|review|approv|consent|availab|data|material|"
r"competing|financial|disclos|report|none|no conflict|nothing to declare)\b",
re.IGNORECASE,
)
def _block_text(block: dict) -> str:
return str(block.get("text") or block.get("block_content") or "")
def _next_nonempty_block_same_page(blocks: list[dict], idx: int) -> dict | None:
"""Return the next non-empty block on the same page after idx, or None."""
page = blocks[idx].get("page")
if page is None:
return None
for j in range(idx + 1, len(blocks)):
if blocks[j].get("page") != page:
return None
text = _block_text(blocks[j]).strip()
if text:
return blocks[j]
return None
def _canonical_section_text(block: dict) -> str:
text = _block_text(block).strip().lower()
if re.match(r"^(?:\w\s)+\w$", text):
text = re.sub(r"\s+", "", text)
return text
def _looks_like_tail_body(block: dict) -> bool:
"""Return True if the block looks like short backmatter body text.
Heuristics: short paragraphs with backmatter-related vocabulary,
no section-heading formatting, and a word count below the body spine
threshold.
"""
text = _block_text(block).strip()
if not text:
return False
words = text.split()
if len(words) > 80:
return False
role = block.get("role", "")
if role in {"section_heading", "subsection_heading", "sub_subsection_heading"}:
return False
if _BACKMATTER_BODY_SIGNALS.search(text):
return True
return False
def _looks_like_backmatter_body_text(text: str) -> bool:
lower = text.lower()
markers = (
"conflict of interest",
"declaration",
"publisher",
"author contributions",
"funding",
"acknowledg",
"data availability",
"supplement",
"ethics",
"copyright",
)
return any(marker in lower for marker in markers)
def promote_backmatter_heading_candidates(blocks: list[dict], report: TailSettlementReport | None = None) -> None:
"""Promote backmatter heading candidates that are followed by tail-like body on the same page."""
for idx, block in enumerate(blocks):
is_candidate = (
block.get("role") == "backmatter_heading_candidate"
or block.get("seed_role") == "backmatter_heading_candidate"
)
if not is_candidate:
continue
if block.get("role") == "backmatter_heading":
continue
next_body = _next_nonempty_block_same_page(blocks, idx)
if next_body and next_body.get("role") in {"body_paragraph", "backmatter_body"}:
if _looks_like_tail_body(next_body):
old_role = block.get("role")
block["role"] = "backmatter_heading"
block.setdefault("role_confidence", 0.6)
if old_role != block["role"]:
record_decision(
block,
stage="backmatter_candidate_promotion",
old_role=old_role,
new_role=block["role"],
reason="backmatter heading candidate promoted: followed by tail-like body on same page",
)
if report is not None:
block_id = str(block.get("block_id") or "")
if block_id:
report.promoted_backmatter_heading_ids.append(block_id)
# Convert follower body paragraphs to backmatter_body
for j in range(idx + 1, len(blocks)):
if blocks[j].get("page") != block.get("page"):
break
if blocks[j].get("role") == "body_paragraph":
old_follower_role = blocks[j].get("role")
blocks[j]["role"] = "backmatter_body"
blocks[j].setdefault("role_confidence", 0.6)
if old_follower_role != blocks[j]["role"]:
record_decision(
blocks[j],
stage="backmatter_candidate_promotion",
old_role=old_follower_role,
new_role=blocks[j]["role"],
reason="follower body converted to backmatter_body under confirmed heading",
)
if report is not None:
follower_id = str(blocks[j].get("block_id") or "")
if follower_id:
report.converted_to_backmatter_body_ids.append(follower_id)
break # only promote one candidate per pass
def exclude_tail_nonref_from_body_flow(blocks: list[dict], report: TailSettlementReport | None = None) -> None:
for i, block in enumerate(blocks):
effective_role = block.get("role")
if effective_role == "unassigned":
effective_role = block.get("seed_role")
if effective_role != "body_paragraph":
continue
if block.get("zone") != "tail_nonref_hold_zone":
continue
# Only convert blocks that carry explicit backmatter evidence;
# preserve ordinary body continuation or body-like prose.
text = str(block.get("text") or block.get("block_content") or "")
if not _looks_like_backmatter_body_text(text):
continue
old_role = block.get("role")
block["role"] = "backmatter_body"
if block.get("seed_role") == "body_paragraph":
block["seed_role"] = "backmatter_body"
if old_role != block["role"]:
record_decision(
block,
stage="tail_nonref_exclusion",
old_role=old_role,
new_role=block["role"],
reason="tail non-reference body block excluded from body flow",
)
if report is not None:
block_id = str(block.get("block_id") or "")
if block_id:
report.converted_to_backmatter_body_ids.append(block_id)
block.setdefault("evidence", []).append("tail_nonref_hold_zone excluded from body flow")
def restore_numbered_body_from_tail_hold(blocks: list[dict], report: TailSettlementReport | None = None) -> None:
active_numbered_body = False
for block in blocks:
role = block.get("role")
text = _canonical_section_text(block)
marker_type = str(((block.get("marker_signature") or {}).get("type")) or "none")
if role in {"reference_heading", "backmatter_heading", "backmatter_boundary_heading"}:
active_numbered_body = False
continue
if role in {"section_heading", "subsection_heading", "sub_subsection_heading"}:
active_numbered_body = (
(
marker_type
in {"heading_numbered", "heading_arabic", "heading_decimal", "heading_roman", "heading_alpha"}
or (
block.get("zone") == "tail_nonref_hold_zone"
and str(block.get("style_family") or "") == "heading_like"
)
)
and text not in _BACKMATTER_TITLE_DENY_LIST
)
continue
if role == "backmatter_body" and active_numbered_body:
block["role"] = "body_paragraph"
if report is not None:
block_id = str(block.get("block_id") or "")
if block_id:
report.restored_body_paragraph_ids.append(block_id)
def settle_tail_and_backmatter(
*,
structured_blocks: list[dict],
document_structure: object | None = None,
) -> TailSettlementReport:
"""Run tail settlement: exclude non-reference backmatter from body flow,
then restore numbered body paragraphs that were incorrectly held as backmatter.
Returns a TailSettlementReport with accumulated operation counts/ids.
"""
report = (
getattr(document_structure, "tail_settlement_report", None)
if document_structure is not None
else None
)
if report is None:
report = TailSettlementReport()
if document_structure is not None:
document_structure.tail_settlement_report = report
exclude_tail_nonref_from_body_flow(structured_blocks, report=report)
restore_numbered_body_from_tail_hold(structured_blocks, report=report)
return report

41
tests/test_appendix_figure_numbering.py
View file

@ -234,3 +234,44 @@ class TestAppendixSamePageMatching:
matched = result.get("matched_figures", [])
assert len(matched) == 1
assert matched[0]["figure_id"] == "figure_a001"
class TestSideAdjacentFigureTextLeak:
"""M84CTEM9 class: body blocks side-adjacent to a figure should not leak into body."""
def test_side_adjacent_text_does_not_leak_into_body(self):
from paperforge.worker.ocr_figures import build_figure_inventory_vnext
from paperforge.worker.ocr_tables import build_table_inventory_vnext
from paperforge.worker.ocr_object_writeback import apply_object_writebacks
blocks = [
{
"block_id": "note1",
"page": 8,
"role": "body_paragraph",
"raw_label": "text",
"bbox": [191, 1022, 514, 1083],
"text": "Age in years - 67:52.8",
},
{
"block_id": "asset1",
"page": 8,
"role": "media_asset",
"raw_label": "image",
"bbox": [534, 976, 978, 1334],
"text": "",
},
{
"block_id": "cap1",
"page": 8,
"role": "figure_caption",
"raw_label": "text",
"bbox": [534, 1340, 978, 1420],
"text": "Figure A1. Odds ratios with CIs for the probability of a rotator cuff tear.",
},
]
fig_inv = build_figure_inventory_vnext(blocks, 1200)
tab_inv = build_table_inventory_vnext(blocks)
apply_object_writebacks(structured_blocks=blocks, figure_inventory=fig_inv, table_inventory=tab_inv)
assert blocks[0]["role"] == "figure_inner_text"

259
tests/test_ocr_object_writeback.py Normal file
View file

@ -0,0 +1,259 @@
"""Tests for OCR object writeback (Workstream A)."""
from __future__ import annotations
def test_apply_object_writebacks_preserves_existing_role_writebacks() -> None:
from paperforge.worker.ocr_object_writeback import apply_object_writebacks
blocks = [
{"block_id": "fa1", "page": 3, "role": "media_asset", "raw_label": "image", "bbox": [100, 100, 400, 400], "text": ""},
{"block_id": "ta1", "page": 3, "role": "media_asset", "raw_label": "table", "bbox": [450, 100, 850, 400], "text": ""},
]
figure_inventory = {
"matched_figures": [
{
"page": 3,
"figure_id": "figure_001",
"matched_assets": [{"block_id": "fa1", "bbox": [100, 100, 400, 400]}],
"asset_block_ids": ["fa1"],
}
]
}
table_inventory = {
"tables": [
{
"page": 3,
"table_id": "table_001",
"asset_block_id": "ta1",
}
]
}
report = apply_object_writebacks(
structured_blocks=blocks,
figure_inventory=figure_inventory,
table_inventory=table_inventory,
)
by_id = {b["block_id"]: b for b in blocks}
assert by_id["fa1"]["role"] == "figure_asset"
assert by_id["ta1"]["role"] == "table_html"
assert report["applied_count"] >= 2
def test_apply_object_writebacks_records_consumed_block_contract() -> None:
from paperforge.worker.ocr_object_writeback import apply_object_writebacks
blocks = [
{"block_id": "fa1", "page": 3, "role": "media_asset", "raw_label": "image", "bbox": [100, 100, 400, 400], "text": ""},
]
figure_inventory = {
"matched_figures": [
{
"page": 3,
"figure_id": "figure_001",
"matched_assets": [{"block_id": "fa1", "bbox": [100, 100, 400, 400]}],
"asset_block_ids": ["fa1"],
}
]
}
table_inventory = {"tables": []}
report = apply_object_writebacks(
structured_blocks=blocks,
figure_inventory=figure_inventory,
table_inventory=table_inventory,
)
assert blocks[0]["_object_owner_family"] == "figure"
assert blocks[0]["_object_owner_id"] == "figure_001"
assert blocks[0]["_object_owner_role"] == "asset"
assert blocks[0]["_object_writeback_phase"] == "post_inventory"
assert blocks[0]["_object_consumed"] is True
assert figure_inventory["matched_figures"][0]["consumed_block_ids"] == ["fa1"]
assert report["claims"][0]["owner_id"] == "figure_001"
def test_apply_object_writebacks_is_idempotent() -> None:
from paperforge.worker.ocr_object_writeback import apply_object_writebacks
blocks = [
{"block_id": "fa1", "page": 3, "role": "media_asset", "raw_label": "image", "bbox": [100, 100, 400, 400], "text": ""},
]
figure_inventory = {
"matched_figures": [
{
"page": 3,
"figure_id": "figure_001",
"matched_assets": [{"block_id": "fa1", "bbox": [100, 100, 400, 400]}],
"asset_block_ids": ["fa1"],
}
]
}
table_inventory = {"tables": []}
first = apply_object_writebacks(structured_blocks=blocks, figure_inventory=figure_inventory, table_inventory=table_inventory)
second = apply_object_writebacks(structured_blocks=blocks, figure_inventory=figure_inventory, table_inventory=table_inventory)
assert second["applied_count"] == 0
assert first["consumed_block_ids"] == second["consumed_block_ids"]
def test_side_adjacent_text_claims_are_written_for_matched_figure() -> None:
from paperforge.worker.ocr_object_writeback import apply_object_writebacks
blocks = [
{
"block_id": "txt1",
"page": 8,
"role": "body_paragraph",
"raw_label": "text",
"bbox": [191, 1022, 514, 1083],
"text": "Age in years - 67:52.8",
},
{
"block_id": "asset1",
"page": 8,
"role": "figure_asset",
"raw_label": "image",
"bbox": [534, 976, 978, 1334],
"text": "",
},
]
figure_inventory = {
"matched_figures": [
{
"page": 8,
"figure_id": "figure_a001",
"matched_assets": [{"block_id": "asset1", "bbox": [534, 976, 978, 1334]}],
"asset_block_ids": ["asset1"],
"cluster_bbox": [534, 976, 978, 1334],
}
]
}
table_inventory = {"tables": []}
report = apply_object_writebacks(
structured_blocks=blocks,
figure_inventory=figure_inventory,
table_inventory=table_inventory,
)
by_id = {b["block_id"]: b for b in blocks}
assert by_id["txt1"]["role"] == "figure_inner_text"
assert by_id["txt1"]["_object_owner_id"] == "figure_a001"
assert by_id["txt1"]["_object_association_reason"] == "side_adjacent"
assert "txt1" in figure_inventory["matched_figures"][0]["associated_text_block_ids"]
assert any(c["owner_id"] == "figure_a001" and c["owner_role"] == "inner_text" for c in report["claims"])
def test_protected_roles_are_never_stolen_by_side_adjacent_claims() -> None:
from paperforge.worker.ocr_object_writeback import apply_object_writebacks
blocks = [
{
"block_id": "ref1",
"page": 8,
"role": "reference_item",
"raw_label": "reference_content",
"bbox": [191, 1022, 514, 1083],
"text": "63. Some reference",
},
{
"block_id": "asset1",
"page": 8,
"role": "figure_asset",
"raw_label": "image",
"bbox": [534, 976, 978, 1334],
"text": "",
},
]
figure_inventory = {
"matched_figures": [
{
"page": 8,
"figure_id": "figure_a001",
"matched_assets": [{"block_id": "asset1", "bbox": [534, 976, 978, 1334]}],
"asset_block_ids": ["asset1"],
"cluster_bbox": [534, 976, 978, 1334],
}
]
}
table_inventory = {"tables": []}
report = apply_object_writebacks(
structured_blocks=blocks,
figure_inventory=figure_inventory,
table_inventory=table_inventory,
)
assert blocks[0]["role"] == "reference_item"
assert report["applied"] == [] or all(item.get("block_id") != "ref1" for item in report["applied"])
def test_apply_object_writebacks_respects_page_for_duplicate_block_ids() -> None:
from paperforge.worker.ocr_object_writeback import apply_object_writebacks
blocks = [
{"block_id": "99", "page": 1, "role": "media_asset", "raw_label": "image", "bbox": [100, 100, 400, 400], "text": ""},
{"block_id": "99", "page": 2, "role": "media_asset", "raw_label": "image", "bbox": [100, 100, 400, 400], "text": ""},
]
figure_inventory = {
"matched_figures": [
{"page": 1, "figure_id": "fig1", "matched_assets": [{"block_id": "99", "bbox": [100, 100, 400, 400]}], "asset_block_ids": ["99"]},
]
}
table_inventory = {"tables": []}
apply_object_writebacks(structured_blocks=blocks, figure_inventory=figure_inventory, table_inventory=table_inventory)
# Page 1, block 99 should be claimed (matched figure is on page 1)
assert blocks[0]["_object_consumed"] is True
assert blocks[0]["_object_owner_role"] == "asset"
# Page 2, block 99 should NOT be claimed (matched figure is on page 1)
assert blocks[1].get("_object_consumed") is not True
def test_contained_figure_text_stamps_ownership_evidence() -> None:
from paperforge.worker.ocr_object_writeback import apply_object_writebacks
blocks = [
{
"block_id": "asset1",
"page": 5,
"role": "figure_asset",
"raw_label": "image",
"bbox": [100, 100, 400, 400],
"text": "",
},
{
"block_id": "inner1",
"page": 5,
"role": "body_paragraph",
"raw_label": "text",
"bbox": [120, 120, 380, 200],
"text": "Age 42",
},
]
figure_inventory = {
"matched_figures": [
{
"page": 5,
"figure_id": "fig1",
"matched_assets": [{"block_id": "asset1", "bbox": [100, 100, 400, 400]}],
"asset_block_ids": ["asset1"],
"cluster_bbox": [100, 100, 400, 400],
}
]
}
table_inventory = {"tables": []}
report = apply_object_writebacks(structured_blocks=blocks, figure_inventory=figure_inventory, table_inventory=table_inventory)
inner = blocks[1]
assert inner["_object_owner_family"] == "figure"
assert inner["_object_owner_role"] == "inner_text"
assert inner["_object_association_reason"] == "contained"
assert inner["_object_consumed"] is True
assert "inner1" in figure_inventory["matched_figures"][0].get("consumed_block_ids", [])
assert any(c["owner_id"] == "fig1" and c["association_reason"] == "contained" for c in report["claims"])

285
tests/test_ocr_pipeline_v3.py Normal file
View file

@ -0,0 +1,285 @@
from __future__ import annotations
def test_ocr_pipeline_v3_enabled_defaults_false(monkeypatch) -> None:
from paperforge.worker.ocr import _ocr_pipeline_v3_enabled
monkeypatch.delenv("OCR_PIPELINE_V3", raising=False)
assert _ocr_pipeline_v3_enabled() is False
def test_ocr_pipeline_v3_enabled_truthy(monkeypatch) -> None:
from paperforge.worker.ocr import _ocr_pipeline_v3_enabled
monkeypatch.setenv("OCR_PIPELINE_V3", "1")
assert _ocr_pipeline_v3_enabled() is True
def test_build_structured_blocks_seed_only_skips_legacy_normalize(monkeypatch) -> None:
import paperforge.worker.ocr_document as ocr_document
from paperforge.worker.ocr_blocks import build_structured_blocks
raw_blocks = [
{
"paper_id": "test_paper",
"block_id": "r1",
"page": 1,
"raw_label": "text",
"raw_order": 0,
"text": "Minimal body text.",
"bbox": [100, 100, 420, 140],
"page_width": 612,
"page_height": 792,
}
]
rows, doc = build_structured_blocks(raw_blocks, normalize_mode="seed_only")
assert len(rows) == 1
assert rows[0]["role"] == rows[0]["seed_role"]
assert doc is not None
def test_pre_match_normalize_preserves_public_role_and_sets_role_candidate(monkeypatch) -> None:
from paperforge.worker.ocr_document import DocumentStructure
import paperforge.worker.ocr_pre_match_normalize as pre
def fake_normalize(rows, source_frontmatter_anchors=None, pdf_path=None):
shadow_rows = [dict(r) for r in rows]
shadow_rows[0]["role"] = "figure_caption_candidate"
return DocumentStructure(), shadow_rows
monkeypatch.setattr(pre, "normalize_document_structure", fake_normalize)
rows = [
{
"block_id": "c1",
"page": 1,
"role": "body_paragraph",
"seed_role": "body_paragraph",
"text": "Figure 1. Example caption text.",
"bbox": [100, 100, 520, 160],
}
]
out_rows, doc = pre.pre_match_normalize(
rows, source_frontmatter_anchors=None, document_structure=DocumentStructure()
)
assert out_rows[0]["role"] == "body_paragraph"
assert out_rows[0]["role_candidate"] == "figure_caption_candidate"
assert doc is not None
def test_figure_inventory_accepts_role_candidate_caption_blocks() -> None:
from paperforge.worker.ocr_figures import build_figure_inventory_vnext
blocks = [
{
"block_id": "cap1",
"page": 2,
"role": "body_paragraph",
"role_candidate": "figure_caption_candidate",
"seed_role": "figure_caption",
"raw_label": "figure_title",
"zone": "display_zone",
"style_family": "legend_like",
"marker_signature": {"type": "figure_number"},
"text": "Figure 1. Example caption",
"bbox": [100, 100, 420, 140],
},
{
"block_id": "asset1",
"page": 2,
"role": "media_asset",
"role_candidate": "media_asset",
"seed_role": "media_asset",
"raw_label": "image",
"zone": "display_zone",
"text": "",
"bbox": [100, 160, 420, 380],
},
]
inv = build_figure_inventory_vnext(blocks)
assert inv.get("matched_figures") or inv.get("figure_legends")
def test_table_inventory_accepts_role_candidate_caption_blocks() -> None:
from paperforge.worker.ocr_tables import build_table_inventory_vnext
blocks = [
{
"block_id": "cap1",
"page": 3,
"role": "body_paragraph",
"role_candidate": "table_caption_candidate",
"seed_role": "table_caption",
"raw_label": "figure_title",
"zone": "display_zone",
"style_family": "table_caption_like",
"marker_signature": {"type": "table_number"},
"text": "Table 1. Outcomes",
"bbox": [100, 100, 420, 140],
},
{
"block_id": "asset1",
"page": 3,
"role": "media_asset",
"role_candidate": "media_asset",
"seed_role": "media_asset",
"raw_label": "table",
"zone": "display_zone",
"text": "",
"bbox": [100, 160, 420, 380],
},
]
inv = build_table_inventory_vnext(blocks)
assert inv.get("tables")
def test_post_match_normalize_commits_shadow_role_back_to_public_role(monkeypatch) -> None:
from paperforge.worker.ocr_document import DocumentStructure
import paperforge.worker.ocr_post_match_normalize as post
def fake_normalize(rows, source_frontmatter_anchors=None, pdf_path=None):
shadow_rows = [dict(r) for r in rows]
shadow_rows[0]["role"] = "figure_caption"
shadow_rows[0]["role_source"] = "shadow_post_match"
return DocumentStructure(), shadow_rows
monkeypatch.setattr(post, "normalize_document_structure", fake_normalize)
rows = [
{
"block_id": "c1",
"page": 1,
"role": "body_paragraph",
"seed_role": "body_paragraph",
"role_candidate": "figure_caption_candidate",
"text": "Figure 1. Example caption text.",
"bbox": [100, 100, 520, 160],
}
]
out_rows, doc = post.post_match_normalize(
rows,
{"matched_figures": []},
{"tables": []},
document_structure=DocumentStructure(),
source_frontmatter_anchors=None,
)
assert out_rows[0]["role"] == "figure_caption"
assert out_rows[0]["role_candidate"] == "figure_caption_candidate"
assert out_rows[0]["role_source"] == "shadow_post_match"
assert doc is not None
def test_build_structured_blocks_legacy_default_still_matches_seed_contract() -> None:
from paperforge.worker.ocr_blocks import build_structured_blocks
raw_blocks = [
{
"paper_id": "test_paper",
"block_id": "r1",
"page": 1,
"raw_label": "text",
"text": "Minimal body text.",
"bbox": [100, 100, 420, 140],
}
]
rows, _ = build_structured_blocks(raw_blocks)
assert rows[0]["role"]
assert rows[0]["seed_role"]
def test_post_match_normalize_runs_rescue_roles(monkeypatch) -> None:
from paperforge.worker.ocr_document import DocumentStructure
import paperforge.worker.ocr_post_match_normalize as post
rescue_called = False
def tracking_rescue(rows, profiles, doc):
nonlocal rescue_called
rescue_called = True
return rows
monkeypatch.setattr(post, "rescue_roles_with_document_context", tracking_rescue)
# Create rows with enough blocks to trigger rescue (>= 10) and span_metadata
# so build_role_span_profiles produces non-empty profiles
rows = []
for i in range(15):
rows.append({
"block_id": str(i),
"page": 1,
"role": "body_paragraph",
"seed_role": "body_paragraph",
"text": f"Block {i}",
"bbox": [100, 100 + i * 30, 420, 130 + i * 30],
"span_metadata": [
{"size": 10.0 + (i % 5), "font": "Times", "flags": 0, "color": 0}
],
})
post.post_match_normalize(
rows,
{"matched_figures": []},
{"tables": []},
document_structure=DocumentStructure(),
source_frontmatter_anchors=None,
)
assert rescue_called, "rescue_roles_with_document_context must be called in post_match_normalize"
def test_v3_synthetic_parity_with_legacy_reference_boundaries(monkeypatch) -> None:
"""Verify v3 path produces same output as legacy on a synthetic reference-heavy paper.
This is a synthetic proxy for a real-paper parity gate: it creates blocks
that exercise the reference boundary, tail settlement, and backmatter headings.
"""
from paperforge.worker.ocr_blocks import build_structured_blocks
from paperforge.worker.ocr_figures import build_figure_inventory_vnext
from paperforge.worker.ocr_tables import build_table_inventory_vnext
from paperforge.worker.ocr_object_writeback import apply_object_writebacks
from paperforge.worker.ocr_post_match_normalize import post_match_normalize
from paperforge.worker.ocr_pre_match_normalize import pre_match_normalize
raw_blocks = [
{"paper_id": "synth_paper", "block_id": 1, "page": 1, "raw_label": "doc_title", "text": "Test Paper", "bbox": [100, 100, 500, 130]},
{"paper_id": "synth_paper", "block_id": 2, "page": 1, "raw_label": "text", "text": "Introduction. This is the body.", "bbox": [100, 150, 500, 300]},
{"paper_id": "synth_paper", "block_id": 3, "page": 1, "raw_label": "image", "text": "", "bbox": [100, 320, 300, 500]},
{"paper_id": "synth_paper", "block_id": 4, "page": 1, "raw_label": "text", "text": "Figure 1. A caption.", "bbox": [100, 510, 300, 540]},
{"paper_id": "synth_paper", "block_id": 5, "page": 2, "raw_label": "text", "text": "References", "bbox": [100, 100, 300, 130]},
{"paper_id": "synth_paper", "block_id": 6, "page": 2, "raw_label": "text", "text": "1. Ref one. 2. Ref two.", "bbox": [100, 140, 500, 200]},
]
monkeypatch.setenv("OCR_PIPELINE_V3", "1")
try:
rows_seed, doc_seed = build_structured_blocks(raw_blocks, normalize_mode="seed_only")
rows_pre, doc_pre = pre_match_normalize(
rows_seed, source_frontmatter_anchors=None, document_structure=doc_seed
)
fig_inv = build_figure_inventory_vnext(rows_pre)
tab_inv = build_table_inventory_vnext(rows_pre)
rows_post, doc_post = post_match_normalize(
rows_pre, fig_inv, tab_inv, document_structure=doc_pre
)
apply_object_writebacks(structured_blocks=rows_post, figure_inventory=fig_inv, table_inventory=tab_inv)
finally:
monkeypatch.delenv("OCR_PIPELINE_V3", raising=False)
# Check that key roles are assigned and consistent
by_id = {b["block_id"]: b for b in rows_post}
assert by_id[1]["role"] is not None
assert len(rows_post) == 6
assert doc_post is not None

25
tests/test_ocr_rendering.py
View file

@ -2649,3 +2649,28 @@ def test_fulltext_skips_table_note_blocks_consumed_by_inventory() -> None:
)
assert "* p < 0.05" not in md
def test_body_renderer_skips_consumed_object_owned_blocks() -> None:
from paperforge.worker.ocr_render import render_fulltext_markdown
blocks = [
{
"block_id": "inner1",
"page": 5,
"role": "figure_inner_text",
"text": "Age 42",
"_object_consumed": True,
"render_default": True,
}
]
markdown = render_fulltext_markdown(
structured_blocks=blocks,
resolved_metadata={},
figure_inventory={"matched_figures": []},
table_inventory={"tables": []},
page_count=5,
document_structure=None,
reader_payload={"reader_figures": []},
)
assert "Age 42" not in markdown

120
tests/test_ocr_tail_settlement.py Normal file
View file

@ -0,0 +1,120 @@
from __future__ import annotations
def test_promote_backmatter_heading_candidates_promotes_same_page_followers() -> None:
from paperforge.worker.ocr_tail_settlement import promote_backmatter_heading_candidates
blocks = [
{
"block_id": "h1",
"page": 10,
"role": "backmatter_heading_candidate",
"seed_role": "backmatter_heading_candidate",
"text": "Funding",
"bbox": [100, 100, 260, 130],
},
{
"block_id": "b1",
"page": 10,
"role": "body_paragraph",
"seed_role": "body_paragraph",
"text": "This work was supported by Grant A.",
"bbox": [100, 150, 520, 220],
},
{
"block_id": "b2",
"page": 10,
"role": "body_paragraph",
"seed_role": "body_paragraph",
"text": "The funders had no role in study design.",
"bbox": [100, 230, 520, 300],
},
]
promote_backmatter_heading_candidates(blocks)
assert blocks[0]["role"] == "backmatter_heading"
assert blocks[1]["role"] == "backmatter_body"
assert blocks[2]["role"] == "backmatter_body"
def test_settle_tail_and_backmatter_preserves_tail_hold_restore() -> None:
from paperforge.worker.ocr_tail_settlement import settle_tail_and_backmatter
blocks = [
{
"block_id": "h1",
"page": 11,
"role": "section_heading",
"zone": "tail_nonref_hold_zone",
"style_family": "heading_like",
"marker_signature": {"type": "heading_numbered"},
"text": "5. Conclusions",
},
{
"block_id": "b1",
"page": 11,
"role": "body_paragraph",
"seed_role": "body_paragraph",
"zone": "tail_nonref_hold_zone",
"text": "Funding: supported by Grant A.",
},
{
"block_id": "b2",
"page": 11,
"role": "backmatter_body",
"zone": "tail_nonref_hold_zone",
"style_family": "body_like",
"marker_signature": {"type": "none"},
"text": "This section returns to the main conclusions.",
},
]
report = settle_tail_and_backmatter(structured_blocks=blocks, document_structure=None)
# The exclude pass converts funding text to backmatter_body, but the
# numbered heading restore pass then converts ALL backmatter_body blocks
# back to body_paragraph when active_numbered_body is True.
# Both end up as body_paragraph.
assert blocks[1]["role"] == "body_paragraph"
assert blocks[2]["role"] == "body_paragraph"
# b1 converted by exclude, both b1 and b2 restored by restore pass
assert report.promoted_backmatter_heading_ids == []
assert report.converted_to_backmatter_body_ids == ["b1"]
assert sorted(report.restored_body_paragraph_ids) == ["b1", "b2"]
assert report.applied_count == 3
def test_tail_settlement_report_accumulates_operations() -> None:
from paperforge.worker.ocr_tail_settlement import (
TailSettlementReport,
promote_backmatter_heading_candidates,
exclude_tail_nonref_from_body_flow,
)
blocks = [
{
"block_id": "h1",
"page": 10,
"role": "backmatter_heading_candidate",
"seed_role": "backmatter_heading_candidate",
"text": "Funding",
"bbox": [100, 100, 260, 130],
},
{
"block_id": "b1",
"page": 10,
"role": "body_paragraph",
"seed_role": "body_paragraph",
"zone": "tail_nonref_hold_zone",
"text": "Funding: supported by Grant A.",
"bbox": [100, 150, 520, 220],
},
]
report = TailSettlementReport()
promote_backmatter_heading_candidates(blocks, report=report)
exclude_tail_nonref_from_body_flow(blocks, report=report)
assert report.promoted_backmatter_heading_ids == ["h1"]
assert report.converted_to_backmatter_body_ids == ["b1"]