feat(ocr): checkpoint layout robustness and figure vnext groundwork on master

This commit is contained in:
LLLin000 2026-07-03 17:48:50 +08:00
parent 3feabf1569
commit 239d44bfe4
33 changed files with 7866 additions and 24 deletions

View file

@ -48,9 +48,36 @@
## 1. 工作指示
### Codebase Memory 优先
### Codebase Memory MCP知识图谱
`codebase-memory-mcp` 知识图,结构查询优先用 `search_graph` / `trace_path`,不用 grep/glob。
`codebase-memory-mcp` 已索引整个项目25K+ nodes, 53K+ edges支持结构化代码查询。
**典型使用场景:**
| 工具 | 用法 | 示例 |
|------|------|------|
| `search_graph` | 自然语言搜代码 | `search_graph("layout profile column detection")` → 秒出相关函数 |
| `trace_path` | 调用链追踪 | `trace_path("build_figure_inventory", direction="both")` → callees + callers |
| `get_architecture` | 架构总览 + 热点函数 | `get_architecture(aspects=["hotspots"])` → fan-in 排名 |
| `get_code_snippet` | 精确取函数源码 | `get_code_snippet("_classify_page_layout")` → 完整代码 |
| `query_graph` | Cypher 自定义查询 | 查循环深度≥3 的函数、查递归函数等 |
| `detect_changes` | 改动影响分析 | `detect_changes(scope="ocr_document.py")` → 哪些测试受影响 |
| `manage_adr` | 记录架构决策 | `manage_adr(mode="update", ...)` → 跨 session 持久化 |
**搜索优先于文本 grep**
```python
# 不要grep / rg / "find references" 文本搜索 — 迷失在 6000+ 行文件里
# 应该:
search_graph("column aware figure heading") # 自然语言 → 相关函数
trace_path("_recover_figure_heading_prefix") # 调用链追踪
get_architecture(aspects=["hotspots"]) # 项目热点
```
**索引维护:**
项目已在索引中。代码有较大改动后手动重新索引fast 模式,跳过语义相似度):
```
index_repository(mode="fast", persistence=True)
```
### 测试命令

View file

@ -0,0 +1,923 @@
# Figure Pipeline VNext Phase 0 + Phase 1 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Create a clean legacy/vnext split for the figure pipeline, add the comparison harness and core vnext contracts, and ship a first working vnext slice that handles ownership-safe same-page primary matching without touching sidecar/bundle/locator logic.
**Architecture:** Keep `build_figure_inventory(...)` stable and legacy-backed while introducing `build_figure_inventory_vnext(...)` as a new orchestrator. Build vnext around four internal seams: immutable corpus facts, derived candidate index, ownership ledger, and pass reports/proposals. The first shippable slice ends at same-page primary matching plus diff tooling; all special fallbacks remain legacy-only.
**Tech Stack:** Python 3, pytest, existing OCR figure helpers in `paperforge/worker/ocr_figures.py`, dev scripts under `scripts/dev/`
## Global Constraints
- Legacy implementation is an immutable baseline. Do not retrofit `OwnershipLedger` into the legacy function.
- Phase 0 only performs mechanical extraction: `build_figure_inventory_legacy = old behavior`, `build_figure_inventory_vnext = new orchestrator shell`, `build_figure_inventory` keeps calling legacy until cutover.
- `ResourceRef` is the only valid ownership key; `block_id` alone is forbidden.
- Define `ClaimProposal` and `PassReport` schemas before implementing passes.
- Split facts and hypotheses: `FigureCorpus` stores immutable facts; `FigureCandidateIndex` stores derived candidates and hypotheses.
- Add a legacy-vnext comparison harness before implementing special fallbacks.
- Final arbitration priority is independent of migration order; disabled passes emit no proposals.
- Do not implement sidecar, bundle, locator, group-aware sequential, classic sequential, composite-parent settlement, or final cutover in this plan.
- Preserve the external interface `build_figure_inventory(structured_blocks, page_width=1200) -> FigureInventory`.
---
### Task 1: Mechanical legacy/vnext split and comparison harness
**Files:**
- Modify: `paperforge/worker/ocr_figures.py`
- Create: `scripts/dev/compare_figure_inventory_legacy_vs_vnext.py`
- Test: `tests/test_ocr_figures.py`
**Interfaces:**
- Consumes: existing `build_figure_inventory(structured_blocks: list[dict], page_width: float = 1200) -> dict[str, Any]`
- Produces:
- `build_figure_inventory_legacy(structured_blocks: list[dict], page_width: float = 1200) -> dict[str, Any]`
- `build_figure_inventory_vnext(structured_blocks: list[dict], page_width: float = 1200) -> dict[str, Any]`
- `compare_inventories(legacy: dict[str, Any], vnext: dict[str, Any]) -> dict[str, Any]`
- [ ] **Step 1: Write the failing wrapper contract tests**
```python
# tests/test_ocr_figures.py
def test_build_figure_inventory_wrapper_stays_legacy_path(monkeypatch):
from paperforge.worker import ocr_figures
called = {"legacy": 0, "vnext": 0}
def fake_legacy(blocks, page_width=1200):
called["legacy"] += 1
return {"source": "legacy"}
def fake_vnext(blocks, page_width=1200):
called["vnext"] += 1
return {"source": "vnext"}
monkeypatch.setattr(ocr_figures, "build_figure_inventory_legacy", fake_legacy)
monkeypatch.setattr(ocr_figures, "build_figure_inventory_vnext", fake_vnext)
result = ocr_figures.build_figure_inventory([], 1200)
assert result == {"source": "legacy"}
assert called == {"legacy": 1, "vnext": 0}
def test_vnext_entrypoint_is_callable_without_cutover():
from paperforge.worker.ocr_figures import build_figure_inventory_vnext
result = build_figure_inventory_vnext([], 1200)
assert isinstance(result, dict)
assert result.get("pipeline_mode") == "vnext"
assert result.get("matched_figures") == []
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `python -m pytest tests/test_ocr_figures.py::test_build_figure_inventory_wrapper_stays_legacy_path tests/test_ocr_figures.py::test_vnext_entrypoint_is_callable_without_cutover -v`
Expected: FAIL because `build_figure_inventory_legacy` and `build_figure_inventory_vnext` do not both exist yet.
- [ ] **Step 3: Rename the current function and add the stable wrapper**
```python
# paperforge/worker/ocr_figures.py
def build_figure_inventory(structured_blocks: list[dict], page_width: float = 1200) -> dict[str, Any]:
return build_figure_inventory_legacy(structured_blocks, page_width)
def build_figure_inventory_vnext(structured_blocks: list[dict], page_width: float = 1200) -> dict[str, Any]:
return {
"pipeline_mode": "vnext",
"matched_figures": [],
"ambiguous_figures": [],
"unmatched_legends": [],
"unmatched_assets": [],
"unresolved_clusters": [],
"held_figures": [],
"rejected_legends": [],
"page_ledger": {},
"residual_ledger": {},
"local_pairing_hypotheses": [],
"completeness": {
"total_numbered_legends": 0,
"accounted_for": 0,
"details": [],
},
}
def build_figure_inventory_legacy(structured_blocks: list[dict], page_width: float = 1200) -> dict[str, Any]:
# body is the exact former build_figure_inventory implementation, unchanged
...
```
- [ ] **Step 4: Add the comparison harness script**
```python
# scripts/dev/compare_figure_inventory_legacy_vs_vnext.py
from __future__ import annotations
import json
from pathlib import Path
from paperforge.worker.ocr_figures import (
build_figure_inventory_legacy,
build_figure_inventory_vnext,
)
def _figure_asset_ids(fig: dict[str, object]) -> list[str]:
ids = {str(x) for x in fig.get("asset_block_ids", [])}
ids.update(str(asset.get("block_id", "")) for asset in fig.get("matched_assets", []))
return sorted(x for x in ids if x)
def compare_inventories(legacy: dict[str, object], vnext: dict[str, object]) -> dict[str, object]:
return {
"legacy_matched_count": len(legacy.get("matched_figures", [])),
"vnext_matched_count": len(vnext.get("matched_figures", [])),
"legacy_unresolved_count": len(legacy.get("unresolved_clusters", [])),
"vnext_unresolved_count": len(vnext.get("unresolved_clusters", [])),
"legacy_unmatched_legend_count": len(legacy.get("unmatched_legends", [])),
"vnext_unmatched_legend_count": len(vnext.get("unmatched_legends", [])),
"legacy_consumed_block_ids": sorted(
{bid for fig in legacy.get("matched_figures", []) for bid in _figure_asset_ids(fig)}
),
"vnext_consumed_block_ids": sorted(
{bid for fig in vnext.get("matched_figures", []) for bid in _figure_asset_ids(fig)}
),
}
def main(blocks_path: str) -> int:
blocks = [json.loads(line) for line in Path(blocks_path).read_text(encoding="utf-8").splitlines() if line.strip()]
legacy = build_figure_inventory_legacy(blocks)
vnext = build_figure_inventory_vnext(blocks)
print(json.dumps(compare_inventories(legacy, vnext), ensure_ascii=False, indent=2))
return 0
if __name__ == "__main__":
import sys
raise SystemExit(main(sys.argv[1]))
```
- [ ] **Step 5: Run tests to verify they pass**
Run: `python -m pytest tests/test_ocr_figures.py::test_build_figure_inventory_wrapper_stays_legacy_path tests/test_ocr_figures.py::test_vnext_entrypoint_is_callable_without_cutover -v`
Expected: PASS
- [ ] **Step 6: Verify the wrapper is the only public call path**
Run: `python -c "import inspect; from paperforge.worker import ocr_figures; print(inspect.signature(ocr_figures.build_figure_inventory)); print(hasattr(ocr_figures, 'build_figure_inventory_legacy')); print(hasattr(ocr_figures, 'build_figure_inventory_vnext'))"`
Expected: the public wrapper still exists and both explicit internal entrypoints exist.
- [ ] **Step 7: Commit**
```bash
git add paperforge/worker/ocr_figures.py scripts/dev/compare_figure_inventory_legacy_vs_vnext.py tests/test_ocr_figures.py
git commit -m "refactor(ocr): split legacy and vnext figure entrypoints"
```
### Task 2: Add vnext core contracts and identity-safe ledger
**Files:**
- Create: `paperforge/worker/ocr_figure_vnext_types.py`
- Create: `paperforge/worker/ocr_figure_vnext_state.py`
- Test: `tests/test_ocr_figure_vnext_types.py`
- Test: `tests/test_ocr_figure_vnext_state.py`
**Interfaces:**
- Consumes: none
- Produces:
- `ResourceRef`
- `OwnershipConflict`
- `ClaimProposal`
- `PassReport`
- `OwnershipLedger`
- `FigurePipelineState.accept_match(...)`
- [ ] **Step 1: Write failing identity tests**
```python
# tests/test_ocr_figure_vnext_types.py
import pytest
from paperforge.worker.ocr_figure_vnext_types import ResourceRef
def test_resource_ref_rejects_page_agnostic_asset():
with pytest.raises(ValueError):
ResourceRef(kind="asset", page=None, block_id="42")
def test_resource_ref_normalizes_block_id_type():
assert ResourceRef(kind="asset", page=1, block_id=42) == ResourceRef(kind="asset", page=1, block_id="42")
def test_resource_ref_rejects_group_without_group_id():
with pytest.raises(ValueError):
ResourceRef(kind="group", page=1, block_id=None)
```
```python
# tests/test_ocr_figure_vnext_state.py
from paperforge.worker.ocr_figure_vnext_types import ClaimProposal, ResourceRef
from paperforge.worker.ocr_figure_vnext_state import FigurePipelineState, OwnershipLedger
def test_ledger_rejects_double_ownership_for_same_asset_and_records_conflict():
ledger = OwnershipLedger()
asset = ResourceRef(kind="asset", page=3, block_id="42")
legend_a = ResourceRef(kind="legend", page=3, block_id="77", figure_no=1)
legend_b = ResourceRef(kind="legend", page=3, block_id="78", figure_no=2)
ledger.claim_assets([asset], owner=legend_a, reason="same_page_primary")
conflict = ledger.try_claim_assets([asset], owner=legend_b, reason="conflicting_match")
assert conflict is not None
assert conflict.resource == asset
assert conflict.current_owner == legend_a
assert any(entry["action"] == "conflict" for entry in ledger.snapshot())
def test_pipeline_state_accept_match_records_diagnostic():
state = FigurePipelineState(corpus=None, candidate_index=None, ledger=OwnershipLedger())
proposal = ClaimProposal(
pass_name="primary_same_page",
figure_no=1,
claim_type="match",
legends=[ResourceRef(kind="legend", page=1, block_id="c1", figure_no=1)],
assets=[ResourceRef(kind="asset", page=1, block_id="a1")],
groups=[],
confidence=0.9,
evidence_rank=1,
reason="same_page_primary",
diagnostics={"evidence": ["test"]},
)
state.accept_match(proposal, {"figure_id": "Figure 1"})
assert state.matches == [{"figure_id": "Figure 1"}]
assert state.diagnostics[-1]["event"] == "match_accepted"
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `python -m pytest tests/test_ocr_figure_vnext_types.py tests/test_ocr_figure_vnext_state.py -v`
Expected: FAIL because the new types/state modules do not exist.
- [ ] **Step 3: Define the core schemas**
```python
# paperforge/worker/ocr_figure_vnext_types.py
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Literal
@dataclass(frozen=True)
class ResourceRef:
kind: Literal["legend", "asset", "group"]
page: int | None
block_id: str | None
group_id: str | None = None
figure_no: int | None = None
origin: str | None = None
def __post_init__(self) -> None:
if self.page is not None:
object.__setattr__(self, "page", int(self.page))
if self.block_id is not None:
object.__setattr__(self, "block_id", str(self.block_id))
if self.group_id is not None:
object.__setattr__(self, "group_id", str(self.group_id))
if self.kind == "asset" and (self.page is None or self.block_id is None):
raise ValueError("asset ResourceRef requires page + block_id")
if self.kind == "legend" and (self.page is None or self.block_id is None):
raise ValueError("legend ResourceRef requires page + block_id")
if self.kind == "group" and (self.page is None or self.group_id is None):
raise ValueError("group ResourceRef requires page + group_id")
@dataclass(frozen=True)
class OwnershipConflict:
resource: ResourceRef
current_owner: ResourceRef | None
attempted_owner: ResourceRef | None
reason: str
@dataclass
class ClaimProposal:
pass_name: str
figure_no: int | None
claim_type: Literal["match", "reserve", "block", "unresolved_cluster", "composite_parent"]
legends: list[ResourceRef]
assets: list[ResourceRef]
groups: list[ResourceRef]
confidence: float
evidence_rank: int
reason: str
diagnostics: dict[str, Any] = field(default_factory=dict)
@dataclass
class PassReport:
pass_name: str
proposals: list[ClaimProposal] = field(default_factory=list)
accepted: list[ClaimProposal] = field(default_factory=list)
rejected: list[ClaimProposal] = field(default_factory=list)
conflicts: list[OwnershipConflict] = field(default_factory=list)
invariant_errors: list[str] = field(default_factory=list)
```
- [ ] **Step 4: Implement the ledger and minimal pipeline state**
```python
# paperforge/worker/ocr_figure_vnext_state.py
from __future__ import annotations
from dataclasses import dataclass, field
from .ocr_figure_vnext_types import ClaimProposal, OwnershipConflict, ResourceRef
class OwnershipLedger:
def __init__(self) -> None:
self._owners: dict[ResourceRef, ResourceRef] = {}
self._journal: list[dict[str, object]] = []
def claim_assets(self, assets: list[ResourceRef], *, owner: ResourceRef, reason: str) -> None:
conflict = self.try_claim_assets(assets, owner=owner, reason=reason)
if conflict is not None:
raise ValueError(f"asset already owned: {conflict.resource}")
def try_claim_assets(self, assets: list[ResourceRef], *, owner: ResourceRef, reason: str) -> OwnershipConflict | None:
for asset in assets:
current = self._owners.get(asset)
if current is not None and current != owner:
conflict = OwnershipConflict(resource=asset, current_owner=current, attempted_owner=owner, reason=reason)
self._journal.append({
"action": "conflict",
"resource": asset,
"current_owner": current,
"attempted_owner": owner,
"reason": reason,
})
return conflict
for asset in assets:
self._owners[asset] = owner
self._journal.append({"action": "claim", "resource": asset, "owner": owner, "reason": reason})
return None
def owner_of(self, resource: ResourceRef) -> ResourceRef | None:
return self._owners.get(resource)
def owner_of_asset(self, *, page: int, block_id: int | str) -> ResourceRef | None:
return self.owner_of(ResourceRef(kind="asset", page=page, block_id=block_id))
def snapshot(self) -> list[dict[str, object]]:
return list(self._journal)
@dataclass
class FigurePipelineState:
corpus: object | None
candidate_index: object | None
ledger: OwnershipLedger
matches: list[dict] = field(default_factory=list)
unresolved: list[dict] = field(default_factory=list)
hypotheses: list[dict] = field(default_factory=list)
diagnostics: list[dict] = field(default_factory=list)
def accept_match(self, proposal: ClaimProposal, match_record: dict) -> None:
self.matches.append(match_record)
self.diagnostics.append({
"event": "match_accepted",
"pass_name": proposal.pass_name,
"figure_no": proposal.figure_no,
"reason": proposal.reason,
"resources": {
"legends": proposal.legends,
"assets": proposal.assets,
"groups": proposal.groups,
},
})
```
- [ ] **Step 5: Run tests to verify they pass**
Run: `python -m pytest tests/test_ocr_figure_vnext_types.py tests/test_ocr_figure_vnext_state.py -v`
Expected: PASS
- [ ] **Step 6: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_types.py paperforge/worker/ocr_figure_vnext_state.py tests/test_ocr_figure_vnext_types.py tests/test_ocr_figure_vnext_state.py
git commit -m "feat(ocr): add vnext figure ownership contracts"
```
### Task 3: Build immutable corpus facts and derived candidate index
**Files:**
- Create: `paperforge/worker/ocr_figure_vnext_corpus.py`
- Modify: `paperforge/worker/ocr_figures.py`
- Test: `tests/test_ocr_figure_vnext_corpus.py`
- Consumes:
- `build_figure_inventory_legacy(...)`
- existing helpers in `ocr_figures.py` such as `_is_formal_legend`, `_build_candidate_figure_groups_from_assets`
- lazy import from `ocr_document` for `_build_page_layout_profiles`
- Produces:
- `FigureCorpus.from_blocks(blocks: list[dict], page_width: float = 1200) -> FigureCorpus`
- `FigureCandidateIndex.from_corpus(corpus: FigureCorpus) -> FigureCandidateIndex`
- [ ] **Step 1: Write failing corpus/index tests**
```python
# tests/test_ocr_figure_vnext_corpus.py
from paperforge.worker.ocr_figure_vnext_corpus import FigureCorpus, FigureCandidateIndex
def test_corpus_keeps_raw_facts_and_no_candidate_groups():
blocks = [
{"block_id": "1", "page": 1, "role": "figure_caption", "text": "Figure 1. Caption"},
{"block_id": "2", "page": 1, "role": "figure_asset", "bbox": [0, 0, 10, 10]},
]
corpus = FigureCorpus.from_blocks(blocks, page_width=1200)
assert [b["block_id"] for b in corpus.raw_legends] == ["1"]
assert [b["block_id"] for b in corpus.raw_assets] == ["2"]
assert corpus.page_width == 1200
def test_candidate_index_holds_derived_hypotheses_not_raw_facts():
blocks = [
{"block_id": "1", "page": 1, "role": "figure_caption", "text": "Figure 1. Caption"},
{"block_id": "2", "page": 1, "role": "figure_asset", "bbox": [0, 0, 10, 10]},
]
corpus = FigureCorpus.from_blocks(blocks, page_width=1200)
index = FigureCandidateIndex.from_corpus(corpus)
assert len(index.formal_legends) == 1
assert hasattr(index, "candidate_groups")
assert corpus.raw_legends[0]["block_id"] == "1"
assert index.bundle_source_legend_ids == set()
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `python -m pytest tests/test_ocr_figure_vnext_corpus.py -v`
Expected: FAIL because the corpus/index module does not exist.
- [ ] **Step 3: Implement `FigureCorpus` and `FigureCandidateIndex`**
```python
# paperforge/worker/ocr_figure_vnext_corpus.py
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
@dataclass
class FigureCorpus:
blocks: list[dict]
page_width: float
raw_legends: list[dict] = field(default_factory=list)
raw_assets: list[dict] = field(default_factory=list)
locator_candidates: list[dict] = field(default_factory=list)
page_layouts: dict[int, Any] = field(default_factory=dict)
@classmethod
def from_blocks(cls, blocks: list[dict], page_width: float = 1200) -> "FigureCorpus":
from . import ocr_figures
from .ocr_document import _build_page_layout_profiles
raw_legends = [b for b in blocks if b.get("role") in {"figure_caption", "figure_caption_candidate"}]
raw_assets = [b for b in blocks if b.get("role") in {"figure_asset", "media_asset"}]
locator_candidates = [b for b in raw_legends if ocr_figures._is_previous_page_legend_locator(b)]
return cls(
blocks=list(blocks),
page_width=page_width,
raw_legends=raw_legends,
raw_assets=raw_assets,
locator_candidates=locator_candidates,
page_layouts=_build_page_layout_profiles(blocks),
)
@dataclass
class FigureCandidateIndex:
formal_legends: list[dict]
held_legends: list[dict]
rejected_legends: list[dict]
deduped_legends: list[dict]
candidate_groups: list[dict]
competing_caption_pages: set[int]
sidecar_candidates: dict[int, list[dict]]
bundle_source_legend_ids: set[str]
locator_candidates: list[dict]
@classmethod
def from_corpus(cls, corpus: FigureCorpus) -> "FigureCandidateIndex":
from . import ocr_figures
formal_legends = [b for b in corpus.raw_legends if ocr_figures._is_formal_legend(str(b.get("text", "")), b, corpus.page_width)]
rejected_legends = [b for b in corpus.raw_legends if b not in formal_legends]
candidate_groups = ocr_figures._build_candidate_figure_groups_from_assets(
corpus.raw_assets,
formal_legends,
corpus.blocks,
page_width=corpus.page_width,
)
competing_caption_pages = {
int(leg.get("page"))
for leg in formal_legends
if leg.get("page") is not None and ocr_figures._extract_figure_number(str(leg.get("text", ""))) is not None
}
return cls(
formal_legends=formal_legends,
held_legends=[],
rejected_legends=rejected_legends,
deduped_legends=formal_legends,
candidate_groups=candidate_groups,
competing_caption_pages=competing_caption_pages,
sidecar_candidates={},
bundle_source_legend_ids=set(),
locator_candidates=corpus.locator_candidates,
)
```
- [ ] **Step 4: Run tests to verify they pass**
Run: `python -m pytest tests/test_ocr_figure_vnext_corpus.py -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_corpus.py tests/test_ocr_figure_vnext_corpus.py
git commit -m "feat(ocr): add vnext figure corpus and candidate index"
```
- [ ] **Step 6: Manual review gate**
Reviewer checkpoint before Task 4:
- verify no circular import between `ocr_figures.py` and `ocr_figure_vnext_corpus.py`
- verify `bundle_source_legend_ids` remains empty in this milestone
- verify `FigureCorpus` stores facts only and `FigureCandidateIndex` stores hypotheses only
```
- [ ] **Step 4: Run tests to verify they pass**
Run: `python -m pytest tests/test_ocr_figure_vnext_corpus.py -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_corpus.py tests/test_ocr_figure_vnext_corpus.py
git commit -m "feat(ocr): add vnext figure corpus and candidate index"
```
### Task 4: Implement `PrimarySamePagePass` and wire vnext milestone
**Files:**
- Create: `paperforge/worker/ocr_figure_vnext_passes.py`
- Modify: `paperforge/worker/ocr_figures.py`
- Modify: `scripts/dev/compare_figure_inventory_legacy_vs_vnext.py`
- Test: `tests/test_ocr_figure_vnext_passes.py`
- Test: `tests/test_ocr_figure_vnext_compare.py`
**Interfaces:**
- Consumes:
- `FigureCorpus`
- `FigureCandidateIndex`
- `OwnershipLedger`
- existing helpers `_score_legend_to_group`, `_project_asset_record`, `_extract_figure_number`, `_extract_figure_namespace`, `_format_figure_id`
- Produces:
- `PrimarySamePagePass.run(state: FigurePipelineState) -> PassReport`
- `build_figure_inventory_vnext(...)` returning same-page primary matches only
**Execution guard:** Do not start Task 4 until Task 3 has been reviewed.
- [ ] **Step 1: Write failing same-page primary tests**
```python
# tests/test_ocr_figure_vnext_passes.py
from paperforge.worker.ocr_figure_vnext_corpus import FigureCorpus, FigureCandidateIndex
from paperforge.worker.ocr_figure_vnext_passes import PrimarySamePagePass
from paperforge.worker.ocr_figure_vnext_state import FigurePipelineState, OwnershipLedger
def test_primary_same_page_pass_matches_single_safe_group():
blocks = [
{"block_id": "c1", "page": 1, "role": "figure_caption", "text": "Figure 1. Caption", "bbox": [0, 100, 200, 150]},
{"block_id": "a1", "page": 1, "role": "figure_asset", "bbox": [0, 0, 200, 90], "raw_label": "image"},
]
corpus = FigureCorpus.from_blocks(blocks, page_width=1200)
index = FigureCandidateIndex.from_corpus(corpus)
state = FigurePipelineState(corpus=corpus, candidate_index=index, ledger=OwnershipLedger())
report = PrimarySamePagePass().run(state)
assert len(report.accepted) == 1
assert report.accepted[0].claim_type == "match"
assert len(state.matches) == 1
def test_primary_same_page_pass_prefers_higher_score_when_two_legends_compete_for_one_asset(monkeypatch):
blocks = [
{"block_id": "c1", "page": 1, "role": "figure_caption", "text": "Figure 1. Caption", "bbox": [0, 100, 200, 150]},
{"block_id": "c2", "page": 1, "role": "figure_caption", "text": "Figure 2. Caption", "bbox": [0, 160, 200, 210]},
{"block_id": "a1", "page": 1, "role": "figure_asset", "bbox": [0, 0, 200, 90], "raw_label": "image"},
]
corpus = FigureCorpus.from_blocks(blocks, page_width=1200)
index = FigureCandidateIndex.from_corpus(corpus)
if not index.candidate_groups:
index.candidate_groups = [{\"group_id\": \"g1\", \"page\": 1, \"asset_block_ids\": [\"a1\"], \"media_blocks\": [{\"block_id\": \"a1\"}], \"group_type\": \"single_asset\", \"cluster_bbox\": [0, 0, 200, 90]}]
scores = [{\"score\": 0.4, \"decision\": \"matched\", \"evidence\": [\"low\"]}, {\"score\": 0.9, \"decision\": \"matched\", \"evidence\": [\"high\"]}]
def fake_score(*args, **kwargs):
return scores.pop(0)
monkeypatch.setattr(\"paperforge.worker.ocr_figures._score_legend_to_group\", fake_score)
monkeypatch.setattr(\"paperforge.worker.ocr_figures.score_figure_caption\", lambda *a, **k: {\"score\": 0.9})
state = FigurePipelineState(corpus=corpus, candidate_index=index, ledger=OwnershipLedger())
report = PrimarySamePagePass().run(state)
assert len(report.accepted) == 1
assert report.accepted[0].figure_no == 2
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -v`
Expected: FAIL because `PrimarySamePagePass` does not exist and arbitration is not implemented.
- [ ] **Step 3: Implement the pass with collect-then-commit arbitration**
```python
# paperforge/worker/ocr_figure_vnext_passes.py
from __future__ import annotations
from .ocr_figure_vnext_types import ClaimProposal, PassReport, ResourceRef
def _resource_page(block: dict) -> int | None:
page = block.get("page")
if page is None:
page = block.get("page_num")
return int(page) if page is not None else None
class PrimarySamePagePass:
name = "primary_same_page"
def _collect_proposals(self, state):
from . import ocr_figures
proposals = []
for legend in state.candidate_index.deduped_legends:
page = _resource_page(legend)
if page is None:
continue
page_groups = [g for g in state.candidate_index.candidate_groups if _resource_page(g) == page]
for group in page_groups:
score = ocr_figures._score_legend_to_group(
legend,
group,
caption_score=ocr_figures.score_figure_caption(legend, nearby_media=True, caption_style_match=False, body_prose_likelihood=False),
page_width=state.corpus.page_width,
)
if score.get("decision") != "matched":
continue
figure_no = ocr_figures._extract_figure_number(str(legend.get("text", "")))
proposals.append(ClaimProposal(
pass_name=self.name,
figure_no=figure_no,
claim_type="match",
legends=[ResourceRef(kind="legend", page=page, block_id=legend.get("block_id"), figure_no=figure_no)],
assets=[ResourceRef(kind="asset", page=page, block_id=bid) for bid in group.get("asset_block_ids", [])],
groups=[ResourceRef(kind="group", page=page, block_id=None, group_id=group.get("group_id"))],
confidence=float(score.get("score", 0.0)),
evidence_rank=1,
reason="same_page_primary",
diagnostics={"evidence": list(score.get("evidence", [])), "legend_block_id": str(legend.get("block_id", ""))},
))
return proposals
def _materialize_match(self, state, proposal):
from . import ocr_figures
legend = proposal.legends[0]
page = legend.page
asset_ids = {str(r.block_id) for r in proposal.assets}
matched_assets = [ocr_figures._project_asset_record(a) for a in state.corpus.raw_assets if _resource_page(a) == page and str(a.get("block_id", "")) in asset_ids]
legend_text = next(str(b.get("text", "")) for b in state.candidate_index.deduped_legends if str(b.get("block_id", "")) == legend.block_id)
namespace = ocr_figures._extract_figure_namespace(legend_text)
return {\n \"figure_id\": ocr_figures._format_figure_id(namespace, proposal.figure_no),\n \"figure_namespace\": namespace,\n \"figure_number\": proposal.figure_no,\n \"legend_block_id\": legend.block_id,\n \"page\": page,\n \"text\": legend_text,\n \"matched_assets\": matched_assets,\n \"asset_block_ids\": sorted(asset_ids),\n \"settlement_type\": \"same_page\",\n \"confidence\": proposal.confidence,\n \"match_score\": {\"score\": proposal.confidence, \"decision\": \"matched\", \"evidence\": proposal.diagnostics[\"evidence\"]},\n \"flags\": [],\n \"bridge_block_ids\": [],\n }
def run(self, state):
report = PassReport(pass_name=self.name)
proposals = self._collect_proposals(state)
report.proposals.extend(proposals)
for proposal in sorted(proposals, key=lambda p: (p.evidence_rank, -p.confidence, -(p.figure_no or -1))):
conflict = state.ledger.try_claim_assets(proposal.assets, owner=proposal.legends[0], reason=proposal.reason)
if conflict is not None:
report.conflicts.append(conflict)
report.rejected.append(proposal)
continue
state.accept_match(proposal, self._materialize_match(state, proposal))
report.accepted.append(proposal)
return report
```
```python
# paperforge/worker/ocr_figures.py
from dataclasses import asdict
def build_figure_inventory_vnext(structured_blocks: list[dict], page_width: float = 1200) -> dict[str, Any]:
from .ocr_figure_vnext_corpus import FigureCandidateIndex, FigureCorpus
from .ocr_figure_vnext_passes import PrimarySamePagePass, _resource_page
from .ocr_figure_vnext_state import FigurePipelineState, OwnershipLedger
corpus = FigureCorpus.from_blocks(structured_blocks, page_width=page_width)
candidate_index = FigureCandidateIndex.from_corpus(corpus)
state = FigurePipelineState(corpus=corpus, candidate_index=candidate_index, ledger=OwnershipLedger())
report = PrimarySamePagePass().run(state)
matched_ids = {str(m.get("legend_block_id", "")) for m in state.matches}
return {
\"pipeline_mode\": \"vnext\",
\"matched_figures\": state.matches,
\"ambiguous_figures\": [],
\"unmatched_legends\": [b for b in candidate_index.deduped_legends if str(b.get(\"block_id\", \"\")) not in matched_ids],
\"unmatched_assets\": [a for a in corpus.raw_assets if (_resource_page(a) is not None and state.ledger.owner_of_asset(page=_resource_page(a), block_id=a.get(\"block_id\")) is None)],
\"unresolved_clusters\": [],
\"held_figures\": list(candidate_index.held_legends),
\"rejected_legends\": list(candidate_index.rejected_legends),
\"page_ledger\": {},
\"residual_ledger\": {},
\"local_pairing_hypotheses\": [],
\"pass_reports\": [asdict(report)],
\"completeness\": {\"total_numbered_legends\": len(candidate_index.deduped_legends), \"accounted_for\": len(state.matches), \"details\": []},
}
```
- [ ] **Step 4: Add comparison smoke test**
```python
# tests/test_ocr_figure_vnext_compare.py
from paperforge.worker.ocr_figures import build_figure_inventory_legacy, build_figure_inventory_vnext
from scripts.dev.compare_figure_inventory_legacy_vs_vnext import compare_inventories
def test_compare_inventories_reports_counts_for_same_page_case():
blocks = [
{\"block_id\": \"c1\", \"page\": 1, \"role\": \"figure_caption\", \"text\": \"Figure 1. Caption\", \"bbox\": [0, 100, 200, 150]},
{\"block_id\": \"a1\", \"page\": 1, \"role\": \"figure_asset\", \"bbox\": [0, 0, 200, 90], \"raw_label\": \"image\"},
]
legacy = build_figure_inventory_legacy(blocks, 1200)
vnext = build_figure_inventory_vnext(blocks, 1200)
diff = compare_inventories(legacy, vnext)
assert diff[\"vnext_matched_count\"] >= 1
assert \"vnext_consumed_block_ids\" in diff
```
- [ ] **Step 5: Run tests to verify they pass**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py tests/test_ocr_figure_vnext_compare.py -v`
Expected: PASS
- [ ] **Step 6: Commit**
```bash
git add paperforge/worker/ocr_figures.py paperforge/worker/ocr_figure_vnext_passes.py scripts/dev/compare_figure_inventory_legacy_vs_vnext.py tests/test_ocr_figure_vnext_passes.py tests/test_ocr_figure_vnext_compare.py
git commit -m "feat(ocr): add vnext same-page figure pass"
```
### Task 5: Portable real-paper diff harness for milestone corpus
**Files:**
- Create: `tests/fixtures/ocr_vnext_real_papers/2HEUD5P9/blocks.structured.jsonl`
- Modify: `scripts/dev/compare_figure_inventory_legacy_vs_vnext.py`
- Create: `tests/test_ocr_figure_vnext_real_papers.py`
- Read-only source: `D:/L/OB/Literature-hub/System/PaperForge/ocr/2HEUD5P9/structure/blocks.structured.jsonl`
**Interfaces:**
- Consumes:
- `build_figure_inventory_legacy(...)`
- `build_figure_inventory_vnext(...)`
- `compare_inventories(...)`
- Produces:
- portable repo fixture for one representative real paper
- `compare_blocks_file(blocks_path: Path) -> dict[str, object]`
- milestone diff report for one representative real paper
- [ ] **Step 1: Copy the representative real-paper blocks into a repo fixture**
```python
from pathlib import Path
from shutil import copy2
src = Path(r"D:/L/OB/Literature-hub/System/PaperForge/ocr/2HEUD5P9/structure/blocks.structured.jsonl")
dst = Path("tests/fixtures/ocr_vnext_real_papers/2HEUD5P9/blocks.structured.jsonl")
dst.parent.mkdir(parents=True, exist_ok=True)
copy2(src, dst)
```
- [ ] **Step 2: Write the failing fixture-backed test**
```python
# tests/test_ocr_figure_vnext_real_papers.py
from pathlib import Path
from scripts.dev.compare_figure_inventory_legacy_vs_vnext import compare_blocks_file
def test_real_paper_same_page_milestone_reports_diff_shape():
blocks_path = Path("tests/fixtures/ocr_vnext_real_papers/2HEUD5P9/blocks.structured.jsonl")
diff = compare_blocks_file(blocks_path)
assert diff["paper"] == "2HEUD5P9"
assert "legacy_matched_count" in diff
assert "vnext_matched_count" in diff
assert "legacy_consumed_block_ids" in diff
assert "vnext_consumed_block_ids" in diff
- [ ] **Step 3: Run the test to verify it fails**
Run: `python -m pytest tests/test_ocr_figure_vnext_real_papers.py -v`
Expected: FAIL because `compare_blocks_file(...)` does not exist yet.
- [ ] **Step 4: Add helper for loading one paper and reporting the milestone diff contract**
```python
# scripts/dev/compare_figure_inventory_legacy_vs_vnext.py
from pathlib import Path
def _figure_asset_ids(fig: dict[str, object]) -> list[str]:
ids = {str(x) for x in fig.get(\"asset_block_ids\", [])}
ids.update(str(a.get(\"block_id\", \"\")) for a in fig.get(\"matched_assets\", []))
return sorted(x for x in ids if x)
- [ ] **Step 4: Add helper for loading one paper and reporting the milestone diff contract**
```python
# scripts/dev/compare_figure_inventory_legacy_vs_vnext.py
from pathlib import Path
def _figure_asset_ids(fig: dict[str, object]) -> list[str]:
ids = {str(x) for x in fig.get("asset_block_ids", [])}
ids.update(str(a.get("block_id", "")) for a in fig.get("matched_assets", []))
return sorted(x for x in ids if x)
def compare_blocks_file(blocks_path: Path) -> dict[str, object]:
blocks = [json.loads(line) for line in blocks_path.read_text(encoding="utf-8").splitlines() if line.strip()]
legacy = build_figure_inventory_legacy(blocks)
vnext = build_figure_inventory_vnext(blocks)
diff = compare_inventories(legacy, vnext)
diff["paper"] = blocks_path.parent.name
return diff
```
- [ ] **Step 5: Re-run the real-paper milestone test**
Run: `python -m pytest tests/test_ocr_figure_vnext_real_papers.py -v`
Expected: PASS
- [ ] **Step 6: Commit**
```bash
git add tests/fixtures/ocr_vnext_real_papers/2HEUD5P9/blocks.structured.jsonl scripts/dev/compare_figure_inventory_legacy_vs_vnext.py tests/test_ocr_figure_vnext_real_papers.py
git commit -m "test(ocr): add portable vnext real-paper comparison harness"
```
## Self-Review
- Spec coverage: This plan deliberately covers only the first safe milestone from the spec: mechanical extraction, comparison harness, core contracts, immutable corpus/candidate split, and same-page primary matching. It intentionally leaves sidecar, bundle, locator, group-aware sequential, classic sequential, composite-parent settlement, and final cutover for later plans.
- Placeholder scan: No `TBD`, `TODO`, temporary fixture placeholders, or deferred implementation markers remain in the task steps. Task 5 uses a repo fixture, not a machine-local pytest path.
- Type consistency: `ResourceRef`, `ClaimProposal`, `PassReport`, `OwnershipLedger`, `FigureCorpus`, `FigureCandidateIndex`, `FigurePipelineState.accept_match(...)`, and `PrimarySamePagePass` are defined before later tasks consume them.
## Execution Handoff
Plan complete and saved to `docs/superpowers/plans/2026-07-03-figure-pipeline-vnext-phase0-phase1.md`. Two execution options:
1. **Subagent-Driven (recommended)** - I dispatch a fresh subagent per task, review between tasks, fast iteration
2. **Inline Execution** - Execute tasks in this session using executing-plans, batch execution with checkpoints
Which approach?

View file

@ -0,0 +1,427 @@
# Figure Pipeline VNext Phase 2 — Cross-Page Reservation + Settlement Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Extend the vnext figure pipeline from same-page primary matching to cross-page reservation and cross-page settlement, while still excluding sidecar, bundle, locator, group-aware sequential, classic sequential, composite-parent settlement, and cutover.
**Architecture:** Build on the completed vnext seams from Phase 0 + Phase 1: `FigureCorpus`, `FigureCandidateIndex`, `OwnershipLedger`, `FigurePipelineState`, and `PrimarySamePagePass`. Add two new passes — `CrossPageReservationPass` and `CrossPageSettlementPass` — that emit and consume explicit proposals without stealing resources from accepted same-page matches.
**Tech Stack:** Python 3, pytest, existing OCR helpers in `paperforge/worker/ocr_figures.py`, vnext modules under `paperforge/worker/`, portable repo fixtures under `tests/fixtures/ocr_vnext_real_papers/`
## Global Constraints
- Build on branch/worktree `feat/figure-pipeline-vnext`; do not edit the main checkout.
- Preserve the external interface `build_figure_inventory(structured_blocks, page_width=1200) -> FigureInventory`.
- Legacy implementation remains the immutable baseline.
- Keep `ResourceRef` as the only ownership key.
- Cross-page reservation/settlement must remain separate from sidecar, bundle, locator, group-aware sequential, classic sequential, and cutover.
- Disabled later passes must emit no proposals.
- Final arbitration priority is independent of migration order.
- New pass reports must remain JSON-safe.
- Add only the tests directly covering this phase; do not broaden to the whole OCR suite inside task steps.
---
### Task 1: Extend state and ledger for reservations
**Files:**
- Modify: `paperforge/worker/ocr_figure_vnext_types.py`
- Modify: `paperforge/worker/ocr_figure_vnext_state.py`
- Create: `tests/test_ocr_figure_vnext_reservations.py`
**Interfaces:**
- Consumes:
- `ResourceRef`
- `ClaimProposal`
- `PassReport`
- `OwnershipLedger`
- `FigurePipelineState`
- Produces:
- `OwnershipLedger.reserve_group(...)`
- `OwnershipLedger.transition_reserved_group_to_claimed(...)`
- `OwnershipLedger.can_claim_group(...) -> bool`
- `FigurePipelineState.accept_reservation(...)`
- [ ] **Step 1: Write failing reservation tests**
```python
# tests/test_ocr_figure_vnext_reservations.py
from paperforge.worker.ocr_figure_vnext_types import ClaimProposal, ResourceRef
from paperforge.worker.ocr_figure_vnext_state import FigurePipelineState, OwnershipLedger
def test_ledger_can_reserve_group_without_claiming_assets():
ledger = OwnershipLedger()
group = ResourceRef(kind="group", page=2, block_id=None, group_id="g2")
ledger.reserve_group(group, reason="cross_page_candidate")
assert ledger.can_claim_group(group) is False
assert any(entry["action"] == "reserve_group" for entry in ledger.snapshot())
def test_pipeline_state_accept_reservation_records_diagnostic():
state = FigurePipelineState(corpus=None, candidate_index=None, ledger=OwnershipLedger())
proposal = ClaimProposal(
pass_name="cross_page_reservation",
figure_no=4,
claim_type="reserve",
legends=[ResourceRef(kind="legend", page=1, block_id="l1", figure_no=4)],
assets=[],
groups=[ResourceRef(kind="group", page=2, block_id=None, group_id="g2")],
confidence=0.7,
evidence_rank=2,
reason="forward_cross_page_candidate",
diagnostics={"evidence": ["page_gap"]},
)
state.accept_reservation(proposal)
assert state.diagnostics[-1]["event"] == "reservation_accepted"
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `python -m pytest tests/test_ocr_figure_vnext_reservations.py -v`
Expected: FAIL because reservation methods/state hooks do not exist yet.
- [ ] **Step 3: Extend types/state minimally for reservation support**
```python
# paperforge/worker/ocr_figure_vnext_state.py
class OwnershipLedger:
def __init__(self) -> None:
self._owners: dict[ResourceRef, ResourceRef] = {}
self._reserved_groups: set[ResourceRef] = set()
self._journal: list[dict[str, object]] = []
def reserve_group(self, group: ResourceRef, *, reason: str) -> None:
self._reserved_groups.add(group)
self._journal.append({"action": "reserve_group", "group": group, "reason": reason})
def can_claim_group(self, group: ResourceRef) -> bool:
return group not in self._reserved_groups
def transition_reserved_group_to_claimed(self, group: ResourceRef, *, owner: ResourceRef, reason: str) -> None:
if group in self._reserved_groups:
self._reserved_groups.remove(group)
self._journal.append({"action": "claim_reserved_group", "group": group, "owner": owner, "reason": reason})
@dataclass
class FigurePipelineState:
...
reservations: list[dict] = field(default_factory=list)
def accept_reservation(self, proposal: ClaimProposal) -> None:
self.reservations.append({
"pass_name": proposal.pass_name,
"figure_no": proposal.figure_no,
"reason": proposal.reason,
"groups": proposal.groups,
"legends": proposal.legends,
})
self.diagnostics.append({
"event": "reservation_accepted",
"pass_name": proposal.pass_name,
"figure_no": proposal.figure_no,
"reason": proposal.reason,
"resources": {
"legends": proposal.legends,
"groups": proposal.groups,
},
})
```
- [ ] **Step 4: Run reservation tests to verify they pass**
Run: `python -m pytest tests/test_ocr_figure_vnext_reservations.py -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_state.py tests/test_ocr_figure_vnext_reservations.py
git commit -m "feat(ocr): add vnext cross-page reservation state"
```
### Task 2: Implement `CrossPageReservationPass`
**Files:**
- Modify: `paperforge/worker/ocr_figure_vnext_passes.py`
- Modify: `tests/test_ocr_figure_vnext_passes.py`
**Interfaces:**
- Consumes:
- `FigurePipelineState`
- `ClaimProposal`
- `PassReport`
- `ResourceRef`
- existing helpers from `ocr_figures.py` for page/group scoring
- Produces:
- `CrossPageReservationPass.run(state) -> PassReport`
- [ ] **Step 1: Add failing reservation pass tests**
```python
# tests/test_ocr_figure_vnext_passes.py
from paperforge.worker.ocr_figure_vnext_passes import CrossPageReservationPass
def test_cross_page_reservation_pass_reserves_forward_group_when_same_page_primary_misses():
# use a synthetic state where legend is on page 1 and only group is on page 2
...
def test_cross_page_reservation_pass_does_not_reserve_group_already_claimed_same_page():
# use a synthetic state with an already-owned same-page asset/group
...
```
- [ ] **Step 2: Run the failing reservation pass tests**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -k cross_page_reservation -v`
Expected: FAIL because `CrossPageReservationPass` does not exist yet.
- [ ] **Step 3: Implement the reservation pass**
```python
# paperforge/worker/ocr_figure_vnext_passes.py
class CrossPageReservationPass:
name = "cross_page_reservation"
def run(self, state):
report = PassReport(pass_name=self.name)
for legend in state.candidate_index.deduped_legends:
page = _resource_page(legend)
if page is None:
continue
if any(str(m.get("legend_block_id", "")) == str(legend.get("block_id", "")) for m in state.matches):
continue
forward_groups = [
g for g in state.candidate_index.candidate_groups
if _resource_page(g) is not None and _resource_page(g) > page
]
for group in forward_groups[:1]:
group_ref = ResourceRef(kind="group", page=_resource_page(group), block_id=None, group_id=group.get("group_id"))
if not state.ledger.can_claim_group(group_ref):
continue
proposal = ClaimProposal(
pass_name=self.name,
figure_no=None,
claim_type="reserve",
legends=[ResourceRef(kind="legend", page=page, block_id=legend.get("block_id"), figure_no=None)],
assets=[],
groups=[group_ref],
confidence=0.6,
evidence_rank=2,
reason="forward_cross_page_candidate",
diagnostics={"evidence": ["page_gap", "same_page_miss"]},
)
report.proposals.append(proposal)
state.ledger.reserve_group(group_ref, reason=proposal.reason)
state.accept_reservation(proposal)
report.accepted.append(proposal)
break
return report
```
- [ ] **Step 4: Run the reservation pass tests**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -k cross_page_reservation -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_passes.py tests/test_ocr_figure_vnext_passes.py
git commit -m "feat(ocr): add vnext cross-page reservation pass"
```
### Task 3: Implement `CrossPageSettlementPass`
**Files:**
- Modify: `paperforge/worker/ocr_figure_vnext_passes.py`
- Modify: `tests/test_ocr_figure_vnext_passes.py`
- Modify: `paperforge/worker/ocr_figures.py`
**Interfaces:**
- Consumes:
- `CrossPageReservationPass` output in `state.reservations`
- `OwnershipLedger.transition_reserved_group_to_claimed(...)`
- Produces:
- `CrossPageSettlementPass.run(state) -> PassReport`
- vnext orchestrator calling same-page pass, then reservation pass, then settlement pass
- [ ] **Step 1: Add failing settlement tests**
```python
# tests/test_ocr_figure_vnext_passes.py
from paperforge.worker.ocr_figure_vnext_passes import CrossPageSettlementPass
def test_cross_page_settlement_pass_claims_reserved_group_into_match_record():
...
def test_cross_page_settlement_pass_keeps_same_page_matches_untouched():
...
```
- [ ] **Step 2: Run the failing settlement tests**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -k cross_page_settlement -v`
Expected: FAIL because `CrossPageSettlementPass` does not exist yet.
- [ ] **Step 3: Implement the settlement pass and wire orchestrator order**
```python
# paperforge/worker/ocr_figure_vnext_passes.py
class CrossPageSettlementPass:
name = "cross_page_settlement"
def run(self, state):
report = PassReport(pass_name=self.name)
for reservation in state.reservations:
legend = reservation["legends"][0]
group = reservation["groups"][0]
state.ledger.transition_reserved_group_to_claimed(group, owner=legend, reason="cross_page_settlement")
proposal = ClaimProposal(
pass_name=self.name,
figure_no=reservation["figure_no"],
claim_type="match",
legends=[legend],
assets=[],
groups=[group],
confidence=0.6,
evidence_rank=2,
reason="cross_page_settlement",
diagnostics={"evidence": ["reservation_claimed"]},
)
state.accept_match(proposal, {
"figure_id": f"figure_reserved_{len(state.matches):03d}",
"figure_namespace": "figure",
"figure_number": reservation["figure_no"],
"legend_block_id": legend.block_id,
"page": legend.page,
"text": "",
"matched_assets": [],
"asset_block_ids": [],
"settlement_type": "cross_page_reservation",
"confidence": 0.6,
"match_score": {"score": 0.6, "decision": "matched", "evidence": ["reservation_claimed"]},
"flags": ["cross_page_reserved"],
"bridge_block_ids": [],
})
report.accepted.append(proposal)
return report
```
```python
# paperforge/worker/ocr_figures.py
from dataclasses import asdict
def build_figure_inventory_vnext(structured_blocks: list[dict], page_width: float = 1200) -> dict[str, Any]:
from .ocr_figure_vnext_corpus import FigureCandidateIndex, FigureCorpus
from .ocr_figure_vnext_passes import PrimarySamePagePass, CrossPageReservationPass, CrossPageSettlementPass, _resource_page
from .ocr_figure_vnext_state import FigurePipelineState, OwnershipLedger
corpus = FigureCorpus.from_blocks(structured_blocks, page_width=page_width)
candidate_index = FigureCandidateIndex.from_corpus(corpus)
state = FigurePipelineState(corpus=corpus, candidate_index=candidate_index, ledger=OwnershipLedger())
reports = []
for pass_cls in (PrimarySamePagePass, CrossPageReservationPass, CrossPageSettlementPass):
reports.append(pass_cls().run(state))
...
"pass_reports": [asdict(r) for r in reports],
```
- [ ] **Step 4: Run the settlement tests**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -k cross_page_settlement -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_passes.py paperforge/worker/ocr_figures.py tests/test_ocr_figure_vnext_passes.py
git commit -m "feat(ocr): add vnext cross-page settlement pass"
```
### Task 4: Portable cross-page real-paper harness
**Files:**
- Create: `tests/fixtures/ocr_vnext_real_papers/DWQQK2YB/blocks.structured.jsonl`
- Modify: `tests/test_ocr_figure_vnext_real_papers.py`
- Modify: `scripts/dev/compare_figure_inventory_legacy_vs_vnext.py`
**Interfaces:**
- Consumes:
- `compare_blocks_file(...)`
- Produces:
- second portable real-paper fixture for a cross-page/locator-style paper
- fixture-backed regression asserting diff contract shape for the cross-page milestone
- [ ] **Step 1: Copy the representative cross-page paper into a repo fixture**
```python
from pathlib import Path
from shutil import copy2
src = Path(r"D:/L/OB/Literature-hub/System/PaperForge/ocr/DWQQK2YB/structure/blocks.structured.jsonl")
dst = Path("tests/fixtures/ocr_vnext_real_papers/DWQQK2YB/blocks.structured.jsonl")
dst.parent.mkdir(parents=True, exist_ok=True)
copy2(src, dst)
```
- [ ] **Step 2: Add the failing cross-page fixture-backed test**
```python
# tests/test_ocr_figure_vnext_real_papers.py
from pathlib import Path
from scripts.dev.compare_figure_inventory_legacy_vs_vnext import compare_blocks_file
def test_real_paper_cross_page_milestone_reports_diff_shape():
blocks_path = Path("tests/fixtures/ocr_vnext_real_papers/DWQQK2YB/blocks.structured.jsonl")
diff = compare_blocks_file(blocks_path)
assert diff["paper"] == "DWQQK2YB"
assert "legacy_matched_count" in diff
assert "vnext_matched_count" in diff
```
- [ ] **Step 3: Run the cross-page fixture test to verify it fails before implementation completes**
Run: `python -m pytest tests/test_ocr_figure_vnext_real_papers.py -k cross_page_milestone -v`
Expected: FAIL until the fixture and cross-page passes are wired.
- [ ] **Step 4: Re-run the real-paper fixture tests after Tasks 2 and 3**
Run: `python -m pytest tests/test_ocr_figure_vnext_real_papers.py -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add tests/fixtures/ocr_vnext_real_papers/DWQQK2YB/blocks.structured.jsonl tests/test_ocr_figure_vnext_real_papers.py
git commit -m "test(ocr): add portable cross-page real-paper harness"
```
## Self-Review
- Spec coverage: This phase covers only cross-page reservation + settlement and the portable harness needed to observe that milestone. It intentionally leaves sidecar, bundle, locator-specific reconciliation, group-aware sequential, classic sequential, composite-parent settlement, and cutover to later plans.
- Placeholder scan: No `TBD`, `TODO`, or machine-local pytest paths remain in task steps; local absolute paths are used only as read-only source locations for copying repo fixtures.
- Type consistency: reservation and settlement interfaces extend the existing vnext seams instead of bypassing them.
## Execution Handoff
Plan complete and saved to `docs/superpowers/plans/2026-07-03-figure-pipeline-vnext-phase2-cross-page.md`. Two execution options:
1. **Subagent-Driven (recommended)** - I dispatch a fresh subagent per task, review between tasks, fast iteration
2. **Inline Execution** - Execute tasks in this session using executing-plans, batch execution with checkpoints
Which approach?

View file

@ -0,0 +1,841 @@
# Figure Pipeline VNext Phase 3 — Special Fallbacks (Sidecar + LegendBundle + LocatorBridge) Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Implement the three special fallback passes — `SidecarPass`, `LegendBundlePass`, `LocatorBridgePass` — that handle the non-same-page matching scenarios the legacy pipeline covers in its 400+ line fallback section. These are spec §7.3 Layer 1 (sidecar, locator) and Layer 2 (legend bundle) passes.
**Architecture:** Build on the completed Phase 02 vnext seams. Each pass follows the established pattern: collect proposals → arbitrate via `OwnershipLedger``accept_match` or `accept_reservation`. The passes reuse existing legacy helper functions (`_same_page_narrow_caption_column`, `_partition_assets_by_caption_bands`, `_identify_bundle_source_legend_ids`, `_is_previous_page_legend_locator`, `_is_tight_asset_cluster`) via lazy import — the helpers are immutable utilities, not legacy pipeline state.
**Tech Stack:** Python 3, pytest, existing OCR helpers in `paperforge/worker/ocr_figures.py`, vnext modules under `paperforge/worker/`, portable repo fixtures under `tests/fixtures/ocr_vnext_real_papers/`
## Global Constraints
- Build on branch/worktree `feat/figure-pipeline-vnext`; do not edit the main checkout.
- Preserve the external interface `build_figure_inventory(structured_blocks, page_width=1200) -> FigureInventory`.
- Legacy implementation remains the immutable baseline.
- Keep `ResourceRef` as the only ownership key.
- Special fallbacks must not steal resources already claimed by same-page or cross-page settlement passes.
- Disabled passes must emit no proposals.
- Final arbitration priority is independent of migration order.
- New pass reports must remain JSON-safe.
- Reuse existing legacy helper functions via lazy import — do not duplicate them.
- Add only the tests directly covering this phase; do not broaden to the whole OCR suite inside task steps.
- Do not implement group-aware sequential, classic sequential, composite-parent settlement, completeness/accounting, or final cutover in this plan.
---
### Task 1: Populate sidecar_candidates and bundle_source_legend_ids in FigureCandidateIndex
**Rationale:** `FigureCandidateIndex` currently leaves `sidecar_candidates={}` and `bundle_source_legend_ids=set()` empty. The special fallback passes need these pre-computed. This task populates them in `from_corpus` without changing any existing field.
**Files:**
- Modify: `paperforge/worker/ocr_figure_vnext_corpus.py`
- Modify: `tests/test_ocr_figure_vnext_corpus.py`
**Interfaces:**
- Consumes:
- `FigureCorpus` (raw_legends, raw_assets, page_width)
- `ocr_figures._same_page_narrow_caption_column(legends, page_width) -> list[dict]`
- `ocr_figures._identify_bundle_source_legend_ids(legends, assets) -> set[str]`
- Produces:
- `FigureCandidateIndex.sidecar_candidates` populated as `dict[int, list[dict]]` (page → narrow caption set, only pages with ≥2 narrow captions)
- `FigureCandidateIndex.bundle_source_legend_ids` populated from `_identify_bundle_source_legend_ids`
- [ ] **Step 1: Write failing tests**
Add to `tests/test_ocr_figure_vnext_corpus.py`:
```python
def test_candidate_index_populates_sidecar_candidates_for_narrow_caption_page():
# 3 narrow captions on page 5, each with width < page_width * 0.4
# Assert index.sidecar_candidates has page 5 with 3 entries
...
def test_candidate_index_populates_bundle_source_legend_ids():
# legends + assets where _identify_bundle_source_legend_ids returns non-empty
# Assert index.bundle_source_legend_ids is non-empty
...
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `python -m pytest tests/test_ocr_figure_vnext_corpus.py -k "sidecar_candidates or bundle_source" -v`
Expected: FAIL
- [ ] **Step 3: Populate the fields in `from_corpus`**
```python
# In FigureCandidateIndex.from_corpus, before the return:
sidecar_candidates: dict[int, list[dict]] = {}
_legends_by_page: dict[int, list[dict]] = {}
for leg in formal_legends:
_legends_by_page.setdefault(int(leg.get("page", 0) or 0), []).append(leg)
for sp, spl in _legends_by_page.items():
narrow_set = ocr_figures._same_page_narrow_caption_column(spl, corpus.page_width)
if len(narrow_set) >= 2:
sidecar_candidates[sp] = narrow_set
bundle_source_legend_ids = ocr_figures._identify_bundle_source_legend_ids(
formal_legends, corpus.raw_assets
)
# Update the return cls(...) to pass:
# sidecar_candidates=sidecar_candidates,
# bundle_source_legend_ids=bundle_source_legend_ids,
```
- [ ] **Step 4: Run tests to verify they pass**
Run: `python -m pytest tests/test_ocr_figure_vnext_corpus.py -v`
Expected: PASS (all existing + new)
- [ ] **Step 5: Run full vnext suite to confirm no regressions**
Run: `python -m pytest tests/test_ocr_figure_vnext_*.py -q`
Expected: All pass
- [ ] **Step 6: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_corpus.py tests/test_ocr_figure_vnext_corpus.py
git commit -m "feat(ocr): populate sidecar_candidates and bundle_source_legend_ids in vnext index"
```
### Task 2: Implement SidecarPass
**Rationale:** The legacy sidecar fallback (ocr_figures.py:4043-4258) handles narrow-caption-column pages where normal spatial matching fails. It detects pages with ≥2 narrow captions, partitions assets into caption bands, and re-matches using band geometry instead of gap/overlap. This pass ports that logic into the proposal-then-commit model.
**Files:**
- Modify: `paperforge/worker/ocr_figure_vnext_passes.py`
- Modify: `tests/test_ocr_figure_vnext_passes.py`
**Interfaces:**
- Consumes:
- `FigurePipelineState` (candidate_index.sidecar_candidates, corpus.raw_assets, state.matches)
- `OwnershipLedger` (try_claim_assets, owner_of_asset)
- `ocr_figures._partition_assets_by_caption_bands(captions, assets, page_height) -> dict[str, list[dict]]`
- `ocr_figures._has_protected_figure_ownership(entry) -> bool`
- `ocr_figures._fallback_eligible_asset_page_ids(...)`
- `ocr_figures._extract_figure_number`, `_extract_figure_namespace`, `_format_figure_id`, `_project_asset_record`, `_cluster_bbox`
- Produces:
- `SidecarPass.run(state) -> PassReport`
- [ ] **Step 1: Write failing sidecar pass tests**
Add to `tests/test_ocr_figure_vnext_passes.py`:
```python
from paperforge.worker.ocr_figure_vnext_passes import SidecarPass
def test_sidecar_pass_matches_narrow_captions_to_asset_bands():
# Page 5 with 3 narrow captions (width < 400 on page_width=1200)
# and 3 assets on the same page. PrimarySamePagePass should miss
# (or mismatch) because narrow captions break spatial matching.
# SidecarPass should partition assets into bands and match each
# caption to its band.
# Assert: report.accepted has matches, state.matches includes
# sidecar entries with settlement_type="sidecar"
...
def test_sidecar_pass_skips_pages_with_fewer_than_two_narrow_captions():
# Page with only 1 narrow caption → no sidecar trigger
# Assert: report.accepted is empty
...
def test_sidecar_pass_does_not_steal_assets_from_same_page_matches():
# Page with 2 narrow captions, but one already matched by
# PrimarySamePagePass with protected ownership.
# Assert: only the unprotected caption gets a sidecar match
...
```
- [ ] **Step 2: Run failing tests**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -k sidecar -v`
Expected: FAIL
- [ ] **Step 3: Implement SidecarPass**
```python
class SidecarPass:
name = "sidecar"
def run(self, state):
from . import ocr_figures
report = PassReport(pass_name=self.name)
for sidecar_page, narrow_set in state.candidate_index.sidecar_candidates.items():
nid_set = {str(cap.get("block_id", "")) for cap in narrow_set}
# Skip if all narrow captions already matched
page_narrow_matched = [
m for m in state.matches
if str(m.get("legend_block_id", "")) in nid_set
]
if len(page_narrow_matched) == len(narrow_set):
continue
page_assets = [
a for a in state.corpus.raw_assets
if _resource_page(a) == sidecar_page
]
if not page_assets:
continue
# Compute page height from blocks
page_height = float(
max(
(b.get("bbox") or [0, 0, 0, 0])[3]
for b in state.corpus.blocks
if _resource_page(b) == sidecar_page
) or 1600
)
band_map = ocr_figures._partition_assets_by_caption_bands(
narrow_set, page_assets, page_height
)
for cap in narrow_set:
lid = str(cap.get("block_id", ""))
if lid in {str(m.get("legend_block_id", "")) for m in page_narrow_matched
if ocr_figures._has_protected_figure_ownership(m)}:
continue
band_assets = band_map.get(lid, [])
if not band_assets:
continue
# Build asset refs and check ownership
asset_refs = [
ResourceRef(kind="asset", page=sidecar_page, block_id=str(a.get("block_id", "")))
for a in band_assets if a.get("block_id")
]
# Skip if any asset already owned
owned = [r for r in asset_refs if state.ledger.owner_of(r) is not None]
if owned:
continue
cap_text = str(cap.get("text", ""))
fig_num = ocr_figures._extract_figure_number(cap_text)
cap_ns = ocr_figures._extract_figure_namespace(cap_text)
legend_ref = ResourceRef(kind="legend", page=sidecar_page, block_id=lid, figure_no=fig_num)
proposal = ClaimProposal(
pass_name=self.name,
figure_no=fig_num,
claim_type="match",
legends=[legend_ref],
assets=asset_refs,
groups=[],
confidence=0.5,
evidence_rank=3,
reason="sidecar_band_partition",
diagnostics={
"evidence": ["narrow_caption_column", "sidecar_fallback", "caption_band_partition"],
"sidecar_page": sidecar_page,
},
)
conflict = state.ledger.try_claim_assets(asset_refs, owner=legend_ref, reason=proposal.reason)
if conflict is not None:
report.conflicts.append(conflict)
report.rejected.append(proposal)
continue
fig_id = (
ocr_figures._format_figure_id(cap_ns, fig_num)
if fig_num
else f"figure_sidecar_{len(state.matches):03d}"
)
matched_assets = [ocr_figures._project_asset_record(a) for a in band_assets]
match_record = {
"figure_id": fig_id,
"figure_namespace": cap_ns,
"figure_number": fig_num,
"legend_block_id": lid,
"page": sidecar_page,
"text": cap_text,
"matched_assets": matched_assets,
"asset_block_ids": sorted(str(r.block_id) for r in asset_refs),
"settlement_type": "sidecar",
"confidence": 0.5,
"match_score": {"score": 0.5, "decision": "matched", "evidence": ["sidecar_fallback"]},
"flags": ["sidecar_match"],
"bridge_block_ids": [],
}
if len(band_assets) > 1:
match_record["cluster_bbox"] = ocr_figures._cluster_bbox(
[a.get("bbox", [0, 0, 0, 0]) for a in band_assets]
)
state.accept_match(proposal, match_record)
report.accepted.append(proposal)
return report
```
- [ ] **Step 4: Run sidecar tests**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -k sidecar -v`
Expected: PASS
- [ ] **Step 5: Run full passes test file**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -v`
Expected: All pass
- [ ] **Step 6: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_passes.py tests/test_ocr_figure_vnext_passes.py
git commit -m "feat(ocr): add vnext SidecarPass"
```
### Task 3: Implement LegendBundlePass
**Rationale:** The legacy legend-bundle fallback (ocr_figures.py:4260-4381) handles pages with ≥3 figure captions and zero same-page assets — it matches captions 1:1 by page order to subsequent pages that each hold unclaimed assets. This is common in preproof/early-view layouts.
**Files:**
- Modify: `paperforge/worker/ocr_figure_vnext_passes.py`
- Modify: `tests/test_ocr_figure_vnext_passes.py`
**Interfaces:**
- Consumes:
- `FigurePipelineState` (candidate_index.deduped_legends, candidate_index.bundle_source_legend_ids, corpus.raw_assets, corpus.blocks, state.matches, ledger)
- `ocr_figures._extract_figure_number`, `_extract_figure_namespace`, `_format_figure_id`, `_project_asset_record`, `score_figure_caption`
- Produces:
- `LegendBundlePass.run(state) -> PassReport`
- [ ] **Step 1: Write failing legend bundle tests**
```python
from paperforge.worker.ocr_figure_vnext_passes import LegendBundlePass
def test_legend_bundle_pass_matches_captions_to_subsequent_asset_pages():
# Page 3 with 3 captions (Figure 1, 2, 3) and zero assets.
# Pages 4, 5, 6 each have 1 unclaimed asset and no body/table blocks.
# Assert: 3 matches created, each caption → one asset page in order.
# Assert: settlement_type="legend_bundle", flags includes "legend_bundle_match"
...
def test_legend_bundle_pass_requires_minimum_three_captions():
# Page with only 2 captions → no bundle trigger
# Assert: report.accepted is empty
...
def test_legend_bundle_pass_skips_when_intervening_pages_have_body_text():
# Captions on page 3, assets on page 5, but page 4 has body_paragraph blocks
# Assert: no bundle match (intervening body breaks the chain)
...
```
- [ ] **Step 2: Run failing tests**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -k legend_bundle -v`
Expected: FAIL
- [ ] **Step 3: Implement LegendBundlePass**
Port the legacy logic (ocr_figures.py:4260-4381) into the proposal-then-commit model. Key structure:
```python
class LegendBundlePass:
name = "legend_bundle"
def run(self, state):
from . import ocr_figures
report = PassReport(pass_name=self.name)
matched_legend_ids = {str(m.get("legend_block_id", "")) for m in state.matches}
unmatched_legends = [
leg for leg in state.candidate_index.deduped_legends
if str(leg.get("block_id", "")) not in matched_legend_ids
]
# Compute unmatched assets (not owned by ledger)
unmatched_assets = [
a for a in state.corpus.raw_assets
if (_resource_page(a) is not None
and state.ledger.owner_of_asset(
page=_resource_page(a), block_id=a.get("block_id")
) is None)
]
if not unmatched_legends or not unmatched_assets:
return report
# Group unmatched numbered legends by page
page_captions: dict[int, list[dict]] = {}
for leg in unmatched_legends:
cp = _resource_page(leg)
if cp is None:
continue
if ocr_figures._extract_figure_number(str(leg.get("text", ""))) is not None:
page_captions.setdefault(cp, []).append(leg)
_NON_PURE_ROLES = {
"body_paragraph", "section_heading", "subsection_heading",
"table_caption", "table_asset", "table_html",
"backmatter_heading", "backmatter_body", "reference_item",
}
for cp, caps in sorted(page_captions.items()):
has_bundle_source = any(
str(cap.get("block_id", "")) in state.candidate_index.bundle_source_legend_ids
for cap in caps
)
if len(caps) < 3 and not has_bundle_source:
continue
page_has_assets = any(_resource_page(a) == cp for a in unmatched_assets)
if page_has_assets:
continue
caps_sorted = sorted(caps, key=lambda b: (b.get("bbox") or [0, 0, 0, 0])[1])
# Collect subsequent pages with unclaimed assets
asset_pages: dict[int, list[dict]] = {}
for ast in unmatched_assets:
ap = _resource_page(ast)
if ap is None or ap <= cp:
continue
if state.ledger.owner_of_asset(page=ap, block_id=ast.get("block_id")) is not None:
continue
asset_pages.setdefault(ap, []).append(ast)
page_order = sorted(asset_pages.keys())
if not page_order:
continue
# Check no body/table on intervening pages
intervening_pages = set(range(cp + 1, page_order[0]))
intervening_body = any(
_resource_page(b) in intervening_pages
and b.get("role", "") in _NON_PURE_ROLES
for b in state.corpus.blocks
)
if intervening_body:
continue
valid_pages = []
for ap in page_order:
page_has_body = any(
_resource_page(b) == ap and b.get("role", "") in _NON_PURE_ROLES
for b in state.corpus.blocks
)
if not page_has_body:
valid_pages.append(ap)
if len(valid_pages) < len(caps_sorted):
caps_sorted = caps_sorted[:len(valid_pages)]
if not valid_pages:
continue
for idx, cap in enumerate(caps_sorted):
if idx >= len(valid_pages):
break
ap = valid_pages[idx]
page_assets = asset_pages[ap]
if not page_assets:
continue
fn = ocr_figures._extract_figure_number(str(cap.get("text", "")))
cap_ns = ocr_figures._extract_figure_namespace(str(cap.get("text", "")))
fig_id = ocr_figures._format_figure_id(cap_ns, fn)
cap_ref = ResourceRef(kind="legend", page=cp, block_id=str(cap.get("block_id", "")), figure_no=fn)
asset_refs = [
ResourceRef(kind="asset", page=ap, block_id=str(a.get("block_id", "")))
for a in page_assets if a.get("block_id")
]
proposal = ClaimProposal(
pass_name=self.name,
figure_no=fn,
claim_type="match",
legends=[cap_ref],
assets=asset_refs,
groups=[],
confidence=0.3,
evidence_rank=4,
reason="legend_bundle_fallback",
diagnostics={
"evidence": ["legend_bundle_fallback", "page_order_match"],
"legend_page": cp,
"asset_page": ap,
},
)
conflict = state.ledger.try_claim_assets(asset_refs, owner=cap_ref, reason=proposal.reason)
if conflict is not None:
report.conflicts.append(conflict)
report.rejected.append(proposal)
continue
matched_assets = [ocr_figures._project_asset_record(a) for a in page_assets]
match_record = {
"figure_id": fig_id,
"figure_namespace": cap_ns,
"figure_number": fn,
"legend_block_id": str(cap.get("block_id", "")),
"page": ap,
"text": str(cap.get("text", "")),
"matched_assets": matched_assets,
"asset_block_ids": sorted(str(r.block_id) for r in asset_refs),
"settlement_type": "legend_bundle",
"confidence": 0.3,
"match_score": {"score": 0.3, "decision": "matched", "evidence": ["legend_bundle_fallback"]},
"flags": ["legend_bundle_match"],
"legend_page": cp,
"asset_pages": [ap],
"bridge_block_ids": [],
}
state.accept_match(proposal, match_record)
report.accepted.append(proposal)
return report
```
- [ ] **Step 4: Run legend bundle tests**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -k legend_bundle -v`
Expected: PASS
- [ ] **Step 5: Run full passes test file**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -v`
Expected: All pass
- [ ] **Step 6: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_passes.py tests/test_ocr_figure_vnext_passes.py
git commit -m "feat(ocr): add vnext LegendBundlePass"
```
### Task 4: Implement LocatorBridgePass
**Rationale:** The legacy locator bridge (ocr_figures.py:4384-4578) connects three components when a paper uses "See legend on previous page" locator captions: (1) full legend on the previous page, (2) visual group on the locator's page, (3) the locator caption itself. This pass ports that three-way bridge into the proposal model.
**Files:**
- Modify: `paperforge/worker/ocr_figure_vnext_passes.py`
- Modify: `tests/test_ocr_figure_vnext_passes.py`
**Interfaces:**
- Consumes:
- `FigurePipelineState` (candidate_index.locator_candidates, candidate_index.rejected_legends, candidate_index.deduped_legends, corpus.raw_assets, corpus.blocks, state.matches, ledger)
- `ocr_figures._is_previous_page_legend_locator`, `_extract_figure_number`, `_extract_figure_namespace`, `_format_figure_id`, `_project_asset_record`, `_cluster_bbox`, `_is_tight_asset_cluster`, `score_figure_caption`
- Produces:
- `LocatorBridgePass.run(state) -> PassReport`
- [ ] **Step 1: Write failing locator bridge tests**
```python
from paperforge.worker.ocr_figure_vnext_passes import LocatorBridgePass
def test_locator_bridge_connects_full_legend_to_visual_group():
# Page 4: full legend "Figure 5. ..." (unmatched, ≥60 chars)
# Page 5: locator "Fig. 5 (See legend on previous page.)" + assets above locator
# Assert: match created with:
# - legend_block_id = full legend's block_id
# - locator_block_id in bridge_block_ids
# - settlement_type = "previous_page_legend_locator"
# - flags includes "previous_page_locator_match"
...
def test_locator_bridge_skips_when_no_full_legend_on_previous_page():
# Locator on page 5, but no matching figure number on page 4
# Assert: report.accepted is empty
...
def test_locator_bridge_skips_when_assets_already_owned():
# Locator on page 5, full legend on page 4, but assets on page 5
# already claimed by same-page match
# Assert: no bridge match (assets protected)
...
```
- [ ] **Step 2: Run failing tests**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -k locator_bridge -v`
Expected: FAIL
- [ ] **Step 3: Implement LocatorBridgePass**
Port the legacy logic (ocr_figures.py:4384-4578) into the proposal model. Key structure:
```python
class LocatorBridgePass:
name = "locator_bridge"
def run(self, state):
from . import ocr_figures
report = PassReport(pass_name=self.name)
locators = state.candidate_index.locator_candidates
if not locators:
return report
matched_legend_ids = {str(m.get("legend_block_id", "")) for m in state.matches}
unmatched_legends = [
leg for leg in state.candidate_index.deduped_legends
if str(leg.get("block_id", "")) not in matched_legend_ids
]
# Also scan rejected_legends for full legends misclassified as body_paragraph
# (legacy pattern: ocr_figures.py:4402-4415)
_unmatched_by_number: dict[tuple[str, int], list[dict]] = {}
for leg in unmatched_legends:
fn = ocr_figures._extract_figure_number(str(leg.get("text", "")))
if fn is None:
continue
ns = ocr_figures._extract_figure_namespace(str(leg.get("text", "")))
_unmatched_by_number.setdefault((ns, fn), []).append(leg)
for leg in state.candidate_index.rejected_legends:
fn = ocr_figures._extract_figure_number(str(leg.get("text", "")))
if fn is None:
continue
ns = ocr_figures._extract_figure_namespace(str(leg.get("text", "")))
key = (ns, fn)
if key in _unmatched_by_number:
continue
style = str(leg.get("style_family") or "")
zone = str(leg.get("zone") or "")
if style == "legend_like" and zone in ("display_zone", ""):
_unmatched_by_number.setdefault(key, []).append(leg)
for locator in locators:
locator_text = str(locator.get("text", "") or "")
fn = ocr_figures._extract_figure_number(locator_text)
if fn is None:
continue
ns = ocr_figures._extract_figure_namespace(locator_text)
locator_page = _resource_page(locator)
if locator_page is None or locator_page <= 1:
continue
prev_page = locator_page - 1
# Find full legend on previous page with same figure_number
full_legends = _unmatched_by_number.get((ns, fn), [])
full_legend = None
for leg in full_legends:
lp = _resource_page(leg)
if lp == prev_page:
leg_text = str(leg.get("text", "") or "")
if len(leg_text) >= 60 and not ocr_figures._is_previous_page_legend_locator(leg):
full_legend = leg
break
if full_legend is None:
continue
# Find visual group above locator on locator's page
locator_bbox = locator.get("bbox") or [0, 0, 0, 0]
locator_top = locator_bbox[1] if len(locator_bbox) >= 4 else 0
# Try candidate groups first
best_group_assets: list[dict] = []
for g in state.candidate_index.candidate_groups:
gp = _resource_page(g)
if gp != locator_page:
continue
g_bbox = g.get("cluster_bbox") or [0, 0, 0, 0]
if len(g_bbox) < 4 or g_bbox[3] > locator_top:
continue
g_asset_ids = g.get("asset_block_ids", [])
g_unowned_ids = [
bid for bid in g_asset_ids
if state.ledger.owner_of_asset(page=locator_page, block_id=bid) is None
]
if not g_unowned_ids:
continue
group_assets = [
a for a in state.corpus.raw_assets
if _resource_page(a) == locator_page
and str(a.get("block_id", "")) in g_unowned_ids
]
if group_assets:
best_group_assets = group_assets
break
# Fallback: tight cluster above locator
if not best_group_assets:
page_assets = [
a for a in state.corpus.raw_assets
if _resource_page(a) == locator_page
and state.ledger.owner_of_asset(
page=locator_page, block_id=a.get("block_id")
) is None
]
above = [a for a in page_assets if (a.get("bbox") or [0,0,0,0])[3] <= locator_top]
if above and ocr_figures._is_tight_asset_cluster(above, locator_top):
best_group_assets = above
if not best_group_assets:
continue
# Build and commit the bridge match
full_ref = ResourceRef(
kind="legend", page=prev_page,
block_id=str(full_legend.get("block_id", "")), figure_no=fn
)
asset_refs = [
ResourceRef(kind="asset", page=locator_page, block_id=str(a.get("block_id", "")))
for a in best_group_assets if a.get("block_id")
]
proposal = ClaimProposal(
pass_name=self.name,
figure_no=fn,
claim_type="match",
legends=[full_ref],
assets=asset_refs,
groups=[],
confidence=0.5,
evidence_rank=3,
reason="previous_page_legend_locator",
diagnostics={
"evidence": ["previous_page_locator_bridge", "explicit_previous_page_locator"],
"locator_block_id": str(locator.get("block_id", "")),
"locator_page": locator_page,
"full_legend_page": prev_page,
},
)
conflict = state.ledger.try_claim_assets(asset_refs, owner=full_ref, reason=proposal.reason)
if conflict is not None:
report.conflicts.append(conflict)
report.rejected.append(proposal)
continue
fig_id = ocr_figures._format_figure_id(ns, fn)
consumed = [ocr_figures._project_asset_record(a) for a in best_group_assets]
asset_bboxes = [a.get("bbox") or [0, 0, 0, 0] for a in best_group_assets]
valid_bboxes = [b for b in asset_bboxes if len(b) >= 4 and b[2] > b[0] and b[3] > b[1]]
cluster_bbox = ocr_figures._cluster_bbox(valid_bboxes) if valid_bboxes else [0, 0, 0, 0]
match_record = {
"figure_id": fig_id,
"figure_namespace": ns,
"figure_number": fn,
"legend_block_id": str(full_legend.get("block_id", "")),
"legend_page": prev_page,
"text": str(full_legend.get("text", "")),
"matched_assets": consumed,
"asset_block_ids": sorted(str(r.block_id) for r in asset_refs),
"cluster_bbox": cluster_bbox,
"group_type": "previous_page_locator_bridge",
"settlement_type": "previous_page_legend_locator",
"confidence": 0.5,
"match_score": {"score": 0.5, "decision": "matched", "evidence": ["previous_page_locator_bridge"]},
"flags": ["previous_page_locator_match"],
"page": locator_page,
"locator_block_id": str(locator.get("block_id", "")),
"locator_page": locator_page,
"asset_pages": [locator_page],
"bridge_block_ids": [str(locator.get("block_id", ""))],
}
state.accept_match(proposal, match_record)
report.accepted.append(proposal)
return report
```
- [ ] **Step 4: Run locator bridge tests**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -k locator_bridge -v`
Expected: PASS
- [ ] **Step 5: Run full passes test file**
Run: `python -m pytest tests/test_ocr_figure_vnext_passes.py -v`
Expected: All pass
- [ ] **Step 6: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_passes.py tests/test_ocr_figure_vnext_passes.py
git commit -m "feat(ocr): add vnext LocatorBridgePass"
```
### Task 5: Wire special fallback passes into orchestrator + portable fixture
**Rationale:** The orchestrator must run all passes in the correct order per spec §7.3: Layer 1 (same-page, sidecar, locator) then Layer 2 (cross-page reservation/settlement, legend bundle). This task wires the three new passes and adds a third real-paper fixture.
**Files:**
- Modify: `paperforge/worker/ocr_figures.py`
- Create: `tests/fixtures/ocr_vnext_real_papers/<KEY>/blocks.structured.jsonl` (a sidecar or locator-bridge paper from the OCR vault)
- Modify: `tests/test_ocr_figure_vnext_real_papers.py`
**Pass ordering (spec §7.3):**
1. `PrimarySamePagePass` (Layer 1)
2. `SidecarPass` (Layer 1)
3. `LocatorBridgePass` (Layer 1)
4. `CrossPageReservationPass` (Layer 2)
5. `CrossPageSettlementPass` (Layer 2)
6. `LegendBundlePass` (Layer 2)
- [ ] **Step 1: Update the orchestrator**
```python
# In build_figure_inventory_vnext:
from .ocr_figure_vnext_passes import (
CrossPageReservationPass,
CrossPageSettlementPass,
LegendBundlePass,
LocatorBridgePass,
PrimarySamePagePass,
SidecarPass,
_resource_page,
)
...
reports = []
for pass_cls in (
PrimarySamePagePass,
SidecarPass,
LocatorBridgePass,
CrossPageReservationPass,
CrossPageSettlementPass,
LegendBundlePass,
):
reports.append(pass_cls().run(state))
```
- [ ] **Step 2: Add a third real-paper fixture**
Find a paper from `D:/L/OB/Literature-hub/System/PaperForge/ocr/` that triggers sidecar or locator-bridge (check for narrow caption columns or "See legend on previous page" text). Copy its `blocks.structured.jsonl` into `tests/fixtures/ocr_vnext_real_papers/<KEY>/`.
If no suitable paper is found, use `2HEUD5P9` (already in the fixture set) and add assertions for the new pass reports instead.
- [ ] **Step 3: Add fixture-backed test**
```python
def test_real_paper_special_fallbacks_reports_diff_shape():
blocks_path = Path("tests/fixtures/ocr_vnext_real_papers/<KEY>/blocks.structured.jsonl")
diff = compare_blocks_file(blocks_path)
assert diff["paper"] == "<KEY>"
assert "legacy_matched_count" in diff
assert "vnext_matched_count" in diff
```
- [ ] **Step 4: Run all vnext tests**
Run: `python -m pytest tests/test_ocr_figure_vnext_*.py -q`
Expected: All pass
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ocr_figures.py tests/fixtures/ocr_vnext_real_papers/<KEY>/ tests/test_ocr_figure_vnext_real_papers.py
git commit -m "feat(ocr): wire vnext special fallback passes into orchestrator"
```
## Self-Review
- Spec coverage: This phase implements spec §7.3 Layer 1 (sidecar, locator bridge) and Layer 2 (legend bundle) special fallbacks. It does not implement group-aware sequential, classic sequential, composite-parent settlement, completeness/accounting, or cutover — those are left to Phase 4+.
- Helper reuse: All three passes reuse existing legacy helper functions via lazy import. No helper duplication.
- Layer ordering: Passes are wired in spec §7.3 layer order. Lower layers cannot steal from higher layers because `OwnershipLedger.try_claim_assets` rejects already-owned assets.
- Placeholder scan: No `TBD` or `TODO` in task steps. The `<KEY>` placeholder in Task 5 is resolved during implementation by selecting a real paper from the OCR vault.
## Execution Handoff
Plan complete and saved to `docs/superpowers/plans/2026-07-03-figure-pipeline-vnext-phase3-special-fallbacks.md`. Two execution options:
1. **Subagent-Driven (recommended)** - One implementer per task, review gate after each
2. **Inline Execution** - Batch execution with checkpoints
Which approach?

View file

@ -0,0 +1,280 @@
# Figure Pipeline VNext Phase 4 — Low-Certainty Fallbacks + Composite Parent + Accounting Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Implement the remaining passes from spec §7.3: `CompositeParentPass` (Layer 1, but runs during same-page scan), `GroupSequentialPass` (Layer 2), `ClassicSequentialPass` (Layer 3), unresolved cluster consolidation, and `FinalAccountingPass`. This completes all spec-defined passes and prepares the pipeline for cutover evaluation.
**Architecture:** Build on the completed Phase 03 vnext seams. Each pass follows the established proposal-then-commit pattern. Composite parent candidates are pre-built in the corpus/candidate index (they're structural, not matching decisions). The sequential passes consume only untouched or explicitly eligible resources per spec §7.3 Rule. The accounting pass verifies completeness and emits invariant violations.
**Tech Stack:** Python 3, pytest, existing OCR helpers in `paperforge/worker/ocr_figures.py`, vnext modules under `paperforge/worker/`
## Global Constraints
- Build on branch/worktree `feat/figure-pipeline-vnext`; do not edit the main checkout.
- Preserve the external interface `build_figure_inventory(structured_blocks, page_width=1200) -> FigureInventory`.
- Legacy implementation remains the immutable baseline.
- Keep `ResourceRef` as the only ownership key.
- Lower layers cannot steal already-claimed resources from higher layers (spec §7.3 Rule 1).
- Classic sequential consumes only untouched or explicitly eligible resources (spec §7.3 Rule 3).
- Disabled passes must emit no proposals.
- Final arbitration priority is independent of migration order.
- New pass reports must remain JSON-safe.
- Reuse existing legacy helper functions via lazy import — do not duplicate them.
- Add only the tests directly covering this phase.
- Do not implement final cutover in this plan.
---
### Task 1: Add composite_parent_candidates to FigureCandidateIndex
**Rationale:** The legacy pipeline builds composite parent candidates (`_build_composite_parent_figure_groups_visual_only` + `_build_dense_composite_parent_candidates`) as structural pre-computation before matching. The vnext `FigureCandidateIndex` needs these candidates available so `CompositeParentPass` (Task 2) can consume them. This is a structural computation, not a matching decision — it belongs in the candidate index.
**Files:**
- Modify: `paperforge/worker/ocr_figure_vnext_corpus.py`
- Modify: `tests/test_ocr_figure_vnext_corpus.py`
**Interfaces:**
- Consumes:
- `ocr_figures._build_composite_parent_figure_groups_visual_only(atomic_groups, assets, structured_blocks, page_width) -> list[dict]`
- Produces:
- `FigureCandidateIndex.composite_parent_candidates: list[dict]` (new field)
- [ ] **Step 1: Write failing test**
Add to `tests/test_ocr_figure_vnext_corpus.py`:
```python
def test_candidate_index_populates_composite_parent_candidates():
# 4 assets on page 1 arranged in a 2x2 grid (two rows, two columns)
# Each pair shares horizontal alignment and vertical adjacency
# Assert: index.composite_parent_candidates is non-empty
...
```
- [ ] **Step 2: Run test to verify it fails**
Run: `python -m pytest tests/test_ocr_figure_vnext_corpus.py -k composite_parent -v`
Expected: FAIL (field doesn't exist)
- [ ] **Step 3: Add composite_parent_candidates to FigureCandidateIndex**
Add `composite_parent_candidates: list[dict]` field to the dataclass. In `from_corpus`, call:
```python
composite_parent_candidates = ocr_figures._build_composite_parent_figure_groups_visual_only(
candidate_groups,
corpus.raw_assets,
corpus.blocks,
corpus.page_width,
)
```
Pass it in the `cls(...)` return.
- [ ] **Step 4: Run test to verify it passes**
Run: `python -m pytest tests/test_ocr_figure_vnext_corpus.py -v`
Expected: All pass
- [ ] **Step 5: Run full vnext suite**
Run: `python -m pytest tests/test_ocr_figure_vnext_*.py -q`
Expected: All pass
- [ ] **Step 6: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_corpus.py tests/test_ocr_figure_vnext_corpus.py
git commit -m "feat(ocr): add composite_parent_candidates to vnext candidate index"
```
### Task 2: Implement CompositeParentPass
**Rationale:** The legacy composite parent arbitration (ocr_figures.py:3480-3598) runs as part of the same-page legend scan: for each numbered legend, if a composite parent candidate exists on the same page with sufficient confidence and ≥2 child groups, the legend claims all child group assets as a single multi-panel figure. This pass ports that into the proposal model.
**Files:**
- Create: `paperforge/worker/ocr_figure_vnext_composite_pass.py`
- Create: `tests/test_ocr_figure_vnext_composite_pass.py`
**Key logic (port from legacy ocr_figures.py:3480-3598):**
1. For each unmatched numbered legend:
- Find composite_parent_candidates on the same page, not yet consumed
- Sort by parent_confidence (highest first)
- Check: confidence ≥ 0.60, no competing numbered legends on same page (or band-scoped), ≥2 effective child groups
- Claim all child group assets via `try_claim_assets`
- Create match with `settlement_type="composite_parent"`, `flags=["composite_parent_match"]`
2. `confidence` = parent_confidence, `evidence_rank=1` (high certainty, same-page)
- [ ] **Step 1: Write failing tests** (3 tests: happy path multi-panel match, competing caption blocks it, insufficient child groups skip)
- [ ] **Step 2: Run failing tests**
- [ ] **Step 3: Implement CompositeParentPass** (standalone file, local `_resource_page`, lazy import `ocr_figures`)
- [ ] **Step 4: Run tests**
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_composite_pass.py tests/test_ocr_figure_vnext_composite_pass.py
git commit -m "feat(ocr): add vnext CompositeParentPass"
```
### Task 3: Implement GroupSequentialPass
**Rationale:** The legacy group-aware sequential fallback (ocr_figures.py:4589-4716) consumes unmatched `distance_cluster` and `single_asset` groups that no same-page legend claimed. It matches unmatched numbered legends to these groups preferring same-page → next-page → previous-page order. This is spec §7.3 Layer 2.
**Files:**
- Create: `paperforge/worker/ocr_figure_vnext_group_seq_pass.py`
- Create: `tests/test_ocr_figure_vnext_group_seq_pass.py`
**Key logic (port from legacy ocr_figures.py:4589-4716):**
1. Collect unmatched groups: `candidate_groups` not yet claimed, type in `{distance_cluster, single_asset}`, no assets owned
2. Sort by page, then y-position
3. For each unmatched numbered legend:
- Find same-page groups, score with `_score_legend_to_group`, take best if score ≥ 0.5
- If no same-page match, take first next-page group
- If no next-page, check previous-page with `_allow_previous_page_sequential_match`
- Claim group assets via `try_claim_assets`
- `confidence=0.45`, `evidence_rank=5`, `settlement_type="group_sequential"`, `flags=["group_sequential_match"]`
- [ ] **Step 1: Write failing tests** (3 tests: same-page group match, next-page fallback, previous-page with `_allow_previous_page_sequential_match` guard)
- [ ] **Step 2: Run failing tests**
- [ ] **Step 3: Implement GroupSequentialPass**
- [ ] **Step 4: Run tests**
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_group_seq_pass.py tests/test_ocr_figure_vnext_group_seq_pass.py
git commit -m "feat(ocr): add vnext GroupSequentialPass"
```
### Task 4: Implement ClassicSequentialPass + UnresolvedClusterConsolidation
**Rationale:** The legacy classic sequential fallback (ocr_figures.py:4720-4862) is the last-resort matcher: it matches unmatched numbered captions to remaining ungrouped assets in reading order. After sequential matching, remaining unmatched assets on pages with rejected legends form unresolved clusters. This is spec §7.3 Layer 3.
**Files:**
- Create: `paperforge/worker/ocr_figure_vnext_classic_seq_pass.py`
- Create: `tests/test_ocr_figure_vnext_classic_seq_pass.py`
**Key logic:**
**ClassicSequentialPass** (port from ocr_figures.py:4720-4830):
1. Compute ungrouped unmatched assets (not in any candidate_group, not owned)
2. Sort captions and assets by (page, y-position)
3. For each unmatched numbered caption:
- Scan for previous-page asset (with `_allow_previous_page_sequential_match` guard) or future-page asset
- Claim via `try_claim_assets`
- `confidence=0.35`, `evidence_rank=6`, `settlement_type="sequential"`, `flags=["sequential_match"]`
**UnresolvedClusterConsolidation** (port from ocr_figures.py:4831-4862):
After classic sequential, build unresolved clusters from remaining unmatched assets on pages with rejected legends using `_media_clusters`. Store in `state.unresolved` (new field on `FigurePipelineState`).
- [ ] **Step 1: Write failing tests** (3 tests: classic sequential match in reading order, previous-page guard, unresolved cluster formation from rejected-legend page assets)
- [ ] **Step 2: Run failing tests**
- [ ] **Step 3: Implement ClassicSequentialPass + unresolved cluster logic**
Add `unresolved: list[dict]` field to `FigurePipelineState` if not already present (check first — it exists in the dataclass but is currently empty). The pass appends to `state.unresolved`.
- [ ] **Step 4: Run tests**
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_classic_seq_pass.py tests/test_ocr_figure_vnext_classic_seq_pass.py paperforge/worker/ocr_figure_vnext_state.py
git commit -m "feat(ocr): add vnext ClassicSequentialPass with unresolved cluster consolidation"
```
### Task 5: Implement FinalAccountingPass
**Rationale:** The legacy pipeline computes `figure_legend_completeness` (ocr_figures.py:4994-5126) which verifies every numbered formal legend lands in an explicit outcome bucket (matched, held, unmatched, gap). Spec §7.4 requires invariant checks after each pass. This pass runs last, computes completeness, and emits invariant violations.
**Files:**
- Create: `paperforge/worker/ocr_figure_vnext_accounting_pass.py`
- Create: `tests/test_ocr_figure_vnext_accounting_pass.py`
**Key logic:**
1. Collect all numbered legend block_ids that entered the pipeline (from `candidate_index.deduped_legends` with figure numbers)
2. Classify each into: matched (in `state.matches`), unmatched (in unmatched pool), held, or gap (silently dropped)
3. Check invariants (spec §7.4):
- Every matched figure's assets are owned in the ledger
- No unmatched legend appears in `state.matches`
- `accounted_for == total - gap_count`
4. Populate `state.completeness` dict with total, accounted_for, gap_count, details
5. Emit invariant violations to `report.invariant_errors`
- [ ] **Step 1: Write failing tests** (3 tests: all legends matched → gap_count=0, one legend dropped → gap_count=1, invariant violation when asset ownership mismatch)
- [ ] **Step 2: Run failing tests**
- [ ] **Step 3: Implement FinalAccountingPass**
- [ ] **Step 4: Run tests**
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ocr_figure_vnext_accounting_pass.py tests/test_ocr_figure_vnext_accounting_pass.py
git commit -m "feat(ocr): add vnext FinalAccountingPass with invariant checks"
```
### Task 6: Wire all passes into orchestrator + populate output fields
**Rationale:** Wire the 4 new passes into the orchestrator in spec §7.3 layer order, and populate the output dict fields that were previously stubs (`unresolved_clusters`, `completeness` with real data).
**Pass ordering (spec §7.3, final):**
1. `PrimarySamePagePass` (Layer 1)
2. `CompositeParentPass` (Layer 1 — runs after same-page, before sidecar, because it's same-page structural)
3. `SidecarPass` (Layer 1)
4. `LocatorBridgePass` (Layer 1)
5. `CrossPageReservationPass` (Layer 2)
6. `CrossPageSettlementPass` (Layer 2)
7. `LegendBundlePass` (Layer 2)
8. `GroupSequentialPass` (Layer 2)
9. `ClassicSequentialPass` (Layer 3)
10. `FinalAccountingPass` (post-processing)
**Files:**
- Modify: `paperforge/worker/ocr_figures.py`
**Changes to orchestrator:**
- Import all 4 new pass classes
- Add to pass ordering tuple
- Update output dict:
- `"unresolved_clusters": state.unresolved` (instead of `[]`)
- `"completeness"`: use `state.completeness` if populated by FinalAccountingPass
- [ ] **Step 1: Update orchestrator**
- [ ] **Step 2: Run all vnext tests**
Run: `python -m pytest tests/test_ocr_figure_vnext_*.py -q`
Expected: All pass
- [ ] **Step 3: Run full figure test suite**
Run: `python -m pytest tests/test_ocr_figures.py tests/test_ocr_render.py -q`
Expected: No regressions (legacy path unchanged, wrapper still calls legacy)
- [ ] **Step 4: Commit**
```bash
git add paperforge/worker/ocr_figures.py
git commit -m "feat(ocr): wire vnext Phase 4 passes into orchestrator with accounting"
```
## Self-Review
- Spec coverage: This phase completes all 10 spec-defined passes (§6.2 E). It implements spec §7.3 Layer 2 (group sequential), Layer 3 (classic sequential, unresolved consolidation), composite parent (structural same-page), and §7.4 invariants/accounting. Only final cutover (§8.3, §9) remains.
- Helper reuse: All passes reuse legacy helpers via lazy import (`_build_composite_parent_figure_groups_visual_only`, `_score_legend_to_group`, `_allow_previous_page_sequential_match`, `_media_clusters`, `_filter_figure_assets`, `_grouped_asset_page_ids`).
- Layer ordering: CompositeParentPass runs at Layer 1 (same-page structural) before sidecar. GroupSequential at Layer 2 after legend bundle. ClassicSequential at Layer 3 (lowest certainty, only untouched resources).
- Invariant checks: FinalAccountingPass implements spec §7.4 invariant examples.
- Placeholder scan: No `TBD` or `TODO` in task steps.
## Execution Handoff
Plan complete. Recommended: Subagent-Driven execution with Tasks 2-5 parallelizable (isolated files), Task 1 as prerequisite, Task 6 as final wiring.

View file

@ -0,0 +1,276 @@
# Figure Pipeline VNext Phase 5 — Cutover Evaluation and Default-Switch Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Decide, with evidence, whether `build_figure_inventory(...)` can switch from legacy to vnext by default. This phase does **not** assume cutover is already safe. It first closes the comparison and gate-evaluation gaps from spec §8.3 and §9, then performs the wrapper switch only if all gates pass.
**Current assessment (2026-07-03):**
- Production path is still safe: `build_figure_inventory(...)` still calls legacy, so **master has zero behavioral regression today**.
- VNext path is much more auditable and structurally safer, but **not yet proven cutover-ready**.
- Repo-fixture evidence so far:
- `2HEUD5P9`: parity (`legacy_matched=7`, `vnext_matched=7`, same consumed assets)
- `YGH7VEX6`: parity on matched count and consumed assets (`12/12`, same consumed assets)
- `DWQQK2YB`: vnext matches **more** (`legacy=4`, `vnext=6`) while consuming the **same** asset set, which is a plausible improvement but still requires manual adjudication before cutover
- Spec-required comparison corpus is still incomplete: we do not yet have explicit coverage/review for all categories in spec §8.3.
**Architecture:** Treat cutover as a gated release process, not a code flip. The work splits into: (1) comparison harness expansion, (2) representative-corpus curation, (3) diff review + regression triage, (4) default wrapper switch if and only if gates pass.
**Tech Stack:** Python 3, pytest, existing figure/render/rebuild tests, dev scripts under `scripts/dev/`, project records under `project/current/`
## Global Constraints
- Build on branch/worktree `feat/figure-pipeline-vnext`; do not edit the main checkout.
- Preserve the external interface `build_figure_inventory(structured_blocks, page_width=1200) -> FigureInventory`.
- Legacy implementation remains the immutable baseline until the final cutover task.
- No silent scope shrinkage: every spec §8.3 corpus category must be either represented or explicitly waived with written rationale.
- Differences between legacy and vnext must be reviewed paper-by-paper before cutover.
- VNext may differ from legacy only if the difference is explicitly recorded and judged strictly better or equivalent.
- No figure card may consume the same asset twice.
- No owned asset may remain in unresolved clusters.
- Add only the tests and scripts directly needed for cutover evaluation.
---
### Task 1: Expand the comparison harness to cover cutover gates
**Rationale:** The current compare script only reports matched count, unresolved count, unmatched legend count, and consumed asset ids. Spec §8.3 / §9 needs more: completeness totals, conflict counts, pass summaries, and render-facing diff surfaces.
**Files:**
- Modify: `scripts/dev/compare_figure_inventory_legacy_vs_vnext.py`
- Modify: `tests/test_ocr_figure_vnext_compare.py`
**Add to compare output:**
- `legacy_conflict_count`
- `vnext_conflict_count`
- `legacy_completeness`
- `vnext_completeness`
- `legacy_figure_ids`
- `vnext_figure_ids`
- `legacy_render_card_count`
- `vnext_render_card_count`
- `vnext_pass_names`
**Notes:**
- `legacy_conflict_count` can be approximated as `0` if legacy does not expose an ownership-conflict bucket.
- `legacy_completeness` should read `figure_legend_completeness` if present.
- `vnext_completeness` should read `completeness`.
- Render-card diff stays structural in this phase: compare normalized matched-figure card payloads, not screenshots.
- [ ] **Step 1: Add failing compare tests**
- [ ] **Step 2: Expand `compare_inventories(...)` and `compare_blocks_file(...)`**
- [ ] **Step 3: Run compare tests**
- [ ] **Step 4: Commit**
```bash
git add scripts/dev/compare_figure_inventory_legacy_vs_vnext.py tests/test_ocr_figure_vnext_compare.py
git commit -m "feat(ocr): expand legacy-vnext comparison harness for cutover gates"
```
### Task 2: Curate the full spec-required comparison corpus
**Rationale:** Spec §8.3 requires explicit coverage of these categories:
- same-page normal figure
- multi-panel same-row group
- sidecar legend page
- bundle-source page
- locator-bridge page
- dense composite parent page
- classic sequential-only rescue page
- unmatched asset / unresolved cluster page
- duplicated / continued legend page
Current repo fixtures cover only 3 papers and do not document category coverage explicitly.
**Files:**
- Create: `tests/fixtures/ocr_vnext_cutover_manifest.json`
- Create or extend fixture dirs under `tests/fixtures/ocr_vnext_real_papers/`
- Create: `project/current/2026-07-03-vnext-cutover-corpus-map.md`
**Manifest fields per paper:**
```json
{
"paper": "YGH7VEX6",
"categories": ["same-page normal figure", "classic sequential-only rescue"],
"source": "D:/L/OB/Literature-hub/System/PaperForge/ocr/YGH7VEX6/structure/blocks.structured.jsonl",
"notes": "why this paper represents the category"
}
```
**Requirements:**
- Use the smallest paper that cleanly demonstrates each category where possible.
- One paper may satisfy multiple categories.
- Every category must be mapped.
- Copy needed `blocks.structured.jsonl` files into the repo fixture tree.
- [ ] **Step 1: Select fixture papers from OCR vault**
- [ ] **Step 2: Copy missing fixtures into repo**
- [ ] **Step 3: Write `ocr_vnext_cutover_manifest.json`**
- [ ] **Step 4: Write the human-readable corpus map markdown**
- [ ] **Step 5: Commit**
```bash
git add tests/fixtures/ocr_vnext_cutover_manifest.json tests/fixtures/ocr_vnext_real_papers/ project/current/2026-07-03-vnext-cutover-corpus-map.md
git commit -m "test(ocr): add full cutover comparison corpus manifest"
```
### Task 3: Generate diff reports for the full corpus
**Rationale:** Before switching the wrapper, we need one auditable diff report per paper plus a roll-up summary.
**Files:**
- Create: `project/current/2026-07-03-vnext-cutover-diff-review.md`
- Create: `project/current/vnext-cutover-diffs/<PAPER>.json`
- Optional helper script: `scripts/dev/compare_figure_inventory_corpus.py`
**Per-paper JSON must include at least:**
- legacy matched count
- vnext matched count
- legacy unresolved count
- vnext unresolved count
- legacy/vnext unmatched legends count
- legacy/vnext completeness
- legacy/vnext consumed block ids
- figure-id lists
- pass names triggered in vnext
- reviewer verdict placeholder (`unknown | equivalent | better | regression`)
**Roll-up markdown should summarize:**
- which papers are parity
- which papers are strict improvements
- which papers need manual explanation
- any suspected regressions
- [ ] **Step 1: Build or script corpus-wide compare run**
- [ ] **Step 2: Emit one JSON diff per paper**
- [ ] **Step 3: Write markdown roll-up**
- [ ] **Step 4: Commit**
```bash
git add project/current/2026-07-03-vnext-cutover-diff-review.md project/current/vnext-cutover-diffs/ scripts/dev/compare_figure_inventory_corpus.py
git commit -m "analysis(ocr): generate vnext cutover diff review package"
```
### Task 4: Regression triage and fix wave
**Rationale:** Any unexplained regression blocks cutover. This task is intentionally open-ended but still bounded: only fix issues surfaced by the cutover diff package and required tests.
**Inputs:**
- `project/current/2026-07-03-vnext-cutover-diff-review.md`
- per-paper diff JSONs
**Outputs:**
- code fixes as needed
- updated diff reports after fixes
- written explanation for every remaining difference judged acceptable
**Rules:**
- If a previously confident legacy figure disappears in vnext, fix or explicitly explain it.
- If vnext consumes the same asset twice, fix immediately.
- If an owned asset appears in unresolved clusters, fix immediately.
- If vnext matches more figures than legacy using the same consumed asset set, that is **not automatically a bug**; it needs manual adjudication and written justification.
- [ ] **Step 1: Triage every non-parity paper**
- [ ] **Step 2: Fix real regressions**
- [ ] **Step 3: Re-run targeted paper diffs and affected tests**
- [ ] **Step 4: Update roll-up verdicts (`equivalent | better | regression`)**
- [ ] **Step 5: Commit**
```bash
git add paperforge/worker/ tests/ project/current/2026-07-03-vnext-cutover-diff-review.md project/current/vnext-cutover-diffs/
git commit -m "fix(ocr): resolve cutover diff regressions"
```
### Task 5: Gate verification sweep
**Rationale:** Spec §9 cutover gates require contract compatibility, regression suite pass, real-paper diff review, and diagnostics superiority.
**Checks:**
**Gate 1 — contract**
- Explicitly verify `FigureInventory` schema keys expected by downstream code remain present.
**Gate 2 — regression suite**
Run:
```bash
python -m pytest tests/test_ocr_figure_vnext_*.py -q
python -m pytest tests/test_ocr_figures.py tests/test_ocr_render.py -q
```
**Gate 3 — real-paper diff review**
- Every paper in the cutover corpus must be marked `equivalent` or `better`.
- No unexplained regression may remain.
**Gate 4 — diagnostics superiority**
- Confirm vnext exposes:
- claim journal
- ownership conflict explanation
- pass-level invariants
- completeness accounting trace
**Artifact:**
- Create: `project/current/2026-07-03-vnext-cutover-gate-checklist.md`
- [ ] **Step 1: Run contract + test gates**
- [ ] **Step 2: Complete gate checklist markdown**
- [ ] **Step 3: Commit**
```bash
git add project/current/2026-07-03-vnext-cutover-gate-checklist.md
git commit -m "test(ocr): complete vnext cutover gate verification"
```
### Task 6: Default wrapper switch (only if Tasks 1-5 are all green)
**Rationale:** This is the only behavior-changing production step. It should happen last.
**Files:**
- Modify: `paperforge/worker/ocr_figures.py`
- Modify: `tests/test_ocr_figures.py`
**Change:**
```python
def build_figure_inventory(structured_blocks: list[dict], page_width: float = 1200) -> dict[str, Any]:
return build_figure_inventory_vnext(structured_blocks, page_width)
```
**Tests:**
- Update wrapper tests so the stable public wrapper is now asserted to call vnext.
- Keep `build_figure_inventory_legacy(...)` callable for comparison tooling.
**Verification:**
```bash
python -m pytest tests/test_ocr_figure_vnext_*.py -q
python -m pytest tests/test_ocr_figures.py tests/test_ocr_render.py -q
```
- [ ] **Step 1: Flip wrapper to vnext**
- [ ] **Step 2: Update wrapper contract tests**
- [ ] **Step 3: Re-run final gates**
- [ ] **Step 4: Commit**
```bash
git add paperforge/worker/ocr_figures.py tests/test_ocr_figures.py
git commit -m "feat(ocr): switch build_figure_inventory wrapper to vnext"
```
## Self-Review
- This plan does **not** assume vnext is already safe to switch on; it treats cutover as gated.
- Current evidence suggests vnext is **promising**, not yet fully certified:
- no production regression today because wrapper still points at legacy
- parity on 2 repo fixtures
- one plausible improvement (`DWQQK2YB`) still needs written adjudication
- comparison corpus required by spec §8.3 is not yet complete
- After Task 6, no architecture work remains — only normal stabilization and future heuristic tuning.
## Execution Handoff
Recommended execution order:
- Task 1
- Task 2
- Task 3
- Task 4
- Task 5
- Task 6 (only if all prior tasks pass)
Tasks 2-4 are analysis/reporting-heavy; they can be partially parallelized once the expanded compare harness exists.

View file

@ -0,0 +1,548 @@
# Figure Pipeline VNext — Robustness-First Modularization Design
Date: 2026-07-03
Scope: `paperforge/worker/ocr_figures.py` figure pipeline only
Status: Draft for review
## 1. Goal
Extract the legacy figure pipeline into a new, clean, robustness-first pipeline without incrementally mutating the existing 1663-line `build_figure_inventory` implementation.
The new pipeline must:
1. preserve the external interface `build_figure_inventory(structured_blocks, page_width=1200) -> FigureInventory`
2. support isolated development in a dedicated branch/worktree
3. make fallback behavior explicit and auditable
4. prevent state corruption across stages
5. allow clean cutover back into the main OCR pipeline without regressions
## 2. Non-goals
- No attempt to redesign the OCR role system, layout pipeline, or table pipeline in this effort.
- No immediate algorithmic rewrite of every heuristic.
- No mixed incremental edits inside the legacy function as the primary strategy.
## 3. Current state
### 3.1 Repository state observed on 2026-07-03
Current branch: `master`
Dirty tracked files:
- `AGENTS.md`
- `paperforge/worker/ocr_document.py`
- `paperforge/worker/ocr_figures.py`
- `tests/test_ocr_document.py`
- `tests/test_ocr_figures.py`
- `tests/test_ocr_real_paper_regressions.py`
Untracked artifacts:
- `project/archive/2026-07-03-layout-capability-audit.md`
- `project/archive/2026-07-03-layout-risk-scan.md`
- `project/archive/2026-07-03-mcp-risk-scan.md`
- `project/current/2026-07-03-figure-pipeline-analysis.md`
- `tests/fixtures/ocr_real_papers/9TW98JH8/`
- `tests/fixtures/ocr_real_papers/MZC482YI/`
Existing worktrees:
- `.worktrees/feat/pdf-annotation-layer`
- `.worktrees/feat-ocr-structured-pipeline`
- `.worktrees/ocr-v2` (detached HEAD)
- `.worktrees/paperforge-stabilization`
- `.worktrees/plugin-ts-migration`
- `../ocr-reading-order-layers`
Existing local branches tied to worktrees:
- `feat/ocr-structured-pipeline`
- `feat/pdf-annotation-layer`
- `feat/plugin-ts-migration`
- `feature/ocr-reading-order-layers`
- `paperforge-stabilization`
- `hotfix-1.5.5`
- `local-pairing-governance`
- `master`
### 3.2 Architectural state
The current figure pipeline is centered on `paperforge/worker/ocr_figures.py::build_figure_inventory`:
- 1663 lines
- 193 cyclomatic complexity
- 546 cognitive complexity
- 51 loops
- 55 direct callees
- 10 sequential processing stages
The primary structural weakness is not merely function size. The real weakness is the combination of:
- shared mutable state (`matched_figures`, `unmatched_legends`, `unmatched_assets`)
- global ownership mutation (`used_group_ids`, `used_asset_page_ids`, `FigureOwnershipRegistry`)
- sequential fallbacks that encode priority through execution order rather than explicit arbitration
## 4. Why a clean extraction is the right strategy
A cleanliness-first branch is justified because robustness work on this pipeline is unusually sensitive to hidden coupling. Editing the legacy function in place would keep all of the following risks alive during the refactor:
- accidental behavior changes inside a giant function
- interleaving old and new state models
- difficulty proving whether a regression came from the new design or the untouched legacy logic
- inability to compare outputs from old and new implementations on the same inputs
The safer strategy is to create a new pipeline implementation in isolation, keep the legacy path intact, and compare them explicitly before cutover.
## 5. Recommended branch/worktree strategy
### 5.1 Isolation model
Use a **dedicated branch and dedicated worktree** for the new figure pipeline only.
Recommended shape:
- branch: `feat/figure-pipeline-vnext`
- worktree: `.worktrees/feat-figure-pipeline-vnext`
The new pipeline should live alongside the old one, not inside it.
### 5.2 Cleanup before starting
Before creating the new worktree, review all existing worktrees and branches and classify each as:
- active and keep
- stale but inspect first
- removable
Special attention:
- `.worktrees/ocr-v2` is detached HEAD and should not be removed blindly
- any branch with unmerged commits or current relevance must be preserved
Cleanup must be conservative and evidence-led. Removal should happen only after verifying:
- branch merged or intentionally abandoned
- no uncommitted work inside the worktree
- no active role in current OCR work
### 5.3 Coexistence model
Do **not** put `if experimental:` branches throughout the old function.
Instead, make the first extraction step purely mechanical:
- rename the current implementation to `build_figure_inventory_legacy(...)`
- add a new `build_figure_inventory_vnext(...)` orchestrator
- keep `build_figure_inventory(...)` calling legacy until explicit cutover
- add a dev comparison harness that imports both functions directly
The stable wrapper during development is:
```python
def build_figure_inventory(structured_blocks, page_width=1200):
return build_figure_inventory_legacy(structured_blocks, page_width)
```
The explicit experimental entrypoint is:
```python
def build_figure_inventory_vnext(structured_blocks, page_width=1200):
...
```
Comparison must happen through tooling/dev scripts, not production switching.
## 6. VNext architecture
## 6.1 External seam
Keep the existing external interface unchanged:
```python
build_figure_inventory(structured_blocks, page_width=1200) -> FigureInventory
```
This preserves downstream compatibility with renderers, rebuild scripts, and tests.
## 6.2 Internal modules
### A. `FigureCorpus`
Responsibility: immutable interpretation layer over `structured_blocks`.
Provides only stable facts and indexes:
- raw legend-like blocks
- raw asset-like blocks
- figure locator candidates
- page context
- layout hints
- block/page/resource indexes
It may perform:
- prefix recovery
- legend classification
- asset family hinting
- normalized block/resource indexing
It must **not** perform:
- candidate group construction
- fallback-specific hypothesis construction
- matching
- ownership mutation
### B. `FigureCandidateIndex`
Responsibility: derived candidate and hypothesis layer built from `FigureCorpus`.
Provides:
- formal legends
- held legends
- rejected legends
- deduped legends
- candidate groups
- competing caption pages
- sidecar candidates
- bundle-source candidates
- locator candidates
This split prevents `FigureCorpus` from becoming a new god object.
### C. `OwnershipLedger`
Responsibility: single authority over claimable resources.
Resources:
- legends
- groups
- assets
The only valid identity carrier is `ResourceRef`:
```python
@dataclass(frozen=True)
class ResourceRef:
kind: Literal["legend", "asset", "group"]
page: int | None
block_id: int | str | None
group_id: str | None = None
figure_no: int | None = None
origin: str | None = None
```
Identity rules:
- `block_id` alone is never a valid ownership key
- asset identity must include `page + block_id` or a future stable `asset_id`
- group identity must include `page + group_id`
- legend identity must include `page + block_id`, with normalized figure number attached when available
Capabilities:
- reserve
- claim
- block
- release
- conflict reporting
- journal emission
- snapshotting for diagnostics
This is the most important new seam. Without it, modularization would only be cosmetic.
### D. `FigurePipelineState`
Responsibility: store derived decision state, not raw OCR facts.
Contains:
- `corpus`
- `candidate_index`
- `ledger`
- `matches`
- `unresolved`
- `hypotheses`
- `diagnostics`
This state must expose controlled mutations rather than naked list edits.
### E. `Pass` modules
Each stage becomes a module with a small interface:
```python
class FigurePass(Protocol):
name: str
def run(self, state: FigurePipelineState) -> PassReport: ...
```
The passes are:
- `PrimarySamePagePass`
- `CrossPageReservationPass`
- `CrossPageSettlementPass`
- `SidecarPass`
- `LegendBundlePass`
- `LocatorBridgePass`
- `GroupSequentialPass`
- `ClassicSequentialPass`
- `CompositeParentPass`
- `FinalAccountingPass`
## 7. Processing model changes for robustness
## 7.1 Proposal-then-commit
Passes must stop mutating global outcome lists directly.
Each pass must first emit an explicit proposal object:
```python
@dataclass
class ClaimProposal:
pass_name: str
figure_no: int | None
claim_type: Literal[
"match",
"reserve",
"block",
"unresolved_cluster",
"composite_parent",
]
legends: list[ResourceRef]
assets: list[ResourceRef]
groups: list[ResourceRef]
confidence: float
evidence_rank: int
reason: str
diagnostics: dict
```
And each pass must report its outcome explicitly:
```python
@dataclass
class PassReport:
pass_name: str
proposals: list[ClaimProposal]
accepted: list[ClaimProposal]
rejected: list[ClaimProposal]
conflicts: list[OwnershipConflict]
invariant_errors: list[str]
```
New flow:
1. pass generates `ClaimProposal`
2. proposal is validated
3. ledger arbitrates resource ownership
4. accepted proposals update state
5. rejected proposals remain in diagnostics
This avoids silent corruption from premature list/set mutation.
## 7.2 Explicit resource lifecycle
### Legend lifecycle
- `available`
- `held`
- `matched`
- `rejected`
- `consumed_as_locator`
- `consumed_as_bundle_source`
### Group lifecycle
- `available`
- `reserved`
- `matched`
- `blocked`
- `promoted_to_composite_parent`
### Asset lifecycle
- `available`
- `reserved`
- `owned`
- `blocked`
- `absorbed_into_unresolved_cluster`
## 7.3 Fallbacks become layered arbitration
Replace the current long sequential fallback chain with explicit layers.
### Layer 1 — high certainty
- same-page primary
- sidecar
- locator bridge
### Layer 2 — structural cross-page
- cross-page reservation/settlement
- legend bundle
- group-aware sequential
### Layer 3 — low certainty fallback
- classic sequential
- dense/unresolved consolidation
Rules:
- lower layers cannot steal already-claimed resources from higher layers
- same-layer conflicts are resolved by confidence, evidence rank, and pass-specific precedence
- classic sequential consumes only untouched or explicitly eligible resources
## 7.4 Invariants become first-class
Each pass should be followed by invariant checks.
Examples:
- one asset cannot be owned by two figures
- matched legends cannot remain in unmatched pools
- unresolved clusters cannot contain owned assets
- completeness totals must equal the numbered legends admitted to the pipeline
- every consumed resource must be explainable through journal entries
## 7.5 Preserve information; do not eagerly delete it
When a pass fails, downgrade state rather than deleting objects from the system.
Examples:
- failed match -> unmatched, not removed
- unsafe group -> blocked with reason
- locator-consumed legend -> marked consumed, not erased
- low-certainty match -> preserved as such in diagnostics
This improves auditability and debugging.
## 8. Migration strategy
## 8.1 First cut
Do **not** start with sidecar or bundle logic.
Phase 0 must be purely mechanical extraction:
- rename current behavior to `build_figure_inventory_legacy(...)`
- add a new `build_figure_inventory_vnext(...)` orchestrator shell
- keep `build_figure_inventory(...)` calling legacy only
- add a `compare_figure_inventory_legacy_vs_vnext` harness
The legacy implementation is an immutable behavioral baseline.
Do **not** retrofit `OwnershipLedger` into the legacy function.
The ledger is introduced only inside `build_figure_inventory_vnext(...)`.
## 8.2 Recommended rollout order
### Step 1
Mechanical extraction + comparison harness
### Step 2
`OwnershipLedger` + `FigurePipelineState` + same-page primary matching inside vnext
### Step 3
Cross-page reservation + settlement
### Step 4
Special fallbacks
- sidecar
- bundle
- locator
### Step 5
Low-certainty and accounting tail
- group-aware sequential
- classic sequential
- completeness/accounting
Rationale: the weakest, broadest fallback logic should be migrated last.
Migration order does **not** define final arbitration priority.
During partial rollout, disabled passes must emit no proposals.
Final arbitration priority is defined only by the layer table in §7.3.
## 8.3 Comparison strategy
During development, run the new path and legacy path on the same papers and diff:
- matched figure count
- unmatched legends
- unresolved clusters
- ownership conflicts
- completeness totals
- downstream rendering effects
Required comparison corpus must include:
- same-page normal figure
- multi-panel same-row group
- sidecar legend page
- bundle-source page
- locator-bridge page
- dense composite parent page
- classic sequential-only rescue page
- unmatched asset / unresolved cluster page
- duplicated / continued legend page
Each corpus entry should emit a diff report containing at least:
- legacy matched count
- vnext matched count
- legacy unresolved count
- vnext unresolved count
- resource conflict count
- rendered figure cards diff
- consumed block ids diff
## 9. Cutover gates
The new pipeline becomes default only if all of the following hold.
### Gate 1 — contract
`FigureInventory` schema remains compatible.
### Gate 2 — regression suite
All relevant figure/render/rebuild tests pass.
### Gate 3 — real-paper diff review
Differences on representative real papers are explained and judged strictly better or equivalent.
VNext may differ from legacy only if:
- the difference is explicitly recorded in the diff report
- no previously rendered confident figure disappears without explanation
- no owned asset appears in unresolved clusters
- no figure card consumes the same asset twice
### Gate 4 — diagnostics superiority
The new pipeline must provide stronger traceability than the old one:
- claim journal
- ownership conflict explanation
- pass-level invariants
- completeness accounting trace
## 10. Risk assessment
### Main benefits
- far better locality
- auditable ownership and fallback behavior
- safer future debugging
- ability to add new figure rules without touching a giant mutable function
### Main risks
- migration period may temporarily create dual complexity
- if `FigurePipelineState` is too broad, it becomes a new god object
- if pass interfaces leak too much detail, the modules stay shallow
## 11. Decision
Proceed with a clean extraction into a dedicated branch/worktree, anchored on a new ownership/state seam rather than incremental edits inside the existing giant function.
This is the best balance of robustness, cleanliness, and regression control.
## 12. Required clarifications before implementation
1. Legacy implementation is an immutable baseline.
Do not retrofit `OwnershipLedger` into the legacy function.
2. Phase 0 only performs mechanical extraction:
- `build_figure_inventory_legacy = old behavior`
- `build_figure_inventory_vnext = new orchestrator shell`
- `build_figure_inventory` keeps calling legacy until cutover
3. Add `ResourceRef` as the only valid ownership key.
`block_id` alone is forbidden.
4. Define `ClaimProposal` and `PassReport` schemas before implementing passes.
5. Split `FigureCorpus` and `FigureCandidateIndex`:
- `FigureCorpus` stores immutable facts
- `FigureCandidateIndex` stores derived candidates and hypotheses
6. Add a legacy-vnext comparison harness before implementing special fallbacks.
7. Final arbitration priority is independent of migration order.

View file

@ -1445,6 +1445,30 @@ def infer_zones(
body_started = True
if not body_started:
frontmatter_main_blocks.append(block)
# Rescue frontmatter_support blocks that sit below body start on page 1.
# Old-style single-column journals (e.g. CAQNW9Q2) place correspondence
# inline after the first body paragraph. Width-vs-body-anchor distinguishes
# narrow metadata furniture from full-width body prose — no text matching.
if body_started and body_anchor_ok:
fm_main_id_set_pre = set(
_artifact_block_id(b, duplicate_block_ids) for b in frontmatter_main_blocks
)
body_width_val = body_anchor.get("width_bucket")
for block in page1_candidates:
if _artifact_block_id(block, duplicate_block_ids) in fm_main_id_set_pre:
continue
role = block.get("role") or block.get("seed_role")
if role == "unassigned":
role = block.get("seed_role")
if role != "frontmatter_support":
continue
bbox = _block_bbox(block)
if bbox is None or body_width_val is None:
continue
block_width = bbox[2] - bbox[0]
if block_width > 0 and block_width <= float(body_width_val) - 100:
frontmatter_main_blocks.append(block)
frontmatter_main_ids = [_artifact_block_id(block, duplicate_block_ids) for block in frontmatter_main_blocks]
frontmatter_main_composite_ids = [_zone_block_key(block) for block in frontmatter_main_blocks]
@ -2153,7 +2177,11 @@ def _veto_tail_spread_body_continuation(boundary: TailBoundary, blocks: list[dic
if page > 0:
by_page.setdefault(page, []).append(b)
spread = boundary.spread_start
while spread is not None and _page_has_strong_body_continuation(by_page.get(spread, [])):
while (
spread is not None
and spread < boundary.spread_end
and _page_has_strong_body_continuation(by_page.get(spread, []))
):
spread += 1
return boundary._replace(spread_start=spread)

View file

@ -0,0 +1,75 @@
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
@dataclass
class FigureCorpus:
blocks: list[dict]
page_width: float
raw_legends: list[dict] = field(default_factory=list)
raw_assets: list[dict] = field(default_factory=list)
locator_candidates: list[dict] = field(default_factory=list)
page_layouts: dict[int, Any] = field(default_factory=dict)
@classmethod
def from_blocks(cls, blocks: list[dict], page_width: float = 1200) -> "FigureCorpus":
from . import ocr_figures
from .ocr_document import _build_page_layout_profiles
raw_legends = [b for b in blocks if b.get("role") in {"figure_caption", "figure_caption_candidate"}]
raw_assets = [b for b in blocks if b.get("role") in {"figure_asset", "media_asset"}]
locator_candidates = [b for b in raw_legends if ocr_figures._is_previous_page_legend_locator(b)]
return cls(
blocks=list(blocks),
page_width=page_width,
raw_legends=raw_legends,
raw_assets=raw_assets,
locator_candidates=locator_candidates,
page_layouts=_build_page_layout_profiles(blocks),
)
@dataclass
class FigureCandidateIndex:
formal_legends: list[dict]
held_legends: list[dict]
rejected_legends: list[dict]
deduped_legends: list[dict]
candidate_groups: list[dict]
competing_caption_pages: set[int]
sidecar_candidates: dict[int, list[dict]]
bundle_source_legend_ids: set[str]
locator_candidates: list[dict]
@classmethod
def from_corpus(cls, corpus: FigureCorpus) -> "FigureCandidateIndex":
from . import ocr_figures
formal_legends = [
b for b in corpus.raw_legends if ocr_figures._is_formal_legend(str(b.get("text", "")), b, corpus.page_width)
]
rejected_legends = [b for b in corpus.raw_legends if b not in formal_legends]
candidate_groups = ocr_figures._build_candidate_figure_groups_from_assets(
corpus.raw_assets,
formal_legends,
corpus.blocks,
page_width=corpus.page_width,
)
competing_caption_pages = {
int(leg.get("page"))
for leg in formal_legends
if leg.get("page") is not None and ocr_figures._extract_figure_number(str(leg.get("text", ""))) is not None
}
return cls(
formal_legends=formal_legends,
held_legends=[],
rejected_legends=rejected_legends,
deduped_legends=formal_legends,
candidate_groups=candidate_groups,
competing_caption_pages=competing_caption_pages,
sidecar_candidates={},
bundle_source_legend_ids=set(),
locator_candidates=corpus.locator_candidates,
)

View file

@ -0,0 +1,104 @@
from __future__ import annotations
from .ocr_figure_vnext_types import ClaimProposal, PassReport, ResourceRef
def _resource_page(block: dict) -> int | None:
page = block.get("page")
if page is None:
page = block.get("page_num")
return int(page) if page is not None else None
class PrimarySamePagePass:
name = "primary_same_page"
def _collect_proposals(self, state):
from . import ocr_figures
proposals = []
for legend in state.candidate_index.deduped_legends:
page = _resource_page(legend)
if page is None:
continue
page_groups = [g for g in state.candidate_index.candidate_groups if _resource_page(g) == page]
for group in page_groups:
score = ocr_figures._score_legend_to_group(
legend,
group,
caption_score=ocr_figures.score_figure_caption(
legend,
nearby_media=True,
caption_style_match=False,
body_prose_likelihood=False,
),
page_width=state.corpus.page_width,
)
if score.get("decision") != "matched":
continue
figure_no = ocr_figures._extract_figure_number(str(legend.get("text", "")))
proposals.append(ClaimProposal(
pass_name=self.name,
figure_no=figure_no,
claim_type="match",
legends=[ResourceRef(kind="legend", page=page, block_id=legend.get("block_id"), figure_no=figure_no)],
assets=[ResourceRef(kind="asset", page=page, block_id=bid) for bid in group.get("asset_block_ids", [])],
groups=[ResourceRef(kind="group", page=page, block_id=None, group_id=group.get("group_id"))],
confidence=float(score.get("score", 0.0)),
evidence_rank=1,
reason="same_page_primary",
diagnostics={
"evidence": list(score.get("evidence", [])),
"legend_block_id": str(legend.get("block_id", "")),
},
))
return proposals
def _materialize_match(self, state, proposal):
from . import ocr_figures
legend = proposal.legends[0]
page = legend.page
asset_ids = {str(r.block_id) for r in proposal.assets}
matched_assets = [
ocr_figures._project_asset_record(a)
for a in state.corpus.raw_assets
if _resource_page(a) == page and str(a.get("block_id", "")) in asset_ids
]
legend_text = next(
str(b.get("text", ""))
for b in state.candidate_index.deduped_legends
if str(b.get("block_id", "")) == legend.block_id
)
namespace = ocr_figures._extract_figure_namespace(legend_text)
return {
"figure_id": ocr_figures._format_figure_id(namespace, proposal.figure_no),
"figure_namespace": namespace,
"figure_number": proposal.figure_no,
"legend_block_id": legend.block_id,
"page": page,
"text": legend_text,
"matched_assets": matched_assets,
"asset_block_ids": sorted(asset_ids),
"settlement_type": "same_page",
"confidence": proposal.confidence,
"match_score": {"score": proposal.confidence, "decision": "matched", "evidence": proposal.diagnostics["evidence"]},
"flags": [],
"bridge_block_ids": [],
}
def run(self, state):
report = PassReport(pass_name=self.name)
proposals = self._collect_proposals(state)
report.proposals.extend(proposals)
for proposal in sorted(proposals, key=lambda p: (p.evidence_rank, -p.confidence, -(p.figure_no or -1))):
conflict = state.ledger.try_claim_assets(proposal.assets, owner=proposal.legends[0], reason=proposal.reason)
if conflict is not None:
report.conflicts.append(conflict)
report.rejected.append(proposal)
continue
state.accept_match(proposal, self._materialize_match(state, proposal))
report.accepted.append(proposal)
return report

View file

@ -0,0 +1,72 @@
from __future__ import annotations
from dataclasses import dataclass, field
from .ocr_figure_vnext_types import ClaimProposal, OwnershipConflict, ResourceRef
class OwnershipLedger:
def __init__(self) -> None:
self._owners: dict[ResourceRef, ResourceRef] = {}
self._journal: list[dict[str, object]] = []
def claim_assets(self, assets: list[ResourceRef], *, owner: ResourceRef, reason: str) -> None:
conflict = self.try_claim_assets(assets, owner=owner, reason=reason)
if conflict is not None:
raise ValueError(f"asset already owned: {conflict.resource}")
def try_claim_assets(
self, assets: list[ResourceRef], *, owner: ResourceRef, reason: str
) -> OwnershipConflict | None:
for asset in assets:
current = self._owners.get(asset)
if current is not None and current != owner:
conflict = OwnershipConflict(
resource=asset, current_owner=current, attempted_owner=owner, reason=reason
)
self._journal.append({
"action": "conflict",
"resource": asset,
"current_owner": current,
"attempted_owner": owner,
"reason": reason,
})
return conflict
for asset in assets:
self._owners[asset] = owner
self._journal.append({"action": "claim", "resource": asset, "owner": owner, "reason": reason})
return None
def owner_of(self, resource: ResourceRef) -> ResourceRef | None:
return self._owners.get(resource)
def owner_of_asset(self, *, page: int, block_id: int | str) -> ResourceRef | None:
return self.owner_of(ResourceRef(kind="asset", page=page, block_id=block_id))
def snapshot(self) -> list[dict[str, object]]:
return list(self._journal)
@dataclass
class FigurePipelineState:
corpus: object | None
candidate_index: object | None
ledger: OwnershipLedger
matches: list[dict] = field(default_factory=list)
unresolved: list[dict] = field(default_factory=list)
hypotheses: list[dict] = field(default_factory=list)
diagnostics: list[dict] = field(default_factory=list)
def accept_match(self, proposal: ClaimProposal, match_record: dict) -> None:
self.matches.append(match_record)
self.diagnostics.append({
"event": "match_accepted",
"pass_name": proposal.pass_name,
"figure_no": proposal.figure_no,
"reason": proposal.reason,
"resources": {
"legends": proposal.legends,
"assets": proposal.assets,
"groups": proposal.groups,
},
})

View file

@ -0,0 +1,61 @@
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Literal
@dataclass(frozen=True)
class ResourceRef:
kind: Literal["legend", "asset", "group"]
page: int | None
block_id: str | None
group_id: str | None = None
figure_no: int | None = None
origin: str | None = None
def __post_init__(self) -> None:
if self.page is not None:
object.__setattr__(self, "page", int(self.page))
if self.block_id is not None:
object.__setattr__(self, "block_id", str(self.block_id))
if self.group_id is not None:
object.__setattr__(self, "group_id", str(self.group_id))
if self.kind == "asset" and (self.page is None or self.block_id is None):
raise ValueError("asset ResourceRef requires page + block_id")
if self.kind == "legend" and (self.page is None or self.block_id is None):
raise ValueError("legend ResourceRef requires page + block_id")
if self.kind == "group" and (self.page is None or self.group_id is None):
raise ValueError("group ResourceRef requires page + group_id")
@dataclass(frozen=True)
class OwnershipConflict:
resource: ResourceRef
current_owner: ResourceRef | None
attempted_owner: ResourceRef | None
reason: str
@dataclass
class ClaimProposal:
pass_name: str
figure_no: int | None
claim_type: Literal["match", "reserve", "block", "unresolved_cluster", "composite_parent"]
legends: list[ResourceRef]
assets: list[ResourceRef]
groups: list[ResourceRef]
confidence: float
evidence_rank: int
reason: str
diagnostics: dict[str, Any] = field(default_factory=dict)
@dataclass
class PassReport:
pass_name: str
proposals: list[ClaimProposal] = field(default_factory=list)
accepted: list[ClaimProposal] = field(default_factory=list)
rejected: list[ClaimProposal] = field(default_factory=list)
conflicts: list[OwnershipConflict] = field(default_factory=list)
invariant_errors: list[str] = field(default_factory=list)

View file

@ -3,10 +3,16 @@ from __future__ import annotations
import contextlib
import itertools
import re
from dataclasses import asdict
from pathlib import Path
from typing import Any
from paperforge.core.io import write_json
from paperforge.worker.ocr_document import (
PageLayoutProfile,
_build_page_layout_profiles,
_col_boundaries_for_page,
_get_column_index,
)
from paperforge.worker.ocr_roles import _PANEL_LABEL_PATTERN, _looks_like_figure_description_opening
from paperforge.worker.ocr_scores import score_figure_caption, score_figure_match
@ -2976,8 +2982,47 @@ def _infer_missing_main_figure_numbers(
}
return inventory
def build_figure_inventory(structured_blocks: list[dict], page_width: float = 1200, page_pdf_lines_by_page: dict[int, list[dict]] | None = None) -> dict[str, Any]:
return build_figure_inventory_legacy(structured_blocks, page_width, page_pdf_lines_by_page)
def build_figure_inventory_vnext(structured_blocks: list[dict], page_width: float = 1200) -> dict[str, Any]:
from .ocr_figure_vnext_corpus import FigureCandidateIndex, FigureCorpus
from .ocr_figure_vnext_passes import PrimarySamePagePass, _resource_page
from .ocr_figure_vnext_state import FigurePipelineState, OwnershipLedger
corpus = FigureCorpus.from_blocks(structured_blocks, page_width=page_width)
candidate_index = FigureCandidateIndex.from_corpus(corpus)
state = FigurePipelineState(corpus=corpus, candidate_index=candidate_index, ledger=OwnershipLedger())
report = PrimarySamePagePass().run(state)
matched_ids = {str(m.get("legend_block_id", "")) for m in state.matches}
return {
"pipeline_mode": "vnext",
"matched_figures": state.matches,
"ambiguous_figures": [],
"unmatched_legends": [b for b in candidate_index.deduped_legends if str(b.get("block_id", "")) not in matched_ids],
"unmatched_assets": [
a for a in corpus.raw_assets
if (_resource_page(a) is not None
and state.ledger.owner_of_asset(page=_resource_page(a), block_id=a.get("block_id")) is None)
],
"unresolved_clusters": [],
"held_figures": list(candidate_index.held_legends),
"rejected_legends": list(candidate_index.rejected_legends),
"page_ledger": {},
"residual_ledger": {},
"local_pairing_hypotheses": [],
"pass_reports": [asdict(report)],
"completeness": {
"total_numbered_legends": len(candidate_index.deduped_legends),
"accounted_for": len(state.matches),
"details": [],
},
}
def build_figure_inventory_legacy(structured_blocks: list[dict], page_width: float = 1200, page_pdf_lines_by_page: dict[int, list[dict]] | None = None) -> dict[str, Any]:
legends: list[dict] = []
held_figures: list[dict] = []
rejected_legends: list[dict] = []
@ -3009,6 +3054,9 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
if block.get("page_width"):
page_width = float(block["page_width"])
# Build per-page column profiles once, reused by prefix recovery
page_layouts = _build_page_layout_profiles(structured_blocks)
for block in structured_blocks:
role = block.get("role", "")
if block.get("_non_body_media") or role == "non_body_insert":
@ -3037,7 +3085,7 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
# PDF prefix recovery: restore "Figure N" heading missed by OCR
# Must run BEFORE the zone/style filter that checks _extract_figure_number()
if page_pdf_lines_by_page and _extract_figure_number(text) is None:
recovered = _recover_figure_heading_prefix(block, page_pdf_lines_by_page)
recovered = _recover_figure_heading_prefix(block, page_pdf_lines_by_page, page_layouts)
if recovered:
block["text"] = recovered
text = recovered
@ -6045,10 +6093,10 @@ def _recover_missing_figure_numbers_from_assets(
"figure_number_recovered_from_asset_text",
]))
def _recover_figure_heading_prefix(
block: dict,
page_pdf_lines_by_page: dict[int, list[dict]],
page_layouts: dict[int, PageLayoutProfile] | None = None,
) -> str | None:
"""Recover 'FIGURE N' heading prefix from PDF text layer for OCR-missed captions.
@ -6056,8 +6104,11 @@ def _recover_figure_heading_prefix(
(e.g. rendered in a bold/small-caps font that the OCR engine doesn't read),
the caption body is captured as figure_caption_candidate but lacks the
figure number prefix. This function checks the PDF text layer for the
heading on the same page and prepends it if the line immediately after
the heading matches the start of the caption text.
heading on the same page and prepends it if a line below the heading
(within 100px) matches the start of the caption text.
In multi-column layouts, only PDF lines in the same column as the heading
are considered, preventing false matches from adjacent columns.
Returns the complete caption text with prefix, or None if no recovery.
"""
@ -6076,6 +6127,10 @@ def _recover_figure_heading_prefix(
key=lambda l: (l.get("bbox") or [0, 0, 0, 0])[1],
)
# Resolve column boundaries once
col_boundaries = _col_boundaries_for_page(page, page_layouts) if page_layouts else []
multi_column = len(col_boundaries) > 2 # N+1 boundaries for N columns
for i, line in enumerate(sorted_lines):
line_text = str(line.get("text", "") or "").strip()
# Check: line starts with Figure N heading (short heading, not in-text ref)
@ -6085,21 +6140,31 @@ def _recover_figure_heading_prefix(
if len(line_text) > 40:
continue # too long for a heading — probably in-text reference
# Check the next PDF line (by y-order) matches the start of block text
if i + 1 >= len(sorted_lines):
continue
next_text = str(sorted_lines[i + 1].get("text", "") or "").strip()
if not next_text or len(next_text) < 10:
continue
# Capture heading's column index for multi-column layouts
heading_col = _get_column_index(line, col_boundaries) if multi_column else None
# Common-prefix match: at least 15 chars case-insensitive
common = 0
for a, b in zip(next_text.lower(), block_text.lower()):
if a == b:
common += 1
else:
# Scan below-heading lines (within 100px y-gap) for prefix match
heading_y = line.get("bbox", [0, 0, 0, 0])[1] if line.get("bbox") else 0
for j in range(i + 1, len(sorted_lines)):
nxt = sorted_lines[j]
nxt_bbox = nxt.get("bbox") or [0, 0, 0, 0]
if nxt_bbox[1] - heading_y > 100:
break
if common >= 15:
return line_text + "\n" + block_text
if multi_column and heading_col is not None:
if _get_column_index(nxt, col_boundaries) not in (heading_col, None):
continue
next_text = str(nxt.get("text", "") or "").strip()
if not next_text or len(next_text) < 10:
continue
# Common-prefix match: at least 15 chars case-insensitive
common = 0
for a, b in zip(next_text.lower(), block_text.lower()):
if a == b:
common += 1
else:
break
if common >= 15:
return line_text + "\n" + block_text
return None

View file

@ -0,0 +1,89 @@
# Layout Capability Audit — 2026-07-03
## Method
For each complex layout type, evaluated:
- Code logic (does the pipeline have a conceptual model?)
- Actual rendered output (fulltext.md correctness)
- Block role/zone assignment (732 vault papers)
- Known failure patterns
## Findings per Type
### 1. Frontmatter — 🔴 Capability Ceiling
**Current approach**: Hardcoded keyword list (`furniture_signals`, 15 phrases) + position
heuristics (top 20% = title, width < 35% page_width + top half = furniture).
Only pages 1-2.
**Evidence**: CAQNW9Q2 page 1 — `frontmatter_support` (correspondence) leaks into
`body_zone`. Keyword list doesn't match.
**Why ceiling**:
- 15 keywords cannot cover all journal frontmatter styles
- Position thresholds are brittle across layouts
- Only pages 1-2 — multi-page frontmatter automatically mislabeled
- Dependent on seed roles being correct
**Fix requires**: ML model or document grammar. Heuristic patching cannot root-fix.
### 2. Mixed_tail — 🟢 Capable, Working
**Current approach**: `_classify_page_layout` detects `mixed_tail` via role
distribution (one side body + other side tail). `_partition_by_reference_zone`
handles per-page ref zones. `_reconcile_tail_spread` handles body/ref/backmatter
boundary.
**Evidence**: 4 papers checked (A8E7SRVS, KUR9PBJC, 24YKLTHQ, YQIC2RDL):
- All mixed_tail pages correctly classified (conf=0.60)
- Body/ref split reasonable
- KUR9PBJC: clean body_end=12 → ref_start=13
**Gap**: Only 7/73 fixtures have mixed_tail pages. 41 non-fixture papers have
mixed_tail — coverage gap, not capability gap.
### 3. Preproof Frontmatter — 🟢 Capable, Working
**Evidence**: DWQQK2YB renders: title → authors → abstract → highlights →
keywords → Introduction (page 4). Special preproof handling works correctly.
### 4. Figure-heavy / Atlas Pages — 🟡 Data Limitation (Acceptable)
**Evidence**:
- NC66N4Q3 (56p atlas): 52/56 pages `single_column conf=0.35`
- 24YKLTHQ page 7: 8 figure_asset + 1 caption → `single_column conf=0.35`
**Why acceptable**: Pages with few text blocks inherently lack layout signal.
`few_eligible_blocks` fallback to low-confidence `single_column` is correct behavior.
NC66N4Q3 body_end=3 is reasonable (only 3 pages of prose text).
### 5. Two-column Reference Ordering — 🟢 Capable, Fixed
**Evidence**: KUR9PBJC pages 17-18 (60+ refs/page in two_column), A8E7SRVS page 13
(25 refs) all render correctly. `_should_attach_reference_item_to_ref_section`
(commit 9aa228d) handles multi-column continuation.
**History**: Previously buggy, now fixed. No remaining issues.
### 6. Sidecar Caption — 🟢 Capable, Fixed
**Evidence**: 37LK5T97 Figure 1 caption (left column) + image (right column)
correctly paired. `_is_sidecar_candidate` (ocr_document.py:4360) checks vertical
overlap when horizontal overlap is absent.
## Summary
| Layout Type | Verdict | Current State | Real Gap |
|---|---|---|---|
| Frontmatter | 🔴 Ceiling | Heuristics can improve but not fix | Needs ML / grammar |
| Mixed_tail | 🟢 Capable | Classification works | 41 papers not in fixtures |
| Preproof frontmatter | 🟢 Capable | Working | None |
| Figure-heavy / atlas | 🟡 Data limit | Correct fallback | None |
| Two-column ref | 🟢 Capable | Fixed in 9aa228d | None |
| Sidecar caption | 🟢 Capable | Fixed | None |
**Key takeaway**: Only frontmatter is a true capability ceiling. All other layout
types have working pipeline code; prior bug-fix cycles (sidecar, ref ordering,
column-aware prefix recovery, weak cluster rejection) have largely cleared the
implementation gaps. The largest practical gap is mixed_tail fixture coverage
(41 candidate papers not in regression set).

View file

@ -0,0 +1,57 @@
# Layout Pipeline — Remaining Risk Scan (2026-07-03)
## Method
Checked every function in the layout analysis pipeline against code + 100+ real vault papers:
- `_cluster_page_column_groups` / `_classify_page_layout`
- `infer_zones` / `_apply_zone_labels` / `_apply_content_zone_fallback`
- `_detect_forward_body_end` / `_detect_backward_backmatter_start` / `_reconcile_tail_spread`
- `_build_page_reading_segments` / `_build_tail_reading_order`
- `_is_frontmatter_side_candidate` / `_is_first_page_body_start`
- `discover_body_family_anchor`
- `_page_has_strong_body_continuation`
## Covered by existing/planned fixes
| Issue | Status |
|-------|--------|
| Sidecar caption (37LK5T97) | ✅ Fixed in `_is_sidecar_candidate` |
| Two-column ref ordering | ✅ Fixed in `_should_attach_reference_item_to_ref_section` |
| Column-aware prefix recovery | ✅ Fixed in `_recover_figure_heading_prefix` |
| Weak cluster rejection (2E4EPHN2) | ✅ Fixed in `_is_weak_isolated_column_cluster` |
| Spread_start > spread_end (AH6Q7DLC) | ✅ Planned Step 1 — 1-line guard |
| Frontmatter zone leak (CAQNW9Q2) | ✅ Planned Step 2 — width-vs-body rescue |
| mixed_tail fixture coverage | ✅ Planned Step 5 — 2 new fixtures |
## Analyzed and found acceptable — not risks
### 1. Low-confidence layout pages (conf<0.35)
**Frequency:** 536/732 papers.
**Root cause:** Figure-heavy pages have 0-1 `_is_layout_eligible_block` (headings excluded by design). Confidence capped at 0.35 as expected fallback.
**Verdict:** ✅ Not a bug. Data limitation — correct behavior.
### 2. body_anchor_ok=False (7% of papers)
**Mechanism:** When `discover_body_family_anchor` returns HOLD, `body_blocks` filter in `infer_zones` (line 1540) produces empty `body_block_ids`. `_apply_content_zone_fallback` assigns body_zone to everything indiscriminately.
**Real impact:** Reference zone still works (separately detected). Fallback assigns reasonable zones. Only loss is body_zone metadata (anchor_family, boundary_band).
**Verdict:** ✅ Acceptable degradation. The 7% are very short or stylistically unusual papers where zone metadata adds minimal value.
### 3. Mixed_tail classification conservative
**Mechanism:** `_classify_page_layout` requires body column to have ZERO tail roles (line 486: `not col_has_tail[0]`). If both columns have some tail + some body, classified as two_column.
**Verdict:** ✅ Conservative by design. False two_column is safer than false mixed_tail. No evidence of actual wrong behavior.
### 4. Headings excluded from layout eligibility
**Mechanism:** `_LAYOUT_ELIGIBLE_ROLES` = `{body_paragraph, list_item, tail_candidate_body, reference_item, backmatter_body}`. But `_cluster_page_column_groups` uses ALL blocks with bbox width > 50px — not just eligible ones.
**Real effect:** Headings' x-centers contribute to column clustering. Eligibility only affects confidence count (line 534: `len(eligible_blocks) < 2 → confidence = min(confidence, 0.35)`).
**Verdict:** ✅ Safe. Clustering is correct; confidence is correctly conservative.
### 5. Reading order segments
**Mechanism:** `_build_page_reading_segments`: single-col = y-sorted, multi-col = column-grouped then y-sorted. Standard academic reading order.
**Verdict:** ✅ Correct.
## Findings that changed the plan
Only one plan change: Step 2's width rescue was confirmed effective on the motivating paper (CAQNW9Q2 correspondence: 161px vs body 393px = 233px diff, far exceeding 100px threshold). The plan was updated in-session.
## Conclusion
**No new critical or high-risk issues found beyond the three planned fixes.** The pipeline is structurally sound — all components have the right conceptual model. The three remaining issues (spread cap, frontmatter rescue, mixed_tail fixtures) are well-understood, bounded-in-impact, and already planned.

View file

@ -0,0 +1,95 @@
# Codebase-Memory-MCP 风险扫描 — 按功能分类 (2026-07-03)
扫描范围:`ocr_*.py` + `ocr_families.py`,覆盖 ~25K 节点知识图谱。
使用 metrics: cyclomatic complexity, cognitive complexity, alloc_in_loop, linear_scan_in_loop, transitive_loop_depth, param_count, lines。
---
## 版面分析 (ocr_document.py) — 七个函数过界
| 函数 | 圈复杂度 | 认知复杂度 | 行数 | 循环/深度 | 风险等级 |
|------|----------|------------|------|-----------|----------|
| `normalize_document_structure` | **84** | 191 | 581 | 20/2 | 🔶 中 |
| `rescue_roles_with_document_context` | **55** | **247** | 307 | 2/2 | 🔶🔶 高 |
| `_detect_body_spine` | **47** | **181** | 294 | 11/3 | 🔶 中 |
| `_resolve_ambiguous_candidates` | **47** | 150 | 325 | 5/2 | 🔶 中 |
| `infer_zones` | 38 | 74 | 381 | 10/1 | 🔹 低 (已规划) |
| `_detect_non_body_insert_clusters` | 36 | 107 | 161 | 6/2 | 🔹 低 |
| `_run_layout_audit` | 25 | 85 | 152 | 7/4 | 🔹 低 |
**重点**: `rescue_roles_with_document_context` — 认知复杂度 247 是圈复杂度 55 的 **4.5 倍**。典型标志:深层嵌套条件里的反向控制流。该函数还包含基于关键词的前言区域检测 (furniture_signals 表),这个已经被标记为"能力天花板"。
---
## 图片管线 (ocr_figures.py) — 一个巨型函数
| 函数 | 圈复杂度 | 认知复杂度 | 行数 | 循环 | 分配数 | 风险等级 |
|------|----------|------------|------|------|--------|----------|
| **`build_figure_inventory`** | **193** | **546** | **1663** | **51** | **53** | 🔶🔶🔶 极高 |
| `_build_composite_parent_figure_groups_visual_only` | 28 | 78 | 142 | 9 | 5 | 🔶 中 |
| `_infer_missing_main_figure_numbers` | 28 | 46 | 192 | 6 | 0 | 🔹 低 |
| `_build_dense_composite_parent_candidates` | 18 | 41 | 97 | 9 | 6 | 🔹 低 |
**重点**: `build_figure_inventory` 是冷读中**最大的结构风险**。1663 行193 圈复杂度546 认知复杂度51 个循环53 次循环内分配。它串联了 5 个连续的回退阶段preproof legend bundling → previous-page locator bridge → sequential → group-aware fallback → 合成父图)。每个阶段都有自己的条件逻辑和数据变换——**所有阶段组合的正确性几乎没有测试能覆盖**。
---
## 渲染 (ocr_render.py) — 性能热点
| 函数 | 圈复杂度 | 认知复杂度 | 行数 | 循环 | 分配数 | 风险等级 |
|------|----------|------------|------|------|--------|----------|
| **`render_fulltext_markdown`** | **144** | **419** | **655** | **39** | **45** | 🔶🔶 高 |
| `_reorder_tail_run` | 39 | 88 | 195 | 5 | **22** | 🔶 中 |
| `_build_heading_style_profiles` | 21 | 64 | 112 | 7 | 6 | 🔹 低 |
**重点**: `render_fulltext_markdown` — 45 次循环内分配对论文生成的峰值内存和性能有影响。`_reorder_tail_run` 195 行中有 22 次 alloc多个列表推导式 + append
---
## 角色分配 (ocr_roles.py)
| 函数 | 圈复杂度 | 认知复杂度 | 风险等级 |
|------|----------|------------|----------|
| `assign_block_role` | **100** | **198** | 🔶 中 |
| `resolve_final_role` | 22 | 53 | 🔹 低 |
`assign_block_role` 是角色的根入口——其缺陷会逐级传播到版面分析和图片匹配中。
---
## 结构门控 (ocr_structural_gate.py)
| 函数 | 圈复杂度 | 认知复杂度 | 行为数 | 风险等级 |
|------|----------|------------|--------|----------|
| `resolve_verified_role` | 43 | 83 | 248 | 🔶 中 |
| `build_verified_reference_zone_from_artifacts` | 29 | 60 | 102 | 🔹 低 |
| `build_document_abstract_span` | 17 | 32 | 136 | 🔹 低 |
---
## 排版 (OCR Families)
| 函数 | 圈复杂度 | 认知复杂度 | 风险等级 |
|------|----------|------------|----------|
| `discover_body_family_anchor` | 6 | 12 | ✅ 低 |
| `discover_reference_family_anchor` | 6 | 14 | ✅ 低 |
---
## 递归
仅有一个递归函数:`ocr_doctor`(带守卫)✅ 无未保护的递归。
---
## 总结
| 风险等级 | 数量 | 关键对应 |
|----------|------|----------|
| 🔶🔶🔶 极高 | 1 | `build_figure_inventory` — 1663行巨型函数 |
| 🔶🔶 高 | 2 | `render_fulltext_markdown` (45 alloc_in_loop), `rescue_roles_with_document_context` (4.5x 认知/圈复杂度比) |
| 🔶 中 | 6 | 版面分析 4 个 + 角色 1 个 + 门控 1 个 |
| 🔹 低 | 5 | 已规划/正常范围内 |
**非风险项(已验证)**: 三个计划内修复覆盖的漏洞、7% body_anchor 失败率被 fallback 兜底、排版分类保守设计、阅读顺序正确、无递归循环。
**新发现的最大风险**: `build_figure_inventory` 的结构复杂度——不是 Bug 问题而是可维护性/变更安全性问题。如果在其中修东西需要极谨慎。

View file

@ -0,0 +1,880 @@
# Worktree cleanup archive — feat-ocr-structured-pipeline
- Path: `D:/L/Med/Research/99_System/LiteraturePipeline/github-release/.worktrees/feat-ocr-structured-pipeline`
- Branch: `feat/ocr-structured-pipeline`
- HEAD: `6d4ecc3 fix: skip single-block unresolved clusters — lonely subpanel assets are not clusters`
- Branches containing HEAD:
```
"feat/ocr-structured-pipeline"
"master"
```
## git status --short
```
M paperforge/worker/ocr.py
M paperforge/worker/ocr_blocks.py
M paperforge/worker/ocr_figures.py
M paperforge/worker/ocr_health.py
M paperforge/worker/ocr_objects.py
M paperforge/worker/ocr_roles.py
M tests/test_ocr_figures.py
M tests/test_ocr_metadata.py
?? docs/superpowers/plans/2026-06-06-ocr-convergence-phase-2-plan.md
?? docs/superpowers/plans/2026-06-06-ocr-dead-code-and-closure-plan.md
?? docs/superpowers/plans/2026-06-06-ocr-final-polish-and-math-normalization-plan.md
?? docs/superpowers/plans/2026-06-06-ocr-full-pipeline-convergence-plan.md
?? docs/superpowers/plans/2026-06-06-ocr-heuristic-gating-remediation-plan.md
?? docs/superpowers/plans/2026-06-06-ocr-layout-aware-tail-reading-plan.md
?? docs/superpowers/plans/2026-06-07-ocr-cross-layout-hardening-plan.md
?? docs/superpowers/plans/2026-06-07-ocr-real-paper-regression-closure-plan.md
?? docs/superpowers/plans/2026-06-07-ocr-structural-convergence-master-plan.md
?? docs/superpowers/plans/2026-06-07-tsckavis-structural-fix-plan.md
?? docs/superpowers/plans/2026-06-08-ocr-decision-log-plan.md
?? docs/superpowers/plans/2026-06-08-ocr-error-taxonomy-plan.md
?? docs/superpowers/plans/2026-06-08-ocr-evidence-scorer-plan.md
?? docs/superpowers/plans/2026-06-08-ocr-p0-bugfix-plan.md
?? docs/superpowers/plans/2026-06-08-ocr-real-fixtures-ci-plan.md
?? docs/superpowers/plans/2026-06-08-ocr-route-audit-plan.md
?? docs/superpowers/plans/2026-06-08-ocr-v1-convergence-master-plan.md
?? docs/superpowers/reports/
?? scripts/dev/audit_10_papers.py
?? scripts/dev/check_m36_figures.py
?? scripts/dev/check_rebuild_results.py
?? scripts/dev/rebuild_10_papers.py
```
## Untracked files
```
docs/superpowers/plans/2026-06-06-ocr-convergence-phase-2-plan.md
docs/superpowers/plans/2026-06-06-ocr-dead-code-and-closure-plan.md
docs/superpowers/plans/2026-06-06-ocr-final-polish-and-math-normalization-plan.md
docs/superpowers/plans/2026-06-06-ocr-full-pipeline-convergence-plan.md
docs/superpowers/plans/2026-06-06-ocr-heuristic-gating-remediation-plan.md
docs/superpowers/plans/2026-06-06-ocr-layout-aware-tail-reading-plan.md
docs/superpowers/plans/2026-06-07-ocr-cross-layout-hardening-plan.md
docs/superpowers/plans/2026-06-07-ocr-real-paper-regression-closure-plan.md
docs/superpowers/plans/2026-06-07-ocr-structural-convergence-master-plan.md
docs/superpowers/plans/2026-06-07-tsckavis-structural-fix-plan.md
docs/superpowers/plans/2026-06-08-ocr-decision-log-plan.md
docs/superpowers/plans/2026-06-08-ocr-error-taxonomy-plan.md
docs/superpowers/plans/2026-06-08-ocr-evidence-scorer-plan.md
docs/superpowers/plans/2026-06-08-ocr-p0-bugfix-plan.md
docs/superpowers/plans/2026-06-08-ocr-real-fixtures-ci-plan.md
docs/superpowers/plans/2026-06-08-ocr-route-audit-plan.md
docs/superpowers/plans/2026-06-08-ocr-v1-convergence-master-plan.md
docs/superpowers/reports/2026-06-07-cross-layout-validation-report.md
scripts/dev/audit_10_papers.py
scripts/dev/check_m36_figures.py
scripts/dev/check_rebuild_results.py
scripts/dev/rebuild_10_papers.py
```
## Binary-safe diff
```diff
diff --git a/paperforge/worker/ocr.py b/paperforge/worker/ocr.py
index bb47777..f4f4926 100644
--- a/paperforge/worker/ocr.py
+++ b/paperforge/worker/ocr.py
@@ -1831,7 +1831,7 @@ def postprocess_ocr_result(vault: Path, key: str, all_results: list[dict]) -> tu
metadata_dir.mkdir(parents=True, exist_ok=True)
frontmatter_candidates = extract_frontmatter_candidates(artifacts.blocks_structured)
page1_raw = [b for b in all_raw_blocks if b.get("page") == 1] if all_raw_blocks else None
- resolved = resolve_metadata(source_meta, frontmatter_candidates, page1_blocks=page1_raw)
+ resolved = resolve_metadata(source_meta, frontmatter_candidates, page1_blocks=page1_raw, structured_blocks=structured)
write_resolved_metadata(metadata_dir / "resolved_metadata.json", resolved)
# --- Phase 2: figure inventory ---
diff --git a/paperforge/worker/ocr_blocks.py b/paperforge/worker/ocr_blocks.py
index 3f8b8b3..a31d7f4 100644
--- a/paperforge/worker/ocr_blocks.py
+++ b/paperforge/worker/ocr_blocks.py
@@ -54,9 +54,9 @@ def build_structured_blocks(
)
render_default = role.role not in ({"noise", "unknown_structural"} | _CANDIDATE_ROLES)
index_default = role.role not in _CANDIDATE_ROLES
- if role.role in {"noise", "page_header", "page_footer", "frontmatter_noise", "non_body_insert", "structured_insert"}:
+ if role.role in {"noise", "page_header", "page_footer", "frontmatter_noise", "non_body_insert", "structured_insert", "figure_inner_text"}:
render_default = False
- if role.role in {"noise", "frontmatter_noise", "table_html", "non_body_insert", "structured_insert"}:
+ if role.role in {"noise", "frontmatter_noise", "table_html", "non_body_insert", "structured_insert", "figure_inner_text"}:
index_default = False
row = {
"paper_id": block["paper_id"],
@@ -141,10 +141,10 @@ def build_structured_blocks(
row["index_default"] = False
else:
row["render_default"] = role not in ({"noise", "unknown_structural"} | _CANDIDATE_ROLES)
- if role in {"noise", "page_header", "page_footer", "frontmatter_noise", "non_body_insert", "structured_insert"}:
+ if role in {"noise", "page_header", "page_footer", "frontmatter_noise", "non_body_insert", "structured_insert", "figure_inner_text"}:
row["render_default"] = False
row["index_default"] = role not in _CANDIDATE_ROLES
- if role in {"noise", "frontmatter_noise", "table_html", "non_body_insert", "structured_insert"}:
+ if role in {"noise", "frontmatter_noise", "table_html", "non_body_insert", "structured_insert", "figure_inner_text"}:
row["index_default"] = False
# Persist document structure artifact for downstream debugging
diff --git a/paperforge/worker/ocr_figures.py b/paperforge/worker/ocr_figures.py
index 5575655..51fffe5 100644
--- a/paperforge/worker/ocr_figures.py
+++ b/paperforge/worker/ocr_figures.py
@@ -54,22 +54,18 @@ def _looks_like_inline_figure_mention(text: str) -> bool:
if not re.search(r"\bfi(?:g(?:ure)?\.?\s*\d+)", lower):
return False
- # Explicitly NOT inline: Frontiers format FIGURE N | ...
if re.match(r"^figure\s+\d+[a-z]?\s*\|", t, re.I):
return False
- # "as shown in Figure X" / "shown in Figure X" / "see Figure X"
if re.search(r"\b(as shown in|shown in|see |according to|consistent with)\s+(fig(?:ure)?\.?\s*\d+)", lower):
return True
- # Long sentence with a prose verb
words = t.split()
if len(words) >= 10 and any(re.search(rf"\b{v}\b", lower) for v in _INLINE_FIGURE_MENTION_VERBS):
return True
return False
-
def _extract_figure_number(text: str) -> int | None:
m = _FIGURE_NUMBER_PATTERN.search(text)
if m:
@@ -107,6 +103,10 @@ def _centroid_y(bbox: list[float]) -> float:
return (bbox[1] + bbox[3]) / 2
+def _asset_key(block: dict) -> str:
+ return f"{int(block.get('page', 0) or 0)}:{block.get('block_id', '')}"
+
+
def _looks_like_figure_narrative_prose(text: str) -> bool:
if not text:
return False
@@ -303,12 +303,24 @@ def _precaption_media_region(media_cluster: list[dict], caption_block: dict) ->
def _compute_candidate_figure_regions(blocks: list[dict], page_width: float = 1200) -> list[dict]:
clusters = _media_clusters(blocks, page_width)
captions = [b for b in blocks if b.get("role") == "figure_caption"]
+
+ def _is_embedded_caption_like(block: dict) -> bool:
+ text = str(block.get("text", "") or "").strip()
+ if not text or _FIGURE_NUMBER_PATTERN.search(text):
+ return False
+ return is_embedded_figure_text(block, blocks, page_width=page_width)
+
regions: list[dict] = []
for i, cluster in enumerate(clusters):
cluster_bbox = _cluster_bbox([b.get("bbox", [0, 0, 0, 0]) for b in cluster])
page = cluster[0].get("page", 0)
attached: list[dict] = []
unvalidated: list[dict] = []
+ page_blocks = [b for b in blocks if int(b.get("page", 0) or 0) == page]
+ panel_like = any(
+ _PANEL_LABEL_PATTERN.match(str(b.get("text", "") or "").strip()) or _is_embedded_caption_like(b)
+ for b in page_blocks
+ )
for cap in captions:
if cap.get("page", 0) != page:
continue
@@ -324,11 +336,97 @@ def _compute_candidate_figure_regions(blocks: list[dict], page_width: float = 12
"media_blocks": cluster,
"attached_captions": attached,
"unvalidated_captions": unvalidated,
+ "region_type": "cluster",
+ "panel_like": panel_like,
}
)
+ regions.sort(key=lambda r: (r.get("page", 0), r.get("cluster_bbox", [0, 0, 0, 0])[1]))
return regions
+def _score_legend_region(
+ legend: dict,
+ region: dict,
+ *,
+ caption_score: dict,
+ same_page_region_exists: bool,
+) -> dict:
+ proxy_asset = {
+ "block_id": region.get("region_id", ""),
+ "page": region.get("page", 0),
+ "bbox": region.get("cluster_bbox", [0, 0, 0, 0]),
+ }
+ if legend.get("page") == region.get("page"):
+ legend_bbox = legend.get("bbox") or legend.get("block_bbox") or [0, 0, 0, 0]
+ if len(region.get("media_blocks", [])) > 1 and len(legend_bbox) >= 4:
+ legend_top = legend_bbox[1]
+ legend_bottom = legend_bbox[3]
+ asset_tops = []
+ asset_bottoms = []
+ for asset in region.get("media_blocks", []):
+ ab = asset.get("bbox") or asset.get("block_bbox") or [0, 0, 0, 0]
+ if len(ab) >= 4:
+ asset_tops.append(ab[1])
+ asset_bottoms.append(ab[3])
+ if asset_tops and asset_bottoms:
+ has_asset_above = any(bottom <= legend_top for bottom in asset_bottoms)
+ has_asset_below = any(top >= legend_bottom for top in asset_tops)
+ if has_asset_above and has_asset_below and region.get("region_type") != "page_plate":
+ return {
+ "score": 0.45,
+ "matched_asset_id": region.get("region_id", ""),
+ "decision": "ambiguous",
+ "evidence": ["caption_between_assets"],
+ }
+ return score_figure_match(legend, proxy_asset, caption_score=caption_score)
+
+ if same_page_region_exists:
+ return {
+ "score": 0.0,
+ "matched_asset_id": region.get("region_id", ""),
+ "decision": "rejected",
+ "evidence": ["same_page_region_exists"],
+ }
+
+ if not region.get("panel_like") and len(region.get("media_blocks", [])) <= 1:
+ return {
+ "score": 0.0,
+ "matched_asset_id": region.get("region_id", ""),
+ "decision": "rejected",
+ "evidence": ["adjacent_match_requires_panel_like_cluster"],
+ }
+
+ page_gap = abs(int(legend.get("page", 0) or 0) - int(region.get("page", 0) or 0))
+ if page_gap != 1:
+ return {
+ "score": 0.0,
+ "matched_asset_id": region.get("region_id", ""),
+ "decision": "rejected",
+ "evidence": ["non_adjacent_region"],
+ }
+
+ score = min(0.15, float(caption_score.get("score", 0.0)) * 0.15) + 0.45
+ evidence = ["adjacent_page_region"]
+ legend_bbox = legend.get("bbox") or legend.get("block_bbox") or [0, 0, 0, 0]
+ if len(legend_bbox) >= 4:
+ if region.get("page", 0) < legend.get("page", 0) and legend_bbox[1] < 320:
+ score += 0.15
+ evidence.append("caption_page_after_figure")
+ elif region.get("page", 0) > legend.get("page", 0) and legend_bbox[3] > 900:
+ score += 0.15
+ evidence.append("caption_page_before_figure")
+ if region.get("panel_like"):
+ score += 0.1
+ evidence.append("panel_like_region")
+ score = max(0.0, min(1.0, score))
+ return {
+ "score": score,
+ "matched_asset_id": region.get("region_id", ""),
+ "decision": "matched_adjacent_region" if score >= 0.6 else "ambiguous" if score >= 0.4 else "rejected",
+ "evidence": evidence,
+ }
+
+
def is_embedded_figure_text(block: dict, all_blocks: list[dict], page_width: float = 1200) -> bool:
block_bbox = block.get("bbox") or block.get("block_bbox")
if not block_bbox or len(block_bbox) < 4:
@@ -373,6 +471,8 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
if block.get("page_width"):
page_width = float(block["page_width"])
+ candidate_regions = _compute_candidate_figure_regions(structured_blocks, page_width)
+
for block in structured_blocks:
role = block.get("role", "")
if block.get("_non_body_media") or role == "non_body_insert":
@@ -381,14 +481,21 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
if _PANEL_LABEL_PATTERN.match(str(block.get("text", "")).strip()):
continue
if role in ("figure_caption", "figure_caption_candidate"):
+ if (
+ _extract_figure_number(block.get("text", "")) is None
+ and is_embedded_figure_text(block, structured_blocks, page_width)
+ ):
+ continue
if _is_body_mention(block):
continue
+ if role == "figure_caption_candidate" and _looks_like_inline_figure_mention(block.get("text", "")):
+ unmatched_legends.append(block)
+ continue
if role == "figure_caption_candidate" and _looks_like_figure_narrative_prose(block.get("text", "")):
continue
if not _is_formal_legend(block.get("text", ""), block, page_width):
block["caption_score"] = score_figure_caption(
block, nearby_media=False, caption_style_match=False,
- body_prose_likelihood=_looks_like_inline_figure_mention(block.get("text", "")),
)
rejected_legends.append(block)
else:
@@ -438,100 +545,160 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
deduped_legends.append(legend)
ordered_legends = deduped_legends
- used_asset_indices: set[int] = set()
+ used_asset_ids: set[str] = set()
+ used_region_ids: set[str] = set()
ambiguous_figures: list[dict] = []
for legend in ordered_legends:
legend_page = legend.get("page", 0)
legend_text = legend.get("text", "")
fig_num = _extract_figure_number(legend_text)
- body_prose_likelihood = _looks_like_inline_figure_mention(legend_text)
-
caption_score = score_figure_caption(
legend,
nearby_media=any(a.get("page", 0) == legend_page for a in assets),
caption_style_match=_caption_style_match(legend, structured_blocks),
- body_prose_likelihood=body_prose_likelihood,
)
candidates = []
- for ai, asset in enumerate(assets):
- if ai in used_asset_indices or asset.get("page", 0) != legend_page:
+ had_ambiguous_region_candidate = False
+ same_page_region_exists = any(
+ r.get("page", 0) == legend_page and r.get("region_id", "") not in used_region_ids
+ for r in candidate_regions
+ )
+ for region in candidate_regions:
+ region_id = region.get("region_id", "")
+ if not region_id or region_id in used_region_ids:
+ continue
+ region_asset_ids = {_asset_key(b) for b in region.get("media_blocks", [])}
+ if region_asset_ids & used_asset_ids:
continue
- match_score = score_figure_match(legend, asset, caption_score=caption_score)
+ match_score = _score_legend_region(
+ legend,
+ region,
+ caption_score=caption_score,
+ same_page_region_exists=same_page_region_exists,
+ )
+ if match_score["decision"] == "ambiguous":
+ had_ambiguous_region_candidate = True
if match_score["decision"] != "rejected":
- candidates.append((ai, asset, match_score))
- candidates.sort(key=lambda item: item[2]["score"], reverse=True)
+ candidates.append((region, match_score))
+ candidates.sort(key=lambda item: item[1]["score"], reverse=True)
matched_assets = []
region_match = None
ambiguous = False
if candidates:
- top_score = candidates[0][2]["score"]
- close = [item for item in candidates if top_score - item[2]["score"] < 0.15]
+ top_score = candidates[0][1]["score"]
+ close = [item for item in candidates if top_score - item[1]["score"] < 0.15]
if top_score < 0.4:
matched_assets = []
elif len(close) > 1:
- # Secondary verification: pick the one in the correct column
- legend_bb = legend.get("bbox") or legend.get("block_bbox") or [0,0,0,0]
- lcx = (legend_bb[0] + legend_bb[2]) / 2 if len(legend_bb) >= 4 else 0
- best = close[0]
- best_col_match = False
- for ci, ca, cs in close:
- ab = ca.get("bbox") or ca.get("block_bbox") or [0,0,0,0]
- acx = (ab[0] + ab[2]) / 2 if len(ab) >= 4 else 0
- ca_col_ok = abs(lcx - acx) < abs(lcx - (best[1].get("bbox",[0,0,0,0])[0] + best[1].get("bbox",[0,0,0,0])[2])/2)
- if ca_col_ok:
- best = (ci, ca, cs)
- best_col_match = True
- break
- if best_col_match:
- best_idx, best_asset, best_score = best
- matched_assets = [best_asset]
- used_asset_indices.add(best_idx)
- region_match = {"media_blocks": [best_asset], "match_score": best_score}
+ same_page = all(item[0].get("page", 0) == close[0][0].get("page", 0) for item in close)
+ if close[0][0].get("panel_like") and top_score >= 0.7:
+ best_region, best_score = candidates[0]
+ matched_assets = list(best_region.get("media_blocks", []))
+ used_region_ids.add(best_region.get("region_id", ""))
+ used_asset_ids.update(_asset_key(a) for a in matched_assets)
+ region_match = {
+ "page": best_region.get("page", legend_page),
+ "media_blocks": matched_assets,
+ "cluster_bbox": best_region.get("cluster_bbox", [0, 0, 0, 0]),
+ "match_score": best_score,
+ }
else:
- ambiguous_figures.append({
- "legend_block_id": legend.get("block_id", ""),
- "page": legend_page,
- "caption_score": caption_score,
- "candidates": [
- {"asset_block_id": a.get("block_id", ""), "match_score": s}
- for _, a, s in close
- ],
- })
- ambiguous = True
- matched_assets = []
+ legend_bb = legend.get("bbox") or legend.get("block_bbox") or [0,0,0,0]
+ lcx = (legend_bb[0] + legend_bb[2]) / 2 if len(legend_bb) >= 4 else 0
+ best = close[0]
+ best_col_match = False
+ for ca, cs in close:
+ ab = ca.get("cluster_bbox") or [0,0,0,0]
+ acx = (ab[0] + ab[2]) / 2 if len(ab) >= 4 else 0
+ best_ab = best[0].get("cluster_bbox") or [0,0,0,0]
+ ca_col_ok = abs(lcx - acx) < abs(lcx - (best_ab[0] + best_ab[2]) / 2)
+ if ca_col_ok:
+ best = (ca, cs)
+ best_col_match = True
+ break
+ if best_col_match and best[0].get("panel_like") and same_page:
+ best_region, best_score = best
+ matched_assets = list(best_region.get("media_blocks", []))
+ used_region_ids.add(best_region.get("region_id", ""))
+ used_asset_ids.update(_asset_key(a) for a in matched_assets)
+ region_match = {
+ "page": best_region.get("page", legend_page),
+ "media_blocks": matched_assets,
+ "cluster_bbox": best_region.get("cluster_bbox", [0, 0, 0, 0]),
+ "match_score": best_score,
+ }
+ else:
+ ambiguous_figures.append({
+ "legend_block_id": legend.get("block_id", ""),
+ "page": legend_page,
+ "caption_score": caption_score,
+ "candidates": [
+ {"region_id": a.get("region_id", ""), "match_score": s}
+ for a, s in close
+ ],
+ })
+ ambiguous = True
+ matched_assets = []
else:
- best_idx, best_asset, best_score = candidates[0]
- matched_assets = [best_asset]
- used_asset_indices.add(best_idx)
- region_match = {"media_blocks": [best_asset], "match_score": best_score}
+ best_region, best_score = candidates[0]
+ if str(best_score.get("decision", "")).startswith("matched"):
+ matched_assets = list(best_region.get("media_blocks", []))
+ used_region_ids.add(best_region.get("region_id", ""))
+ used_asset_ids.update(_asset_key(a) for a in matched_assets)
+ region_match = {
+ "page": best_region.get("page", legend_page),
+ "media_blocks": matched_assets,
+ "cluster_bbox": best_region.get("cluster_bbox", [0, 0, 0, 0]),
+ "match_score": best_score,
+ }
+ else:
+ matched_assets = []
# Fallback: if no match found but legends == assets on the same page,
# assign sequentially by vertical position
- if not matched_assets and fig_num is not None:
- page_assets = [
- (ai, a) for ai, a in enumerate(assets)
- if ai not in used_asset_indices and a.get("page", 0) == legend_page
+ if not matched_assets and fig_num is not None and not had_ambiguous_region_candidate:
+ page_regions = [
+ r for r in candidate_regions
+ if r.get("region_id", "") not in used_region_ids and r.get("page", 0) == legend_page
]
page_legends = [
l for l in ordered_legends
if l is not legend and _extract_figure_number(l.get("text", "")) is not None
and l.get("page", 0) == legend_page
]
- if page_assets and len(page_assets) >= len(page_legends) + 1:
- page_assets.sort(key=lambda item: (item[1].get("bbox",[0,0,0,0])[1] if len(item[1].get("bbox",[]))>=4 else 0))
- # Count how many matched legends already consumed assets on this page
- consumed_on_page = sum(1 for i in used_asset_indices if assets[i].get("page",0) == legend_page)
- asset_idx = min(consumed_on_page, len(page_assets) - 1)
- best_idx, best_asset = page_assets[asset_idx]
- matched_assets = [best_asset]
- used_asset_indices.add(best_idx)
- region_match = {"media_blocks": [best_asset], "match_score": {"score": 0.5, "decision": "matched_fallback", "evidence": ["sequential_fallback"]}}
+ if page_regions and len(page_regions) >= len(page_legends) + 1:
+ page_regions.sort(key=lambda r: (r.get("cluster_bbox",[0,0,0,0])[1] if len(r.get("cluster_bbox",[]))>=4 else 0))
+ consumed_on_page = sum(1 for rid in used_region_ids if any(r.get("page",0) == legend_page and r.get("region_id","") == rid for r in candidate_regions))
+ region_idx = min(consumed_on_page, len(page_regions) - 1)
+ best_region = page_regions[region_idx]
+ matched_assets = list(best_region.get("media_blocks", []))
+ used_region_ids.add(best_region.get("region_id", ""))
+ used_asset_ids.update(_asset_key(a) for a in matched_assets)
+ region_match = {
+ "page": best_region.get("page", legend_page),
+ "media_blocks": matched_assets,
+ "cluster_bbox": best_region.get("cluster_bbox", [0, 0, 0, 0]),
+ "match_score": {"score": 0.5, "decision": "matched_fallback", "evidence": ["sequential_fallback"]},
+ }
is_legend_only = len(matched_assets) == 0
+ if is_legend_only and had_ambiguous_region_candidate:
+ ambiguous = True
+ if not any(item.get("legend_block_id", "") == legend.get("block_id", "") for item in ambiguous_figures):
+ ambiguous_figures.append({
+ "legend_block_id": legend.get("block_id", ""),
+ "page": legend_page,
+ "caption_score": caption_score,
+ "candidates": [
+ {"region_id": region.get("region_id", ""), "match_score": match_score}
+ for region, match_score in candidates
+ if match_score.get("decision") == "ambiguous"
+ ],
+ })
if caption_score.get("score", 0.0) < 0.4:
unmatched_legends.append(legend)
@@ -551,13 +718,8 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
"page": legend_page,
"text": legend_text,
"figure_number": fig_num,
- "matched_assets": [
- {
- "block_id": a.get("block_id", ""),
- "bbox": a.get("bbox", [0, 0, 0, 0]),
- }
- for a in matched_assets
- ],
+ "asset_page": region_match.get("page", legend_page) if region_match is not None else legend_page,
+ "matched_assets": [{"block_id": a.get("block_id", ""), "bbox": a.get("bbox", [0, 0, 0, 0])} for a in matched_assets],
"confidence": match_score["score"],
"match_score": match_score,
"flags": [] if not is_legend_only else ["legend_only"],
@@ -570,8 +732,45 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
if is_legend_only:
unmatched_legends.append(legend)
- for i, asset in enumerate(assets):
- if i not in used_asset_indices:
+ # Post-match expansion: iteratively pull in nearby unused assets on the
+ # same page so chained multi-panel layouts can merge into one figure.
+ for entry in matched_figures:
+ fig_page = entry.get("page", 0)
+ mas = entry.get("matched_assets", [])
+ if not mas or entry.get("cluster_bbox"):
+ continue
+ matched = {m.get("block_id", ""): m for m in mas}
+ changed = True
+ while changed:
+ changed = False
+ current_bboxes = [m.get("bbox", [0, 0, 0, 0]) for m in matched.values() if len(m.get("bbox", [])) >= 4]
+ if not current_bboxes:
+ break
+ fig_x1, fig_y1, fig_x2, fig_y2 = _cluster_bbox(current_bboxes)
+ for a in assets:
+ aid = a.get("block_id", "")
+ if not aid or _asset_key(a) in used_asset_ids or aid in matched or a.get("page", 0) != fig_page:
+ continue
+ ab = a.get("bbox") or a.get("block_bbox") or [0, 0, 0, 0]
+ if len(ab) < 4:
+ continue
+ ax1, ay1, ax2, ay2 = ab[0], ab[1], ab[2], ab[3]
+ x_overlap = max(0, min(fig_x2, ax2) - max(fig_x1, ax1))
+ x_span = max(1, max(fig_x2, ax2) - min(fig_x1, ax1))
+ overlap_ratio = x_overlap / x_span
+ y_gap = max(ay1 - fig_y2, fig_y1 - ay2, 0)
+ page_h = a.get("page_height") or 1700
+ if overlap_ratio >= 0.25 and y_gap < page_h * 0.18:
+ matched[aid] = {"block_id": aid, "bbox": a.get("bbox", [0, 0, 0, 0])}
+ used_asset_ids.add(_asset_key(a))
+ changed = True
+ if len(matched) > len(mas):
+ entry["matched_assets"] = list(matched.values())
+ entry["cluster_bbox"] = _cluster_bbox([m.get("bbox", [0, 0, 0, 0]) for m in matched.values()])
+ entry["flags"] = [f for f in entry.get("flags", []) if f != "legend_only"]
+
+ for asset in assets:
+ if _asset_key(asset) not in used_asset_ids:
unmatched_assets.append(asset)
# Build unresolved clusters: spatial clusters of unmatched assets on
diff --git a/paperforge/worker/ocr_health.py b/paperforge/worker/ocr_health.py
index b4127b0..7a749fc 100644
--- a/paperforge/worker/ocr_health.py
+++ b/paperforge/worker/ocr_health.py
@@ -150,6 +150,16 @@ def build_ocr_health(
}
report.update(decision_summary)
+ from paperforge.worker.ocr_roles import is_preproof_marker
+
+ preproof_pages = list({
+ b.get("page") for b in structured_blocks
+ if b.get("role") == "frontmatter_noise"
+ and is_preproof_marker(str(b.get("text", "") or b.get("block_content", "") or ""))
+ })
+ report["preproof_marker_detected"] = len(preproof_pages) > 0
+ report["preproof_marker_pages"] = preproof_pages
+
degraded_reasons = []
if span.get("coverage_quality", "weak") == "weak":
degraded_reasons.append(f"weak span coverage ({span.get('coverage_ratio', 0):.0%})")
diff --git a/paperforge/worker/ocr_objects.py b/paperforge/worker/ocr_objects.py
index 58d4cac..0350ef5 100644
--- a/paperforge/worker/ocr_objects.py
+++ b/paperforge/worker/ocr_objects.py
@@ -205,7 +205,8 @@ def extract_and_write_objects(
fig_id = match.get("figure_id", f"figure_{i + 1:03d}")
caption_text = match.get("text", "")
page = match.get("page", 0)
- page_width, page_height = _page_dims(page)
+ asset_page = match.get("asset_page", page)
+ page_width, page_height = _page_dims(asset_page)
asset_path_rel = f"assets/figures/{fig_id}.jpg"
asset_path_abs = figures_asset_dir / f"{fig_id}.jpg"
@@ -213,7 +214,7 @@ def extract_and_write_objects(
cluster_bbox = match.get("cluster_bbox")
if cluster_bbox and all(v > 0 for v in cluster_bbox):
was_cropped = _crop_asset_from_pdf(
- pdf_path, page, cluster_bbox, asset_path_abs,
+ pdf_path, asset_page, cluster_bbox, asset_path_abs,
page_width=page_width, page_height=page_height,
page_cache_dir=page_cache_dir,
)
@@ -222,7 +223,7 @@ def extract_and_write_objects(
bbox = asset_info.get("bbox", [0, 0, 0, 0])
if pdf_path and bbox and all(v > 0 for v in bbox) and _crop_asset_from_pdf(
pdf_path,
- page,
+ asset_page,
bbox,
asset_path_abs,
page_width=page_width,
diff --git a/paperforge/worker/ocr_roles.py b/paperforge/worker/ocr_roles.py
index bcf37e7..edf534d 100644
--- a/paperforge/worker/ocr_roles.py
+++ b/paperforge/worker/ocr_roles.py
@@ -95,7 +95,7 @@ _FRONTIERS_FIGURE_TITLE_PATTERN = re.compile(
)
_PANEL_LABEL_PATTERN = re.compile(
- r"^\(?[A-Z]\)?[\.:]?$",
+ r"^\(?[A-Za-z]\)?[\.:]?$",
)
_ROMAN_SECTION_PATTERN = re.compile(
diff --git a/tests/test_ocr_figures.py b/tests/test_ocr_figures.py
index 9231892..f093e10 100644
--- a/tests/test_ocr_figures.py
+++ b/tests/test_ocr_figures.py
@@ -189,6 +189,159 @@ def test_compute_candidate_figure_regions_caption_before_media() -> None:
assert len(regions[0]["attached_captions"]) == 0
+def test_figure_inventory_ignores_lowercase_panel_labels_and_embedded_inner_text() -> None:
+ from paperforge.worker.ocr_figures import build_figure_inventory
+
+ structured_blocks = [
+ {
+ "paper_id": "K001",
+ "page": 10,
+ "block_id": "p10_b1",
+ "role": "figure_caption",
+ "raw_label": "figure_title",
+ "text": "(a)",
+ "bbox": [80, 80, 110, 110],
+ "page_width": 1200,
+ "page_height": 1600,
+ },
+ {
+ "paper_id": "K001",
+ "page": 10,
+ "block_id": "p10_b2",
+ "role": "figure_caption",
+ "raw_label": "figure_title",
+ "text": "0 mg/mL",
+ "bbox": [120, 220, 200, 240],
+ "page_width": 1200,
+ "page_height": 1600,
+ },
+ {
+ "paper_id": "K001",
+ "page": 10,
+ "block_id": "p10_b3",
+ "role": "figure_caption",
+ "raw_label": "figure_title",
+ "text": "Fig. 2. Multi-panel figure with one formal legend.",
+ "bbox": [60, 1300, 1140, 1450],
+ "page_width": 1200,
+ "page_height": 1600,
+ },
+ {
+ "paper_id": "K001",
+ "page": 10,
+ "block_id": "p10_b4",
+ "role": "media_asset",
+ "raw_label": "image",
+ "text": "",
+ "bbox": [80, 120, 300, 420],
+ "page_width": 1200,
+ "page_height": 1600,
+ },
+ {
+ "paper_id": "K001",
+ "page": 10,
+ "block_id": "p10_b5",
+ "role": "media_asset",
+ "raw_label": "image",
+ "text": "",
+ "bbox": [320, 120, 540, 420],
+ "page_width": 1200,
+ "page_height": 1600,
+ },
+ {
+ "paper_id": "K001",
+ "page": 10,
+ "block_id": "p10_b6",
+ "role": "media_asset",
+ "raw_label": "image",
+ "text": "",
+ "bbox": [80, 450, 300, 780],
+ "page_width": 1200,
+ "page_height": 1600,
+ },
+ ]
+
+ inventory = build_figure_inventory(structured_blocks)
+
+ assert inventory["official_figure_count"] == 1
+ assert len(inventory["matched_figures"]) == 1
+ assert inventory["matched_figures"][0]["figure_number"] == 2
+ assert len(inventory["matched_figures"][0]["matched_assets"]) == 3
+ assert not any(leg.get("text") == "(a)" for leg in inventory["figure_legends"])
+ assert not any(leg.get("text") == "0 mg/mL" for leg in inventory["figure_legends"])
+
+
+def test_figure_inventory_matches_adjacent_caption_page_to_merged_panel_region() -> None:
+ from paperforge.worker.ocr_figures import build_figure_inventory
+
+ structured_blocks = [
+ {
+ "paper_id": "K002",
+ "page": 20,
+ "block_id": "p20_b1",
+ "role": "media_asset",
+ "raw_label": "image",
+ "text": "",
+ "bbox": [120, 120, 380, 420],
+ "page_width": 1200,
+ "page_height": 1600,
+ },
+ {
+ "paper_id": "K002",
+ "page": 20,
+ "block_id": "p20_b2",
+ "role": "media_asset",
+ "raw_label": "chart",
+ "text": "",
+ "bbox": [420, 120, 760, 430],
+ "page_width": 1200,
+ "page_height": 1600,
+ },
+ {
+ "paper_id": "K002",
+ "page": 20,
+ "block_id": "p20_b3",
+ "role": "figure_caption",
+ "raw_label": "figure_title",
+ "text": "(a)",
+ "bbox": [110, 90, 140, 115],
+ "page_width": 1200,
+ "page_height": 1600,
+ },
+ {
+ "paper_id": "K002",
+ "page": 20,
+ "block_id": "p20_b4",
+ "role": "figure_caption",
+ "raw_label": "figure_title",
+ "text": "(b)",
+ "bbox": [410, 90, 440, 115],
+ "page_width": 1200,
+ "page_height": 1600,
+ },
+ {
+ "paper_id": "K002",
+ "page": 21,
+ "block_id": "p21_b1",
+ "role": "figure_caption",
+ "raw_label": "figure_title",
+ "text": "Fig. 3. Caption lives on the following page, but should still match the previous multi-panel figure page.",
+ "bbox": [80, 80, 1120, 260],
+ "page_width": 1200,
+ "page_height": 1600,
+ },
+ ]
+
+ inventory = build_figure_inventory(structured_blocks)
+
+ assert inventory["official_figure_count"] == 1
+ assert len(inventory["matched_figures"]) == 1
+ assert inventory["matched_figures"][0]["figure_number"] == 3
+ assert inventory["matched_figures"][0]["page"] == 21
+ assert len(inventory["matched_figures"][0]["matched_assets"]) == 2
+ assert inventory["matched_figures"][0]["match_score"]["decision"] in {"matched", "matched_adjacent_region"}
+
+
# --- existing tests ---
diff --git a/tests/test_ocr_metadata.py b/tests/test_ocr_metadata.py
index 34d5744..20dee84 100644
--- a/tests/test_ocr_metadata.py
+++ b/tests/test_ocr_metadata.py
@@ -223,3 +223,50 @@ def test_normalize_author_name_strips_superscripts() -> None:
assert _normalize_author_name("Smith $^{1}") == "Smith"
assert _normalize_author_name("Ebrahim Esfandiari $^{1}$") == "Ebrahim Esfandiari"
+
+
+def test_initials_match_ami_yoo() -> None:
+ from paperforge.worker.ocr_metadata import _initials_match
+
+ assert _initials_match("A. Yoo", "Ami Yoo") is True
+
+
+def test_initials_match_w_h_marks() -> None:
+ from paperforge.worker.ocr_metadata import _initials_match
+
+ assert _initials_match("W. H. Marks", "William H. Marks") is True
+
+
+def test_initials_match_g_go() -> None:
+ from paperforge.worker.ocr_metadata import _initials_match
+
+ assert _initials_match("G. Go", "Gwangjun Go") is True
+
+
+def test_initials_no_match() -> None:
+ from paperforge.worker.ocr_metadata import _initials_match
+
+ assert _initials_match("A. Yoo", "John Smith") is False
+
+
+def test_get_ocr_author_names() -> None:
+ from paperforge.worker.ocr_metadata import _get_ocr_author_names
+
+ blocks = [
+ {"role": "authors", "text": "Ami Yoo, Gwangjun Go, Kim Tien Nguyen, Kyungmin Lee"},
+ {"role": "body_paragraph", "text": "Some body text"},
+ ]
+ names = _get_ocr_author_names(blocks)
+ assert "Ami Yoo" in names
+ assert "Gwangjun Go" in names
+ assert len(names) >= 3
+
+
+def test_ocr_authors_verified_by_first_author() -> None:
+ from paperforge.worker.ocr_metadata import resolve_metadata
+
+ source_meta = {"first_author": "A. Yoo", "title": "Test paper", "doi": "10.1234/test"}
+ blocks = [{"role": "authors", "text": "Ami Yoo, Gwangjun Go", "page": 1, "bbox": [100, 100, 500, 130]}]
+ result = resolve_metadata(source_meta, {}, structured_blocks=blocks)
+ assert "Ami Yoo" in result.get("authors", {}).get("value", [])
+ assert result.get("authors", {}).get("source") == "ocr_blocks_verified_by_first_author"
```

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,88 @@
# Worktree cleanup archive — paperforge-stabilization
- Path: `D:/L/Med/Research/99_System/LiteraturePipeline/github-release/.worktrees/paperforge-stabilization`
- Branch: `paperforge-stabilization`
- HEAD: `8dc58ce fix: close review findings for auto-sync, sync PFResult, and global actions`
- Branches containing HEAD:
```
"feat/ocr-structured-pipeline"
"feat/pdf-annotation-layer"
"master"
"paperforge-stabilization"
```
## git status --short
```
M manifest.json
M paperforge/__init__.py
M paperforge/plugin/manifest.json
M paperforge/worker/vector_db.py
```
## Untracked files
```
(none)
```
## Binary-safe diff
```diff
diff --git a/manifest.json b/manifest.json
index 623492d..d9287d3 100644
--- a/manifest.json
+++ b/manifest.json
@@ -1,7 +1,7 @@
{
"id": "paperforge",
"name": "PaperForge",
- "version": "1.5.6rc3",
+ "version": "1.5.6rc4",
"minAppVersion": "1.9.0",
"description": "Zotero literature pipeline for Obsidian. Sync PDFs, run OCR, and read with AI-assisted deep reading.",
"author": "Lin Zhaoxuan",
diff --git a/paperforge/__init__.py b/paperforge/__init__.py
index 4ec6722..10b1908 100644
--- a/paperforge/__init__.py
+++ b/paperforge/__init__.py
@@ -1,3 +1,3 @@
"""paperforge — PaperForge package."""
-__version__ = "1.5.6rc3"
+__version__ = "1.5.6rc4"
diff --git a/paperforge/plugin/manifest.json b/paperforge/plugin/manifest.json
index 623492d..d9287d3 100644
--- a/paperforge/plugin/manifest.json
+++ b/paperforge/plugin/manifest.json
@@ -1,7 +1,7 @@
{
"id": "paperforge",
"name": "PaperForge",
- "version": "1.5.6rc3",
+ "version": "1.5.6rc4",
"minAppVersion": "1.9.0",
"description": "Zotero literature pipeline for Obsidian. Sync PDFs, run OCR, and read with AI-assisted deep reading.",
"author": "Lin Zhaoxuan",
diff --git a/paperforge/worker/vector_db.py b/paperforge/worker/vector_db.py
index 6a746ff..1ea7f94 100644
--- a/paperforge/worker/vector_db.py
+++ b/paperforge/worker/vector_db.py
@@ -57,12 +57,11 @@ def _preflight_check(vault, settings: dict) -> dict:
def get_embed_status(vault) -> dict:
"""Check if vector index exists and has content."""
from pathlib import Path
- from paperforge.config import paperforge_paths
- paths = paperforge_paths(vault)
- vectors_dir = paths.get("vectors", paths.get("paperforge", Path()) / "vectors")
-
+ from paperforge.memory.vector_db import get_vector_db_path
+ vectors_dir = get_vector_db_path(vault)
+
status = {"exists": False, "chunk_count": 0, "collection_name": ""}
-
+
if not vectors_dir or not vectors_dir.exists():
return status
```

View file

@ -0,0 +1,484 @@
# OCR 图片管线全面分析 (2026-07-03)
分析途径codebase-memory-mcp 图谱查询 + 代码阅读。
核心文件:`paperforge/worker/ocr_figures.py` (6132 行)。
管线入口:`build_figure_inventory(structured_blocks, page_width=1200)` → 返回完整的 `FigureInventory`
55 个直接被调用者1663 行纯逻辑51 个循环结构10 个处理阶段。
---
## 一、架构总览
```
build_figure_inventory
┌───────────┼───────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Phase 0 │ │ Phase 1 │ │ Stage 1 │
│ 预计算 │ │ 候选组 │ │ 跨页结算 │
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────┐
│ Sidecar 侧车厢回退 │
│ 窄列同列题注 → 分区覆盖 │
└──────────────┬──────────────────┘
┌─────────────────────────────────┐
│ Preproof Legend Bundle │
│ 3+ 图例无资产 → 1:1 打包 │
└──────────────┬──────────────────┘
┌─────────────────────────────────┐
│ Locator Bridge │
│ "前页可见" 图例桥接 │
└──────────────┬──────────────────┘
┌─────────────────────────────────┐
│ Group-aware Sequential │
│ 未匹配组感知顺序回退 │
└──────────────┬──────────────────┘
┌─────────────────────────────────┐
│ Classic Sequential Fallback │
│ 按阅读顺序配对 │
└──────────────┬──────────────────┘
┌─────────────────────────────────┐
│ Unresolved Clusters + │
│ Composite Parent │
│ 未解析簇 + 合成父图 │
└──────────────┬──────────────────┘
┌─────────────────────────────────┐
│ Final Stage │
│ ID 冲突 / 缺失编号 / 完整性 │
└─────────────────────────────────┘
```
---
## 二、55 个直接被调用者按功能分组
### 图例检测 (Legend Detection) — 11 个函数
| 函数 | 圈复杂度 | 做的事 |
|------|----------|--------|
| `_is_formal_legend` | 13 | 正式图例识别(前缀+编号+分数阈值) |
| `_is_validation_first_legend_candidate` | 13 | "验证优先"图例候选(DWQQK2YB 风格) |
| `_has_strong_explicit_caption_text` | 7 | 强明确题注文本检测 |
| `_has_anchor_supported_legend_context` | 5 | 锚点支持的图例上下文 |
| `_is_insufficient_legend_evidence` | 5 | 证据不足的回绝 |
| `_looks_like_figure_narrative_prose` | 13 | 排除叙述性散文(非真正图例) |
| `_looks_like_inline_figure_mention` | 9 | 排除行内引用("as shown in Fig. 1") |
| `_extract_figure_number` | 8 | 抽取图号(如 "Fig. 1" → 1) |
| `_extract_figure_namespace` | 3 | 抽取图命名空间(figure/supplementary) |
| `_normalized_caption_body` | 6 | 规范化图例文本用于去重 |
| `_validate_page_local_caption_grammar` | 9 | 页级图例句法验证(hypothesis 管理) |
### 计分 (Scoring) — 2 个函数
| 函数 | 圈复杂度 | 做的事 |
|------|----------|--------|
| `score_figure_caption` (ocr_scores.py) | 78 | 图例评分(8 项体征: 编号清晰度/上下文/风格等) |
| `_caption_style_match` | 8 | 图例风格匹配检查 |
### 几何/聚类 (Geometry & Clustering) — 6 个函数
| 函数 | 圈复杂度 | 做的事 |
|------|----------|--------|
| `_media_clusters` | 6 | 媒体块空间聚类(gap 阈值) |
| `_build_candidate_figure_groups_from_assets` | 4 | 从资产构建候选组(语义聚类+聚类) |
| `_build_semantic_figure_groups_from_assets` | 4 | 语义感知聚类 |
| `_cluster_bbox` | 2 | 计算一组 bbox 的合并矩形 |
| `_estimate_page_height` | 2 | 从 blocks 估算页面高度 |
| `_partition_assets_by_caption_bands` | 11 | 按题注带切分资产 |
### 配对/计分 (Pairing & Scoring) — 5 个函数
| 函数 | 圈复杂度 | 做的事 |
|------|----------|--------|
| `_score_legend_to_group` | 13 | 图例→候选组评分(含多资产一致性加值) |
| `_score_legend_to_asset_with_orientation` | 13 | 单资产评分(含几何方向感知) |
| `_make_local_pairing_hypothesis` | 7 | 构建局部配对假设 |
| `_infer_local_pairing_mode` | 5 | 推断配对模式(same_page/cross_page) |
| `_mark_hypothesis_conflict` | 3 | 标记假设冲突 |
### 匹配/扩展 (Matching & Expansion) — 5 个函数
| 函数 | 圈复杂度 | 做的事 |
|------|----------|--------|
| `_expand_matched_assets_locally` | 8 | 本地扩展已匹配资产(吸收相邻块) |
| `_promote_sequence_matches` | 8 | 序列匹配提升(严格分层) |
| `_allow_previous_page_sequential_match` | 5 | 前页匹配门控 |
| `_same_page_narrow_caption_column` | 9 | 侧车厢窄列检测 |
| `_identify_bundle_source_legend_ids` | 7 | 打包源标识 |
### 所有权管理 (Ownership) — 3 个类方法 + 2 个函数
| 函数 | 圈复杂度 | 做的事 |
|------|----------|--------|
| `FigureOwnershipRegistry.__init__` | 5 | 注册表构造(持 used_group_ids/used_asset_page_ids) |
| `FigureOwnershipRegistry.match_group` | 4 | 组匹配标记 |
| `FigureOwnershipRegistry.mark_assets_owned` | 2 | 资产标记为已拥有 |
| `FigureOwnershipRegistry.can_consume_assets` | 4 | 资产消耗可行性检查 |
| `_has_protected_figure_ownership` | 4 | 受保护所有权检查 |
| `_fallback_eligible_asset_page_ids` | 5 | 回退合格资产页 ID |
### 账本/结算 (Ledger & Settlement) — 6 个函数
| 函数 | 圈复杂度 | 做的事 |
|------|----------|--------|
| `_build_page_ledger` | 6 | 页级账本(图例数 vs 组数 delta) |
| `_build_residual_ledger` | 8 | 剩余账本(强图例 vs 可匹配组) |
| `_recompute_final_unmatched_assets` | 5 | 重新计算最终未匹配资产 |
| `_reserve_cross_page_objects` | 8 | 跨页预留对象 |
| `_settle_cross_page_reserved_objects` | 10 | 跨页预留结算 |
| `_resolve_figure_id_collisions` | 7 | 图 ID 冲突解决 |
### 合成父图 (Composite Parent) — 4 个函数
| 函数 | 圈复杂度 | 做的事 |
|------|----------|--------|
| `_build_composite_parent_figure_groups_visual_only` | 28 | 视觉仅合成父图组构建 |
| `_build_dense_composite_parent_candidates` | 18 | 稠密合成父图候选构建 |
| `_should_suppress_panel_title_candidate` | 10 | 面板标题候选抑制 |
| `_is_safe_page_assets_group` | 16 | 页资产组安全性门控 |
### 资产工具 (Asset Utilities) — 5 个函数
| 函数 | 圈复杂度 | 做的事 |
|------|----------|--------|
| `_filter_figure_assets` | 2 | 筛选图资产 |
| `_project_asset_record` | 2 | 投影资产记录 |
| `_asset_page_id` | 2 | 资产页 ID | 标识 ID |
| `_grouped_asset_page_ids` | 4 | 分组资产页 ID |
| `_asset_vertical_side` | 4 | 资产垂直侧判定 |
### 缺失编号恢复 (Missing Number Recovery) — 3 个函数
| 函数 | 圈复杂度 | 做的事 |
|------|----------|--------|
| `_infer_missing_main_figure_numbers` | 28 | 推理缺失主图编号 |
| `_recover_missing_figure_numbers_from_assets` | 18 | 从 PDF 行恢复缺失图号 |
| `_apply_bbox_only_synthetic_vector_fallback` | 6 | 仅 bbox 合成矢量图回退 |
### 完整性 (Completeness) — 1 个函数
| 函数 | 圈复杂度 | 做的事 |
|------|----------|--------|
| `compute_figure_legend_completeness` | 27 | 图例完整性审计(匹配/持有/拒绝/未匹配/含混 五类) |
---
## 三、Phase 0 — 预计算与 Block 分类
**输入**: `structured_blocks` (全部带有 role/zone/bbox 的 block)
**输出**:
- `legends` — 正式图例(已通过 `_is_formal_legend`)
- `rejected_legends` — 不通过正式图例检测但有候选时序
- `assets` — figure_asset + media_asset(过滤 panel label、non_body_insert)
- `figure_locators` — "前页可见图例"标记
- `held_figures` — 验证优先暂缓的图例
**分类逻辑** (行 3021-3111):
1. 跳过 `_non_body_media``non_body_insert`
2. 跳过单字母面板标签 (`[A-Z]`, `(A)`, `A.`)
3. 对 `body_paragraph` 但 seed_role 是 figure_caption_candidate 的 → 原路线判定为散文, 直接丢到 `rejected_legends`
4. `figure_caption`/`figure_caption_candidate`/`validation_first_candidate`:
- 运行 PDF 题注前缀恢复(`_recover_figure_heading_prefix`)
- `_is_formal_legend` 检测 → 通过则 `legends`, 否则 `rejected_legends`
- 特殊检测: `vision_footnote`+旋转文本+图描述开头 → 挽救到 `legends`
- 特殊检测: `_is_previous_page_legend_locator``figure_locators`
5. 资产分配:
- `figure_asset``assets`
- `media_asset` — 仅 raw_label 为 image/chart/figure_title/figure 或空时加入
### 资产家族提示 (`asset_family_hint`)
- `figure_like` (raw_label=image/chart/figure_title/figure) — 置信度 0.70
- `table_like` (raw_label=table) — 置信度 0.70
- `ambiguous` (其余) — 置信度 0.35
### 图例去重
对编号图例按 `(namespace, number)` 分组去重, 优先保留:
1. 非 bundle-source 页上的
2. 非 caption-list 页上的
3. 分数更高的
4. 相同分数的优先保留首次出现的
不同编号但相同 `number` 不同文本的 → `_same_number_distinct_keys` (后续保持独立)
---
## 四、Phase 1 — 候选组构建与页内匹配
**输入**: `legends`, `assets`, `ordered_legends`, `candidate_groups`
**处理流程**:
1. **Dense composite parent** (行 3265-3286): 在图例循环之前构建稠密父组合
2. **Panel title suppression** (行 3289-3327):
- 有编号图例的页面上, 短无编号面板标题被抑制
- `_should_suppress_panel_title_candidate` 检查文本长度 < 70 + 不在显示区
3. **Candidate group gating** (行 3328-3348):
- `page_assets` 组在竞争图页面被抑制
4. **Page ledger** (行 3349-3350):
- `_build_page_ledger` 计算每页图例数 - 候选组数的 delta
5. **Scoring** (行 ~3354-3367):
- `_score_legend_to_group` 对每对 legend↔group 评分
- 使用多维度评分: 空间接近度、方向、家族支持、区域支持
- 三种组类型不同评分策略:
- `distance_cluster`: 基础分 + 多资产一致性加值 0.15
- `page_assets`: 需通过 `_is_safe_page_assets_group` 门控
- `single_asset`: 直接使用 `_score_legend_to_asset_with_orientation`
---
## 五、Stage 1 — 跨页预留与结算
**输入**: `matched_figures`, `ordered_legends`, `candidate_groups`, `_page_blocks_by_page`
**处理流程**:
1. **Cross-page reservation** (行 ~3367-3947):
- `_reserve_cross_page_objects` 在页面间预留双向引用
- 预留基于图例-组配对的证据强度
- reserved legends look backward (寻找前页资产)
- reserved groups look forward (寻找后页图例)
2. **Primary cross-page settlement** (行 3948-3964):
- `_settle_cross_page_reserved_objects` 执行预留对象的实际结算
3. **Failed groups handling** (行 3973-4002):
- 多资产组 → `unresolved_clusters`
- 单资产 → `unmatched_assets`
---
## 六、Sidecar 侧车厢回退
**位置**: 行 ~4004-4222
**适用**: 窄列同列正式题注的页面 (如横排两个独立图, 各有自己的窄列题注)
**机制**:
- 通过 `_same_page_narrow_caption_column` 检测是否为侧车厢页
- 对侧车厢页, 使用 `_partition_assets_by_caption_bands` 按题注带切分资产
- 题注带的资产按垂直区域分配给最近题注
- 非侧车厢页保持常规 gap/overlap 匹配逻辑
**与正常匹配的差异**: 侧车厢的可见图/题注配对是基于列的, 不是基于 gap/overlap 的。
常规空间匹配器无法可靠地将资产分配给窄列侧车厢题注——所以需要这里特殊处理。
---
## 七、预印版图例打包回退 (Preproof Legend Bundling)
**位置**: 行 ~4223-4346
**适用**: 单页有 >=3 个图例且没有任何同页资产 → 按页码顺序 1:1 打包到后续资产页
**机制**:
- 从 bundle-source 图例页收集 >=3 个图例
- 检查通过页面 (无 body/table 中断)
- 按顺序匹配图例 → 后续资产页
- **保护距离聚类的多资产组不被回退消耗**
- 结果: `matched_figures` 中有 `legend_bundle_match` 标记, 置信度 0.3
---
## 八、前页图例定位桥 (Previous-page Locator Bridge)
**位置**: 行 ~4347-4542
**适用**: "Fig. 10 (See legend on previous page.)" 模式
**机制**:
- 定位图例的页面 → 获取前页的同编号完整图例
- 定位页 → 获取定位图例上方的视觉组
- 桥接三个组件: 前页完整图例 + 定位页视觉组 + 定位图例本身
- 结果: 匹配置信度 0.5, flags=previous_page_locator_match
- 替换 ambiguity 条目以避免重复匹配
**保护条件**:
- 仅当定位图例在前页有完整图例(>=60 字符, 非自身定位标记)
- 仅当定位页上方有视觉组
- 仅当页面间无 body/table 中断
---
## 九、组感知顺序回退 (Group-aware Sequential Fallback)
**位置**: 行 ~4543-4672
**适用**: 没有被同页图例认领的未匹配 `distance_cluster`
**机制**:
- 收集未被使用的组 (包括 `distance_cluster``single_asset`)
- 按页面 + 垂直位置排序
- 对每个未匹配的图例, 在之后的页面上查找最合适的组
- **在旧式单资产回退之前执行**, 确保组优先于裸资产
---
## 十、经典顺序回退 (Sequential Fallback)
**位置**: 行 ~4674-4783
**适用**: 未匹配图例 → 任何剩余资产
**机制**:
- 图例和图形经常出现在不同页——按阅读顺序匹配
- 从候选组过滤后, 对未聚类的资产与未匹配图例配对
- 限制: 不能消耗属于任何候选组的资产
- 支持前页匹配 (`_allow_previous_page_sequential_match`)
**缺陷**: 这是最宽泛的回退——如果前序阶段都没有匹配成功, 它会尝试任何组合。
这是已知的 tradeoff: "Captions and figures often appear on different pages — humans
match them by sequential reading order, not spatial proximity."
---
## 十一、未解析簇 + 合成父图 (Unresolved Clusters + Composite Parent)
**位置**: 行 ~4785-4873
**处理流程**:
1. **未解析簇**: 被回绝图例页面上的未匹配资产 → `_media_clusters` 聚簇
2. **合成父图候选**: `_build_composite_parent_figure_groups_visual_only` + `_build_dense_composite_parent_candidates`
3. **稠密页面整合**: 有合成父图候选的页面上, 剩余未匹配资产被组合到未解析簇
4. **跨页结算后清理**: 已匹配 figure 的 block 从未解析簇中移除
---
## 十二、最终阶段 (Final Stage)
**位置**: 行 ~4874-5114
**处理流程**:
1. 页内图例句法验证 (`_validate_page_local_caption_grammar`)
2. 从未解析簇中移除已匹配 figure 的 block
3. `_resolve_figure_id_collisions` — 处理 ID 冲突
4. `_infer_missing_main_figure_numbers` — 推理缺失的编号
5. `_apply_bbox_only_synthetic_vector_fallback` — bbox-only 合成矢量图回退
6. `_recover_missing_figure_numbers_from_assets` — 从 PDF 行推断资产内部缺失图号
7. `compute_figure_legend_completeness` — 五类图例审计
---
## 十三、关键数据结构
### FigureOwnershipRegistry
核心状态管理类, 追踪:
- `used_group_ids: set[str]` — 已被匹配的组 ID
- `used_asset_page_ids: set[tuple[int, str]]` — 已被占有的资产 (page, block_id)
- 方法: `match_group`, `mark_assets_owned`, `can_consume_assets`
**设计**: 可变集合, 在 1663 行函数中就地更新。没有事务/回滚机制。
这意味着某个阶段的错误匹配会污染所有后续阶段的可用性计算。
### matched_figures
管线的主产物。每个条目:
- `figure_id`, `figure_namespace`, `figure_number`
- `legend_block_id`, `text`
- `matched_assets: list[dict]`, `asset_block_ids`
- `page`, `legend_page`, `asset_pages`
- `match_score: {score, decision, evidence}`
- `confidence: float`
- `flags: list[str]`
- `settlement_type: str` (same_page / cross_page / legend_bundle / previous_page_legend_locator / sequential)
- `caption_score: dict`
- `bridge_block_ids: list[str]` (只在 locator bridge 时非空)
---
## 十四、处理流程全景
```
structured_blocks
[Phase 0: Block Classification]
│ legends, rejected_legends, assets, figure_locators, held_figures
│ asset_family_hint annotation
│ legend dedup by number+namespace
[Phase 1: Candidate Groups + Page-internal Matching]
│ candidate_groups (distance_cluster / page_assets / single_asset)
│ _score_legend_to_group → matched_figures (same_page)
[Stage 1: Cross-page Reservation + Settlement]
│ _reserve_cross_page_objects → reserved_legend_ids / reserved_group_ids
│ _settle_cross_page_reserved_objects → matched_figures (cross_page)
[Sidecar Fallback]
│ narrow same-column captions → caption-band partition
│ override normal matching when detected
[Preproof Legend Bundling]
│ 3+ legends, 0 same-page assets → 1:1 bundle to subsequent asset pages
[Previous-page Locator Bridge]
│ "See legend on previous page" → connect full legend + visual group
[Group-aware Sequential Fallback]
│ unmatched distance_clusters → unmatched legends (in page order)
[Classic Sequential Fallback]
│ unmatched legends → any remaining assets (reading order)
[Unresolved Clusters + Composite Parent]
│ spatial clusters of unmatched assets on rejected-legend pages
│ composite parent + dense parent constructions
[Final Stage]
│ ID collision resolution
│ Missing number inference
│ Completeness audit
FigureInventory
```
---
## 十五、风险总结
### 高风险
1. **`build_figure_inventory` 巨型函数 (1663 行)**
- 10 个顺序阶段, 每个阶段有自己的条件逻辑
- 没有事务/回滚: 所有权注册表是可变集合
- 一个阶段的错误匹配污染所有后续阶段
- 5 个回退阶段之间有复杂的优先级和互斥条件
2. **5 个串行回退阶段的组合覆盖**
- 每个回退阶段对前一阶段的输出进行操作
- 没有测试覆盖所有阶段组合
- 某些论文可能触发多个回退阶段, 它们的交互未明确测试
3. **Performance: allocate_in_loop=53**
- 在 51 个循环中有 53 次分配
- 对大型论文(50+ 图), 每次重建都会触发大量内存分配
- `render_fulltext_markdown` 也有 45 次循环内分配
### 中风险
4. **组反转/图例去重依赖顺序**
- 去重仅基于编号+命名空间, 不包含页面位置上下文
- 若两块编号相同的图例在不同页, 系统可能只保留一个
5. **sidecar 检测是硬编码的**
- `_same_page_narrow_caption_column` 依赖固定的列宽阈值
- 对某些自由格式期刊可能误判
6. **Bundle 回退的门控条件复杂**
- 跨越多个页面的 body/table 中断检测
- `_NON_PURE_ROLES` 集硬编码了 8 个角色
### 低风险 (设计选择)
7. 顺序回退是宽泛的但已知 tradeoff (代码注释已承认)
8. 合成父图构造使用视觉聚类而非语义信号
9. 置信度阈值 (0.3/0.5/0.72/0.85) 是硬编码的, 没有从数据中学习
### 已覆盖 (不在此模块)
- 版面分析的前言区域泄漏已在三个计划修复中涵盖
- 列感知题注前缀恢复已实现

View file

@ -0,0 +1,45 @@
from __future__ import annotations
import json
from pathlib import Path
from paperforge.worker.ocr_figures import (
build_figure_inventory_legacy,
build_figure_inventory_vnext,
)
def _figure_asset_ids(fig: dict[str, object]) -> list[str]:
ids = {str(x) for x in fig.get("asset_block_ids", [])}
ids.update(str(asset.get("block_id", "")) for asset in fig.get("matched_assets", []))
return sorted(x for x in ids if x)
def compare_inventories(legacy: dict[str, object], vnext: dict[str, object]) -> dict[str, object]:
return {
"legacy_matched_count": len(legacy.get("matched_figures", [])),
"vnext_matched_count": len(vnext.get("matched_figures", [])),
"legacy_unresolved_count": len(legacy.get("unresolved_clusters", [])),
"vnext_unresolved_count": len(vnext.get("unresolved_clusters", [])),
"legacy_unmatched_legend_count": len(legacy.get("unmatched_legends", [])),
"vnext_unmatched_legend_count": len(vnext.get("unmatched_legends", [])),
"legacy_consumed_block_ids": sorted(
{bid for fig in legacy.get("matched_figures", []) for bid in _figure_asset_ids(fig)}
),
"vnext_consumed_block_ids": sorted(
{bid for fig in vnext.get("matched_figures", []) for bid in _figure_asset_ids(fig)}
),
}
def main(blocks_path: str) -> int:
blocks = [json.loads(line) for line in Path(blocks_path).read_text(encoding="utf-8").splitlines() if line.strip()]
legacy = build_figure_inventory_legacy(blocks)
vnext = build_figure_inventory_vnext(blocks)
print(json.dumps(compare_inventories(legacy, vnext), ensure_ascii=False, indent=2))
return 0
if __name__ == "__main__":
import sys
raise SystemExit(main(sys.argv[1]))

View file

@ -0,0 +1,93 @@
page,block_id,raw_label,content_preview,bbox,role,role_confidence,evidence,seed_role,seed_confidence,zone,style_family,marker_type,render_default,index_default
1,0,number,264,"[85, 29, 118, 48]",noise,0.9,"[""page number label""]",noise,0.9,frontmatter_main_zone,support_like,short_fragment,False,False
1,1,header,Acta Orthop Scand 1996; 67 (3): 264268,"[584, 29, 886, 50]",noise,0.9,"[""header label""]",noise,0.9,frontmatter_main_zone,support_like,none,False,False
1,2,doc_title,Atrophy of the supraspinatus belly,"[83, 114, 664, 155]",paper_title,0.8,"[""page-1 zone title_zone: Atrophy of the supraspinatus belly""]",paper_title,0.8,frontmatter_main_zone,support_like,none,True,True
1,3,doc_title,Assessment by MRI in 55 patients with rotator cuff pathology,"[83, 169, 769, 199]",paper_title,0.8,"[""page-1 zone title_zone: Assessment by MRI in 55 patients with rotator cuff pathology""]",paper_title,0.8,frontmatter_main_zone,support_like,none,True,True
1,4,text,"Hervé Thomazeau, Yann Rolland, Christophe Lucas, Jean-Marie Duval and Frantz Langlais","[84, 243, 781, 293]",authors,0.8,"[""page-1 zone author_zone: Herv\u00e9 Thomazeau, Yann Rolland, Christophe Lucas, Jean-Marie ""]",authors,0.8,frontmatter_main_zone,support_like,none,True,True
1,5,abstract,A study of 5 fresh cadaveric shoulders demonstrated that an oblique-sagittal plane which crosses the scapula through the medial border of the coracoid process offers a view of the supraspinatus fossa ,"[85, 339, 478, 439]",abstract_body,0.85,"[""abstract label from Paddle OCR""]",abstract_body,0.85,frontmatter_main_zone,support_like,none,True,True
1,6,abstract,This view could easily be reproduced by MRI and we called it the Y-shaped view. It allowed a reliable measurement of supraspinatus muscle atrophy by the calculation of the occupation ratio (R) which i,"[85, 439, 480, 563]",abstract_body,0.85,"[""abstract label from Paddle OCR""]",abstract_body,0.85,frontmatter_main_zone,support_like,none,True,True
1,7,abstract,"This ratio was calculated in a prospective study based on 55 shoulders divided into 3 groups with different rotator cuff status: group I, 15 controls; group II, 10 degenerative cuffs, without tears; g","[495, 338, 890, 562]",abstract_body,0.85,"[""abstract label from Paddle OCR""]",abstract_body,0.85,frontmatter_main_zone,support_like,none,True,True
1,8,text,"Rennes University Hospital, France. Correspondence: Dr. Hervé Thomazeau, University Hospital, 16 Bd de Bulgarie, F-35056 Rennes, France. Tel +33 99-26-71-67. Fax -32-80-06
Submitted 95-11-04. Accepted","[87, 578, 842, 631]",frontmatter_noise,0.8,"[""page-1 zone journal_furniture_zone: Rennes University Hospital, France. Correspondence: Dr. Herv""]",frontmatter_noise,0.8,frontmatter_main_zone,support_like,none,False,False
1,9,text,"Occasionally, the anatomical and functional results of reconstruction of full-thickness rotator cuff tears, where two tendons or more have been primarily sutured, are disappointing (Harryman et al. 19","[86, 700, 480, 855]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
1,10,text,"Such a limit cannot be appreciated solely on the basis of duration of symptoms, clinical presentation, or radiological and sonographic evaluation of the size of the tear. Probably an evaluation of the","[87, 856, 482, 1058]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
1,11,paragraph_title,Material and methods,"[90, 1109, 299, 1132]",section_heading,0.5,"[""unnumbered paragraph_title on page 1 outside title zone: Material and methods""]",section_heading,0.5,body_zone,heading_like,none,True,True
1,12,text,The aims of our anatomical study were to define the oblique-sagittal plane which offers the most extended osseous limits for the supraspinatus fossa and to define the lateral limits of the supraspinat,"[90, 1143, 482, 1254]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
1,13,text,The purpose of the radioclinical study was then to find a simple and reproducible method of measuring the muscular atrophy based on the oblique-sagittal plane previously defined by the anatomical stud,"[90, 1255, 484, 1321]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
1,14,text,,"[496, 700, 889, 745]",ocr_text_missing,0.8,"[""ocr detected text region (raw_label=text) but no text extracted; no pdf backfill available""]",ocr_text_missing,0.8,body_zone,body_like,empty,True,True
1,15,paragraph_title,Anatomical study,"[498, 762, 652, 784]",section_heading,0.5,"[""unnumbered paragraph_title on page 1 outside title zone: Anatomical study""]",section_heading,0.5,body_zone,body_like,short_fragment,True,True
1,16,text,"At first, 5 non-embalmed cadaveric shoulders were studied to assess the limits of the supraspinatus fossa. As this evaluation was independent of the rotator cuff status, dissection of the supraspinatu","[497, 788, 893, 1098]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
1,17,text,Another 5 non-embalmed shoulders were then dissected to define the lateral attachment of the supraspinatus muscle on the scapula. The clavicle and the acromion were removed en bloc with the trapezoid ,"[499, 1098, 894, 1320]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
1,18,footer,Copyright © Scandinavian University Press 1996. ISSN 0001-6470. Printed in Sweden all rights reserved.,"[92, 1361, 794, 1383]",noise,0.9,"[""footer label""]",noise,0.9,body_zone,body_like,none,False,False
2,0,header,Acta Orthop Scand 1996; 67 (3): 264268,"[83, 28, 388, 49]",noise,0.9,"[""header label""]",noise,0.9,body_zone,body_like,none,False,False
2,1,number,265,"[853, 30, 884, 48]",noise,0.9,"[""page number label""]",noise,0.9,body_zone,body_like,short_fragment,False,False
2,2,image,,"[89, 117, 882, 344]",media_asset,0.85,"[""media label: image""]",media_asset,0.85,body_zone,body_like,empty,True,True
2,3,figure_title,Figure 3. Position of the lateral attachment of the supraspinatus muscle in its fossa from an above view.,"[557, 275, 884, 309]",figure_caption,0.92,"[""figure_title label: Figure 3. Position of the lateral attachment of the supraspi""]",figure_caption,0.92,display_zone,legend_like,figure_number,True,True
2,4,figure_title,Figure 2. Calculation of the ratio O/M between osseous (O) and muscular (M) limits of the supraspinatus fossa on an oblique-sagittal anatomical slice.,"[410, 329, 886, 364]",figure_caption,0.92,"[""figure_title label: Figure 2. Calculation of the ratio O/M between osseous (O) a""]",figure_caption,0.92,display_zone,legend_like,figure_number,True,True
2,5,figure_title,Figure 1. 4 levels of oblique-sagittal cross-sections of frozen shoulders (scapula viewed from the back).,"[252, 386, 580, 421]",figure_caption,0.92,"[""figure_title label: Figure 1. 4 levels of oblique-sagittal cross-sections of fro""]",figure_caption,0.92,display_zone,legend_like,figure_number,True,True
2,6,text,"muscular fibers and the aponeurosis attached to the bone. To define the lateral limit of this attachment, the distances were recorded between this limit and 3 defined landmarks: A, bottom of the scapu","[82, 456, 477, 700]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
2,7,paragraph_title,Radioclinical study,"[83, 717, 250, 739]",subsection_heading,0.6,"[""unnumbered paragraph_title, inferred level subsection_heading: Radioclinical study""]",subsection_heading,0.6,body_zone,body_like,short_fragment,True,True
2,8,text,"A prospective clinical and MRI study was then carried out on 55 persons divided into 3 groups. Group I consisted of 15 asymptomatic volunteers: 5 men and 10 women, mean age 45 (1474) years. Group II ","[82, 745, 477, 1076]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
2,9,text,"At the time of surgery, the size of the tear was anatomically described in both the sagittal and coronal planes, independently of the size of the patient. In the sagittal plane, the lesions were class","[83, 1077, 479, 1320]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
2,10,text,,"[492, 456, 887, 634]",ocr_text_missing,0.8,"[""ocr detected text region (raw_label=text) but no text extracted; no pdf backfill available""]",ocr_text_missing,0.8,body_zone,body_like,empty,True,True
2,11,text,"All MRI images were obtained with a 1Tesla unit (Magneton Impact, Siemens, Germany) The patient lay supine in a pain-free position, with the arm in neutral rotation along the chest and a curved loop s","[492, 633, 887, 922]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
2,12,text,"The quantitative analysis was then performed on the spin-echo T1-weighted oblique-sagittal images (TR: 480ms, TE: 12ms, FOV: 250×250, matrix: 380×512).","[493, 923, 888, 1009]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
2,13,text,"To evaluate the atrophy of the supraspinatus muscle, we calculated the occupation ratio (R) of the supraspinatus fossa by the muscle belly. This analysis was based on the ratio between the surface of ","[493, 1012, 889, 1275]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
2,14,text,The correlation and relative error between the results achieved by the 3 readers were calculated and,"[494, 1276, 889, 1319]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,0,number,266,"[85, 23, 118, 42]",noise,0.9,"[""page number label""]",noise,0.9,body_zone,body_like,short_fragment,False,False
3,1,header,Acta Orthop Scand 1996; 67 (3): 264268,"[585, 23, 888, 43]",noise,0.9,"[""header label""]",noise,0.9,body_zone,body_like,none,False,False
3,2,image,,"[176, 109, 370, 337]",media_asset,0.85,"[""media label: image""]",media_asset,0.85,body_zone,unknown_like,empty,True,True
3,3,figure_title,"Figure 4. Calculation of the occupation ratio R on the oblique-sagittal view. S1 Surface of the supraspinatus muscle, S2 surface of the entire supraspinatus fossa.","[85, 359, 479, 412]",figure_caption,0.92,"[""figure_title label: Figure 4. Calculation of the occupation ratio R on the obliq""]",figure_caption,0.92,display_zone,legend_like,figure_number,True,True
3,4,text,"the final result was a mean of these values expressed from 0 (empty fossa) to 1 (full fossa). Then the mean values of R in the different clinical populations were analyzed, using a statistical analysi","[84, 452, 480, 586]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,5,paragraph_title,Results Radioanatomical study,"[86, 640, 287, 691]",subsection_heading,0.6,"[""unnumbered paragraph_title, inferred level subsection_heading: Results Radioanatomical study""]",subsection_heading,0.6,body_zone,heading_like,none,True,True
3,6,footer,,"[86, 670, 287, 691]",noise,0.9,"[""footer label""]",noise,0.9,body_zone,body_like,empty,False,False
3,7,text,"In the first 5 shoulders, slice 1 demonstrated the highest 0/M ratio, between 2.2 and 2.75 (Table 1). At this level, the scapula is cut through the medial border of the coracoid process and the latera","[85, 695, 482, 983]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,8,text,"The lateral bony attachment of the supraspinatus muscle always remained medial to points A and B (mean distances were, respectively, -5.4 mm and -1.8 mm), but spread laterally to point C in 4 of the 5","[87, 985, 482, 1161]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,9,text,This preliminary anatomical study suggested that slice 1 provided the largest bony limits for the supraspinatus fossa and probably represented the most reliable level for the calculation of the occupa,"[87, 1162, 484, 1317]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,10,figure_title,Table 1. Ratio between osseous (O) and muscular (M) limits of the supraspinatus fossa,"[495, 113, 888, 150]",table_caption,0.9,"[""table prefix matched: Table 1. Ratio between osseous (O) and muscular (M) limits o""]",table_caption,0.9,display_zone,table_caption_like,table_number,True,True
3,11,table,<table><tr><td>Shoulder</td><td>Slice 1</td><td>Slice 2</td><td>Slice 3</td><td>Slice 4</td></tr><tr><td>1</td><td>2.5</td><td>0.6</td><td>1</td><td>0.75</td></tr><tr><td>2</td><td>2.4</td><td>0.6</td,"[502, 170, 882, 322]",table_html,0.95,"[""inline table HTML""]",table_html,0.95,body_zone,body_like,none,True,False
3,12,table,<table><tr><td>Shoulder</td><td>Point A</td><td>Point B</td><td>Point C</td></tr><tr><td>1</td><td>$ -5 $</td><td>$ -3 $</td><td>$ -2 $</td></tr><tr><td>2</td><td>$ -5 $</td><td>$ -2 $</td><td>$ +5 $<,"[504, 418, 876, 571]",table_html,0.95,"[""inline table HTML""]",table_html,0.95,body_zone,body_like,none,True,False
3,13,figure_title,Table 2. Position of the lateral attachment of the supraspinatus (in mm),"[496, 362, 888, 400]",table_caption,0.9,"[""table prefix matched: Table 2. Position of the lateral attachment of the supraspin""]",table_caption,0.9,display_zone,table_caption_like,table_number,True,True
3,14,text,ue the radioclinical study by measuring the occupa-tion ratio on the MRI view corresponding to this anat-tomical slice 1: we called it the Y-shaped view.,"[496, 629, 889, 696]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,15,paragraph_title,Radioclinical study,"[498, 712, 665, 734]",subsection_heading,0.6,"[""unnumbered paragraph_title, inferred level subsection_heading: Radioclinical study""]",subsection_heading,0.6,body_zone,body_like,short_fragment,True,True
3,16,text,"This Y-shaped view was found to be easily reproducible, since it was available for all the 55 clinical cases. When 3 readers (r1, r2, r3) calculated the occupation ratio, the relative error was 12 (3","[496, 738, 891, 872]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,17,text,"The highest value obtained for the occupation ratio, 0.87, was found in group I. Aggravation of the tendinopathy resulted in a decreased ratio (Table 3). The differences in the mean values of R betwee","[497, 873, 893, 1116]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,18,text,"In group III, there was no difference in the occupation ratio when the tear involved only the supraspinatus or extended to the rotator interval. The difference dramatically increased when the infraspi","[499, 1117, 893, 1247]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,19,text,"The drop in this ratio also correlated with the coronal retraction of the tendon stump: 0.59 (SD 0.11) for grade 1, 0.41 (0.12) for grade 2 and 0.30 (0.11) for grade 3.","[500, 1248, 895, 1316]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
4,0,header,Acta Orthop Scand 1996; 67 (3): 264268,"[80, 24, 384, 45]",noise,0.9,"[""header label""]",noise,0.9,body_zone,body_like,none,False,False
4,1,number,267,"[851, 23, 882, 42]",noise,0.9,"[""page number label""]",noise,0.9,body_zone,body_like,short_fragment,False,False
4,2,figure_title,Table 3. Occupation ratio of the supraspinatus fossa related to grade of supraspinatus tendinopathy,"[79, 115, 472, 151]",table_caption,0.9,"[""table prefix matched: Table 3. Occupation ratio of the supraspinatus fossa related""]",table_caption,0.9,display_zone,table_caption_like,table_number,True,True
4,3,table,"<table><tr><td>Grade</td><td>n</td><td>Occupation ratio, mean (SD)</td></tr><tr><td>I</td><td>15</td><td>0.7 (0.09)</td></tr><tr><td>II</td><td>10</td><td>0.62 (0.11)</td></tr><tr><td>III</td><td>30</","[87, 171, 467, 270]",table_html,0.95,"[""inline table HTML""]",table_html,0.95,body_zone,body_like,none,True,False
4,4,figure_title,Table 4. Occupation ratio of the supraspinatus fossa related to the extent of the tear,"[82, 316, 476, 352]",table_caption,0.9,"[""table prefix matched: Table 4. Occupation ratio of the supraspinatus fossa related""]",table_caption,0.9,display_zone,table_caption_like,table_number,True,True
4,5,table,"<table><tr><td>Tears</td><td>n</td><td>Occupation ratio, mean (SD)</td></tr><tr><td>Supraspinatus alone</td><td>11</td><td>0.55 (0.14)</td></tr><tr><td>Supraspinatus and rotator interval</td><td>4</td","[90, 373, 469, 516]",table_html,0.95,"[""inline table HTML""]",table_html,0.95,body_zone,body_like,none,True,False
4,6,vision_footnote,"Decrease of the occupation ratio is significant between a tear of the supraspinatus alone, and a tear which extends to the infraspinatus or to more than 2 tendons (p 0.001)","[89, 526, 470, 578]",footnote,0.7,"[""vision_footnote label: Decrease of the occupation ratio is significant between a te""]",footnote,0.7,body_zone,body_like,none,True,True
4,7,text,"grade 3 (p 0.0004). Lastly, when a large capsulotomy was required for the repair, the ratio was 0.33 (0.09) and up to 0.50 (0.16) when such capsulotomy was not necessary (p 0.004).","[84, 630, 478, 720]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
4,8,paragraph_title,Discussion,"[87, 774, 195, 796]",section_heading,0.9,"[""explicit scholarly heading: Discussion""]",section_heading,0.9,body_zone,heading_like,canonical_section_name,True,True
4,9,text,Tendinopathies of the rotator cuff muscles should be considered as part of a disease of the whole muscle. Previous experimental and radioclinical studies have demonstrated that the muscles atrophy and,"[85, 808, 481, 1074]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
4,10,text,"The calculation of the occupation ratio of the supraspinatus is reliable. It concerns an easily delineated surface, which is compared to an anatomical landmark represented by the limits of the suprasp","[87, 1075, 483, 1319]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
4,11,image,,"[563, 119, 846, 509]",media_asset,0.85,"[""media label: image""]",media_asset,0.85,body_zone,unknown_like,empty,True,True
4,12,figure_title,"Figure 5. MRI oblique-sagittal view demonstrating the lateral attachment of the supraspinatus muscle on the spine of the scapula, above the the supraglenoid notch (black arrows).","[495, 527, 888, 579]",figure_caption,0.92,"[""figure_title label: Figure 5. MRI oblique-sagittal view demonstrating the latera""]",figure_caption,0.92,display_zone,legend_like,figure_number,True,True
4,13,text,"We found that this bony landmark made the analysis easily reproducible from one patient to another and in the course of time. Thus, pre- and postoperative evaluations become possible, especially as th","[496, 632, 892, 876]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
4,14,text,Our anatomical study demonstrated that the supraspinatus belly insertion spreads more laterally than has been classically described (Gardner et al. 1980). This attachment on the anterior aspect of the,"[497, 876, 893, 1075]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
4,15,text,"The radioclinical part of our study demonstrated that the ratio was never as high as 1.0 because of the fatty environment, particularly on the anterosuprior aspect of the supraspinatus muscle. This fa","[499, 1076, 896, 1320]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
5,0,number,268,"[86, 31, 118, 49]",noise,0.9,"[""page number label""]",noise,0.9,body_zone,body_like,short_fragment,False,False
5,1,header,Acta Orthop Scand 1996; 67 (3): 264268,"[585, 31, 888, 50]",noise,0.9,"[""header label""]",noise,0.9,body_zone,body_like,none,False,False
5,2,image,,"[80, 108, 355, 441]",media_asset,0.85,"[""media label: image""]",media_asset,0.85,body_zone,unknown_like,empty,True,True
5,3,image,,"[370, 110, 639, 441]",media_asset,0.85,"[""media label: image""]",media_asset,0.85,body_zone,unknown_like,empty,True,True
5,4,image,,"[653, 109, 878, 440]",media_asset,0.85,"[""media label: image""]",media_asset,0.85,body_zone,unknown_like,empty,True,True
5,5,figure_title,"Figure 6. 3 grades of supraspinatus atrophy. From left to right: stage I, stage II, and stage III (the latter with associated infraspinatus atrophy).","[87, 460, 889, 495]",figure_caption,0.92,"[""figure_title label: Figure 6. 3 grades of supraspinatus atrophy. From left to ri""]",figure_caption,0.92,display_zone,legend_like,figure_number,True,True
5,6,text,phy and fatty infiltration are two different expressions of the same muscle disease and this previous disagreement could be explained by the better sensitivity of the quantitative analysis we propose.,"[86, 547, 479, 635]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
5,7,text,"In any case, the supraspinatus tendon tear represents the first turning-point of this muscle disease. The occupation ratio is still above 0.5 when the tear involves the supraspinatus alone or extends ","[87, 637, 481, 924]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,tail_nonref_hold_zone,body_like,none,True,True
5,8,text,"The need for a large capsulotomy is associated with such a low occupation ratio, itself correlated with the extent of the tear, and one may question the elasticity and viability of the supraspinatus t","[88, 926, 482, 1121]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,tail_nonref_hold_zone,body_like,none,True,True
5,9,text,"We propose a classification of supraspinatus belly atrophy based on the occupation ratio of the supraspinatus fossa (Figure 6). In the case of a ratio between 1.00 and 0.60 (stage I), the muscle can b","[89, 1124, 484, 1300]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,tail_nonref_hold_zone,body_like,none,True,True
5,10,text,,"[497, 546, 891, 681]",ocr_text_missing,0.8,"[""ocr detected text region (raw_label=text) but no text extracted; no pdf backfill available""]",ocr_text_missing,0.8,body_zone,body_like,empty,True,True
5,11,paragraph_title,References,"[500, 713, 611, 734]",reference_heading,0.9,"[""references heading: References""]",reference_heading,0.9,reference_zone,heading_like,short_fragment,True,True
5,12,reference_content,"Bigliani L U, Dalsey R M, McCann P D, April E W. An anatomical study of the suprascapular nerve. Arthroscopy 1990; 6 (4): 301-5.","[501, 744, 890, 798]",reference_item,0.85,"[""reference content label: Bigliani L U, Dalsey R M, McCann P D, April E W. An anatomic""]",reference_item,0.85,reference_zone,reference_like,citation_line,True,True
5,13,reference_content,Björkenheim J M. Structure and function of the rabbit's supraspinatus muscle after resection of its tendon. Acta Orthop Scand 1989; 60 (4): 461-3.,"[501, 803, 889, 857]",reference_item,0.85,"[""reference content label: Bj\u00f6rkenheim J M. Structure and function of the rabbit's supr""]",reference_item,0.85,reference_zone,reference_like,none,True,True
5,14,reference_content,"Gardner E, Gray DJ, O'Rahilly R. Anatomy. W.B. Saunders, Philadelphia-London-Toronto 1980.","[501, 861, 890, 897]",reference_item,0.85,"[""reference content label: Gardner E, Gray DJ, O'Rahilly R. Anatomy. W.B. Saunders, Phi""]",reference_item,0.85,reference_zone,reference_like,citation_line,True,True
5,15,reference_content,"Goutallier D, Bernageau J, Patte D. Assessment of the trophicity of the muscles of the ruptured rotator cuff by CT scan. In: Surgery of the shoulder. (Eds. Post M, Morrey B F, Hawkins R J). Mosby-Year","[503, 902, 891, 990]",reference_item,0.85,"[""reference content label: Goutallier D, Bernageau J, Patte D. Assessment of the trophi""]",reference_item,0.85,reference_zone,reference_like,citation_line,True,True
5,16,reference_content,"Goutallier D, Postel J M, Bernageau J, Lavau L, Voisin M C. Fatty degeneration in cuff rupture. Pre- and postoperative evaluation by CT scan. Clin Orthop 1994; 304: 78-83.","[502, 996, 891, 1050]",reference_item,0.85,"[""reference content label: Goutallier D, Postel J M, Bernageau J, Lavau L, Voisin M C. ""]",reference_item,0.85,reference_zone,reference_like,citation_line,True,True
5,17,reference_content,"Harryman D T, Mack L A, Wang K Y, Jackins S E, Richardson M L, Matsen F A. Repairs of the rotator cuff: correlation of functional results with integrity of the cuff. J. Bone Joint Surg (Am) 1991; 73 (","[503, 1054, 891, 1126]",reference_item,0.85,"[""reference content label: Harryman D T, Mack L A, Wang K Y, Jackins S E, Richardson M ""]",reference_item,0.85,reference_zone,reference_like,citation_line,True,True
5,18,reference_content,"Maughan R J, Watson J S, Weir J. Strength and cross-sectional area of human skeletal muscle. J Physiol 1983; 338: 37-49.","[502, 1131, 891, 1184]",reference_item,0.85,"[""reference content label: Maughan R J, Watson J S, Weir J. Strength and cross-sectiona""]",reference_item,0.85,reference_zone,reference_like,citation_line,True,True
5,19,reference_content,"Nakagaki K, Osaki J, Tomita Y, Tamai S. Alterations in the supraspinatus belly with rotator cuff tearing: Evaluation with magnetic resonance imaging. J Shoulder Elbow Surg 1994; 3 (2): 88-93.","[504, 1189, 892, 1260]",reference_item,0.85,"[""reference content label: Nakagaki K, Osaki J, Tomita Y, Tamai S. Alterations in the s""]",reference_item,0.85,reference_zone,reference_like,citation_line,True,True
5,20,reference_content,Patte D. Classification of rotator cuff lesions. Clin Orthop 1990; 254: 81-6.,"[504, 1265, 892, 1299]",reference_item,0.85,"[""reference content label: Patte D. Classification of rotator cuff lesions. Clin Orthop""]",reference_item,0.85,reference_zone,reference_like,citation_line,True,True
1 page block_id raw_label content_preview bbox role role_confidence evidence seed_role seed_confidence zone style_family marker_type render_default index_default
2 1 0 number 264 [85, 29, 118, 48] noise 0.9 ["page number label"] noise 0.9 frontmatter_main_zone support_like short_fragment False False
3 1 1 header Acta Orthop Scand 1996; 67 (3): 264–268 [584, 29, 886, 50] noise 0.9 ["header label"] noise 0.9 frontmatter_main_zone support_like none False False
4 1 2 doc_title Atrophy of the supraspinatus belly [83, 114, 664, 155] paper_title 0.8 ["page-1 zone title_zone: Atrophy of the supraspinatus belly"] paper_title 0.8 frontmatter_main_zone support_like none True True
5 1 3 doc_title Assessment by MRI in 55 patients with rotator cuff pathology [83, 169, 769, 199] paper_title 0.8 ["page-1 zone title_zone: Assessment by MRI in 55 patients with rotator cuff pathology"] paper_title 0.8 frontmatter_main_zone support_like none True True
6 1 4 text Hervé Thomazeau, Yann Rolland, Christophe Lucas, Jean-Marie Duval and Frantz Langlais [84, 243, 781, 293] authors 0.8 ["page-1 zone author_zone: Herv\u00e9 Thomazeau, Yann Rolland, Christophe Lucas, Jean-Marie "] authors 0.8 frontmatter_main_zone support_like none True True
7 1 5 abstract A study of 5 fresh cadaveric shoulders demonstrated that an oblique-sagittal plane which crosses the scapula through the medial border of the coracoid process offers a view of the supraspinatus fossa [85, 339, 478, 439] abstract_body 0.85 ["abstract label from Paddle OCR"] abstract_body 0.85 frontmatter_main_zone support_like none True True
8 1 6 abstract This view could easily be reproduced by MRI and we called it the Y-shaped view. It allowed a reliable measurement of supraspinatus muscle atrophy by the calculation of the occupation ratio (R) which i [85, 439, 480, 563] abstract_body 0.85 ["abstract label from Paddle OCR"] abstract_body 0.85 frontmatter_main_zone support_like none True True
9 1 7 abstract This ratio was calculated in a prospective study based on 55 shoulders divided into 3 groups with different rotator cuff status: group I, 15 controls; group II, 10 degenerative cuffs, without tears; g [495, 338, 890, 562] abstract_body 0.85 ["abstract label from Paddle OCR"] abstract_body 0.85 frontmatter_main_zone support_like none True True
10 1 8 text Rennes University Hospital, France. Correspondence: Dr. Hervé Thomazeau, University Hospital, 16 Bd de Bulgarie, F-35056 Rennes, France. Tel +33 99-26-71-67. Fax -32-80-06 Submitted 95-11-04. Accepted [87, 578, 842, 631] frontmatter_noise 0.8 ["page-1 zone journal_furniture_zone: Rennes University Hospital, France. Correspondence: Dr. Herv"] frontmatter_noise 0.8 frontmatter_main_zone support_like none False False
11 1 9 text Occasionally, the anatomical and functional results of reconstruction of full-thickness rotator cuff tears, where two tendons or more have been primarily sutured, are disappointing (Harryman et al. 19 [86, 700, 480, 855] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
12 1 10 text Such a limit cannot be appreciated solely on the basis of duration of symptoms, clinical presentation, or radiological and sonographic evaluation of the size of the tear. Probably an evaluation of the [87, 856, 482, 1058] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
13 1 11 paragraph_title Material and methods [90, 1109, 299, 1132] section_heading 0.5 ["unnumbered paragraph_title on page 1 outside title zone: Material and methods"] section_heading 0.5 body_zone heading_like none True True
14 1 12 text The aims of our anatomical study were to define the oblique-sagittal plane which offers the most extended osseous limits for the supraspinatus fossa and to define the lateral limits of the supraspinat [90, 1143, 482, 1254] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
15 1 13 text The purpose of the radioclinical study was then to find a simple and reproducible method of measuring the muscular atrophy based on the oblique-sagittal plane previously defined by the anatomical stud [90, 1255, 484, 1321] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
16 1 14 text [496, 700, 889, 745] ocr_text_missing 0.8 ["ocr detected text region (raw_label=text) but no text extracted; no pdf backfill available"] ocr_text_missing 0.8 body_zone body_like empty True True
17 1 15 paragraph_title Anatomical study [498, 762, 652, 784] section_heading 0.5 ["unnumbered paragraph_title on page 1 outside title zone: Anatomical study"] section_heading 0.5 body_zone body_like short_fragment True True
18 1 16 text At first, 5 non-embalmed cadaveric shoulders were studied to assess the limits of the supraspinatus fossa. As this evaluation was independent of the rotator cuff status, dissection of the supraspinatu [497, 788, 893, 1098] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
19 1 17 text Another 5 non-embalmed shoulders were then dissected to define the lateral attachment of the supraspinatus muscle on the scapula. The clavicle and the acromion were removed en bloc with the trapezoid [499, 1098, 894, 1320] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
20 1 18 footer Copyright © Scandinavian University Press 1996. ISSN 0001-6470. Printed in Sweden – all rights reserved. [92, 1361, 794, 1383] noise 0.9 ["footer label"] noise 0.9 body_zone body_like none False False
21 2 0 header Acta Orthop Scand 1996; 67 (3): 264–268 [83, 28, 388, 49] noise 0.9 ["header label"] noise 0.9 body_zone body_like none False False
22 2 1 number 265 [853, 30, 884, 48] noise 0.9 ["page number label"] noise 0.9 body_zone body_like short_fragment False False
23 2 2 image [89, 117, 882, 344] media_asset 0.85 ["media label: image"] media_asset 0.85 body_zone body_like empty True True
24 2 3 figure_title Figure 3. Position of the lateral attachment of the supraspinatus muscle in its fossa from an above view. [557, 275, 884, 309] figure_caption 0.92 ["figure_title label: Figure 3. Position of the lateral attachment of the supraspi"] figure_caption 0.92 display_zone legend_like figure_number True True
25 2 4 figure_title Figure 2. Calculation of the ratio O/M between osseous (O) and muscular (M) limits of the supraspinatus fossa on an oblique-sagittal anatomical slice. [410, 329, 886, 364] figure_caption 0.92 ["figure_title label: Figure 2. Calculation of the ratio O/M between osseous (O) a"] figure_caption 0.92 display_zone legend_like figure_number True True
26 2 5 figure_title Figure 1. 4 levels of oblique-sagittal cross-sections of frozen shoulders (scapula viewed from the back). [252, 386, 580, 421] figure_caption 0.92 ["figure_title label: Figure 1. 4 levels of oblique-sagittal cross-sections of fro"] figure_caption 0.92 display_zone legend_like figure_number True True
27 2 6 text muscular fibers and the aponeurosis attached to the bone. To define the lateral limit of this attachment, the distances were recorded between this limit and 3 defined landmarks: A, bottom of the scapu [82, 456, 477, 700] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
28 2 7 paragraph_title Radioclinical study [83, 717, 250, 739] subsection_heading 0.6 ["unnumbered paragraph_title, inferred level subsection_heading: Radioclinical study"] subsection_heading 0.6 body_zone body_like short_fragment True True
29 2 8 text A prospective clinical and MRI study was then carried out on 55 persons divided into 3 groups. Group I consisted of 15 asymptomatic volunteers: 5 men and 10 women, mean age 45 (14–74) years. Group II [82, 745, 477, 1076] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
30 2 9 text At the time of surgery, the size of the tear was anatomically described in both the sagittal and coronal planes, independently of the size of the patient. In the sagittal plane, the lesions were class [83, 1077, 479, 1320] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
31 2 10 text [492, 456, 887, 634] ocr_text_missing 0.8 ["ocr detected text region (raw_label=text) but no text extracted; no pdf backfill available"] ocr_text_missing 0.8 body_zone body_like empty True True
32 2 11 text All MRI images were obtained with a 1Tesla unit (Magneton Impact, Siemens, Germany) The patient lay supine in a pain-free position, with the arm in neutral rotation along the chest and a curved loop s [492, 633, 887, 922] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
33 2 12 text The quantitative analysis was then performed on the spin-echo T1-weighted oblique-sagittal images (TR: 480ms, TE: 12ms, FOV: 250×250, matrix: 380×512). [493, 923, 888, 1009] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
34 2 13 text To evaluate the atrophy of the supraspinatus muscle, we calculated the occupation ratio (R) of the supraspinatus fossa by the muscle belly. This analysis was based on the ratio between the surface of [493, 1012, 889, 1275] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
35 2 14 text The correlation and relative error between the results achieved by the 3 readers were calculated and [494, 1276, 889, 1319] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
36 3 0 number 266 [85, 23, 118, 42] noise 0.9 ["page number label"] noise 0.9 body_zone body_like short_fragment False False
37 3 1 header Acta Orthop Scand 1996; 67 (3): 264–268 [585, 23, 888, 43] noise 0.9 ["header label"] noise 0.9 body_zone body_like none False False
38 3 2 image [176, 109, 370, 337] media_asset 0.85 ["media label: image"] media_asset 0.85 body_zone unknown_like empty True True
39 3 3 figure_title Figure 4. Calculation of the occupation ratio R on the oblique-sagittal view. S1 Surface of the supraspinatus muscle, S2 surface of the entire supraspinatus fossa. [85, 359, 479, 412] figure_caption 0.92 ["figure_title label: Figure 4. Calculation of the occupation ratio R on the obliq"] figure_caption 0.92 display_zone legend_like figure_number True True
40 3 4 text the final result was a mean of these values expressed from 0 (empty fossa) to 1 (full fossa). Then the mean values of R in the different clinical populations were analyzed, using a statistical analysi [84, 452, 480, 586] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
41 3 5 paragraph_title Results Radioanatomical study [86, 640, 287, 691] subsection_heading 0.6 ["unnumbered paragraph_title, inferred level subsection_heading: Results Radioanatomical study"] subsection_heading 0.6 body_zone heading_like none True True
42 3 6 footer [86, 670, 287, 691] noise 0.9 ["footer label"] noise 0.9 body_zone body_like empty False False
43 3 7 text In the first 5 shoulders, slice 1 demonstrated the highest 0/M ratio, between 2.2 and 2.75 (Table 1). At this level, the scapula is cut through the medial border of the coracoid process and the latera [85, 695, 482, 983] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
44 3 8 text The lateral bony attachment of the supraspinatus muscle always remained medial to points A and B (mean distances were, respectively, -5.4 mm and -1.8 mm), but spread laterally to point C in 4 of the 5 [87, 985, 482, 1161] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
45 3 9 text This preliminary anatomical study suggested that slice 1 provided the largest bony limits for the supraspinatus fossa and probably represented the most reliable level for the calculation of the occupa [87, 1162, 484, 1317] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
46 3 10 figure_title Table 1. Ratio between osseous (O) and muscular (M) limits of the supraspinatus fossa [495, 113, 888, 150] table_caption 0.9 ["table prefix matched: Table 1. Ratio between osseous (O) and muscular (M) limits o"] table_caption 0.9 display_zone table_caption_like table_number True True
47 3 11 table <table><tr><td>Shoulder</td><td>Slice 1</td><td>Slice 2</td><td>Slice 3</td><td>Slice 4</td></tr><tr><td>1</td><td>2.5</td><td>0.6</td><td>1</td><td>0.75</td></tr><tr><td>2</td><td>2.4</td><td>0.6</td [502, 170, 882, 322] table_html 0.95 ["inline table HTML"] table_html 0.95 body_zone body_like none True False
48 3 12 table <table><tr><td>Shoulder</td><td>Point A</td><td>Point B</td><td>Point C</td></tr><tr><td>1</td><td>$ -5 $</td><td>$ -3 $</td><td>$ -2 $</td></tr><tr><td>2</td><td>$ -5 $</td><td>$ -2 $</td><td>$ +5 $< [504, 418, 876, 571] table_html 0.95 ["inline table HTML"] table_html 0.95 body_zone body_like none True False
49 3 13 figure_title Table 2. Position of the lateral attachment of the supraspinatus (in mm) [496, 362, 888, 400] table_caption 0.9 ["table prefix matched: Table 2. Position of the lateral attachment of the supraspin"] table_caption 0.9 display_zone table_caption_like table_number True True
50 3 14 text ue the radioclinical study by measuring the occupa-tion ratio on the MRI view corresponding to this anat-tomical slice 1: we called it the Y-shaped view. [496, 629, 889, 696] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
51 3 15 paragraph_title Radioclinical study [498, 712, 665, 734] subsection_heading 0.6 ["unnumbered paragraph_title, inferred level subsection_heading: Radioclinical study"] subsection_heading 0.6 body_zone body_like short_fragment True True
52 3 16 text This Y-shaped view was found to be easily reproducible, since it was available for all the 55 clinical cases. When 3 readers (r1, r2, r3) calculated the occupation ratio, the relative error was 12 (3– [496, 738, 891, 872] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
53 3 17 text The highest value obtained for the occupation ratio, 0.87, was found in group I. Aggravation of the tendinopathy resulted in a decreased ratio (Table 3). The differences in the mean values of R betwee [497, 873, 893, 1116] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
54 3 18 text In group III, there was no difference in the occupation ratio when the tear involved only the supraspinatus or extended to the rotator interval. The difference dramatically increased when the infraspi [499, 1117, 893, 1247] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
55 3 19 text The drop in this ratio also correlated with the coronal retraction of the tendon stump: 0.59 (SD 0.11) for grade 1, 0.41 (0.12) for grade 2 and 0.30 (0.11) for grade 3. [500, 1248, 895, 1316] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
56 4 0 header Acta Orthop Scand 1996; 67 (3): 264–268 [80, 24, 384, 45] noise 0.9 ["header label"] noise 0.9 body_zone body_like none False False
57 4 1 number 267 [851, 23, 882, 42] noise 0.9 ["page number label"] noise 0.9 body_zone body_like short_fragment False False
58 4 2 figure_title Table 3. Occupation ratio of the supraspinatus fossa related to grade of supraspinatus tendinopathy [79, 115, 472, 151] table_caption 0.9 ["table prefix matched: Table 3. Occupation ratio of the supraspinatus fossa related"] table_caption 0.9 display_zone table_caption_like table_number True True
59 4 3 table <table><tr><td>Grade</td><td>n</td><td>Occupation ratio, mean (SD)</td></tr><tr><td>I</td><td>15</td><td>0.7 (0.09)</td></tr><tr><td>II</td><td>10</td><td>0.62 (0.11)</td></tr><tr><td>III</td><td>30</ [87, 171, 467, 270] table_html 0.95 ["inline table HTML"] table_html 0.95 body_zone body_like none True False
60 4 4 figure_title Table 4. Occupation ratio of the supraspinatus fossa related to the extent of the tear [82, 316, 476, 352] table_caption 0.9 ["table prefix matched: Table 4. Occupation ratio of the supraspinatus fossa related"] table_caption 0.9 display_zone table_caption_like table_number True True
61 4 5 table <table><tr><td>Tears</td><td>n</td><td>Occupation ratio, mean (SD)</td></tr><tr><td>Supraspinatus alone</td><td>11</td><td>0.55 (0.14)</td></tr><tr><td>Supraspinatus and rotator interval</td><td>4</td [90, 373, 469, 516] table_html 0.95 ["inline table HTML"] table_html 0.95 body_zone body_like none True False
62 4 6 vision_footnote Decrease of the occupation ratio is significant between a tear of the supraspinatus alone, and a tear which extends to the infraspinatus or to more than 2 tendons (p 0.001) [89, 526, 470, 578] footnote 0.7 ["vision_footnote label: Decrease of the occupation ratio is significant between a te"] footnote 0.7 body_zone body_like none True True
63 4 7 text grade 3 (p 0.0004). Lastly, when a large capsulotomy was required for the repair, the ratio was 0.33 (0.09) and up to 0.50 (0.16) when such capsulotomy was not necessary (p 0.004). [84, 630, 478, 720] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
64 4 8 paragraph_title Discussion [87, 774, 195, 796] section_heading 0.9 ["explicit scholarly heading: Discussion"] section_heading 0.9 body_zone heading_like canonical_section_name True True
65 4 9 text Tendinopathies of the rotator cuff muscles should be considered as part of a disease of the whole muscle. Previous experimental and radioclinical studies have demonstrated that the muscles atrophy and [85, 808, 481, 1074] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
66 4 10 text The calculation of the occupation ratio of the supraspinatus is reliable. It concerns an easily delineated surface, which is compared to an anatomical landmark represented by the limits of the suprasp [87, 1075, 483, 1319] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
67 4 11 image [563, 119, 846, 509] media_asset 0.85 ["media label: image"] media_asset 0.85 body_zone unknown_like empty True True
68 4 12 figure_title Figure 5. MRI oblique-sagittal view demonstrating the lateral attachment of the supraspinatus muscle on the spine of the scapula, above the the supraglenoid notch (black arrows). [495, 527, 888, 579] figure_caption 0.92 ["figure_title label: Figure 5. MRI oblique-sagittal view demonstrating the latera"] figure_caption 0.92 display_zone legend_like figure_number True True
69 4 13 text We found that this bony landmark made the analysis easily reproducible from one patient to another and in the course of time. Thus, pre- and postoperative evaluations become possible, especially as th [496, 632, 892, 876] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
70 4 14 text Our anatomical study demonstrated that the supraspinatus belly insertion spreads more laterally than has been classically described (Gardner et al. 1980). This attachment on the anterior aspect of the [497, 876, 893, 1075] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
71 4 15 text The radioclinical part of our study demonstrated that the ratio was never as high as 1.0 because of the fatty environment, particularly on the anterosuprior aspect of the supraspinatus muscle. This fa [499, 1076, 896, 1320] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
72 5 0 number 268 [86, 31, 118, 49] noise 0.9 ["page number label"] noise 0.9 body_zone body_like short_fragment False False
73 5 1 header Acta Orthop Scand 1996; 67 (3): 264–268 [585, 31, 888, 50] noise 0.9 ["header label"] noise 0.9 body_zone body_like none False False
74 5 2 image [80, 108, 355, 441] media_asset 0.85 ["media label: image"] media_asset 0.85 body_zone unknown_like empty True True
75 5 3 image [370, 110, 639, 441] media_asset 0.85 ["media label: image"] media_asset 0.85 body_zone unknown_like empty True True
76 5 4 image [653, 109, 878, 440] media_asset 0.85 ["media label: image"] media_asset 0.85 body_zone unknown_like empty True True
77 5 5 figure_title Figure 6. 3 grades of supraspinatus atrophy. From left to right: stage I, stage II, and stage III (the latter with associated infraspinatus atrophy). [87, 460, 889, 495] figure_caption 0.92 ["figure_title label: Figure 6. 3 grades of supraspinatus atrophy. From left to ri"] figure_caption 0.92 display_zone legend_like figure_number True True
78 5 6 text phy and fatty infiltration are two different expressions of the same muscle disease and this previous disagreement could be explained by the better sensitivity of the quantitative analysis we propose. [86, 547, 479, 635] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
79 5 7 text In any case, the supraspinatus tendon tear represents the first turning-point of this muscle disease. The occupation ratio is still above 0.5 when the tear involves the supraspinatus alone or extends [87, 637, 481, 924] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 tail_nonref_hold_zone body_like none True True
80 5 8 text The need for a large capsulotomy is associated with such a low occupation ratio, itself correlated with the extent of the tear, and one may question the elasticity and viability of the supraspinatus t [88, 926, 482, 1121] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 tail_nonref_hold_zone body_like none True True
81 5 9 text We propose a classification of supraspinatus belly atrophy based on the occupation ratio of the supraspinatus fossa (Figure 6). In the case of a ratio between 1.00 and 0.60 (stage I), the muscle can b [89, 1124, 484, 1300] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 tail_nonref_hold_zone body_like none True True
82 5 10 text [497, 546, 891, 681] ocr_text_missing 0.8 ["ocr detected text region (raw_label=text) but no text extracted; no pdf backfill available"] ocr_text_missing 0.8 body_zone body_like empty True True
83 5 11 paragraph_title References [500, 713, 611, 734] reference_heading 0.9 ["references heading: References"] reference_heading 0.9 reference_zone heading_like short_fragment True True
84 5 12 reference_content Bigliani L U, Dalsey R M, McCann P D, April E W. An anatomical study of the suprascapular nerve. Arthroscopy 1990; 6 (4): 301-5. [501, 744, 890, 798] reference_item 0.85 ["reference content label: Bigliani L U, Dalsey R M, McCann P D, April E W. An anatomic"] reference_item 0.85 reference_zone reference_like citation_line True True
85 5 13 reference_content Björkenheim J M. Structure and function of the rabbit's supraspinatus muscle after resection of its tendon. Acta Orthop Scand 1989; 60 (4): 461-3. [501, 803, 889, 857] reference_item 0.85 ["reference content label: Bj\u00f6rkenheim J M. Structure and function of the rabbit's supr"] reference_item 0.85 reference_zone reference_like none True True
86 5 14 reference_content Gardner E, Gray DJ, O'Rahilly R. Anatomy. W.B. Saunders, Philadelphia-London-Toronto 1980. [501, 861, 890, 897] reference_item 0.85 ["reference content label: Gardner E, Gray DJ, O'Rahilly R. Anatomy. W.B. Saunders, Phi"] reference_item 0.85 reference_zone reference_like citation_line True True
87 5 15 reference_content Goutallier D, Bernageau J, Patte D. Assessment of the trophicity of the muscles of the ruptured rotator cuff by CT scan. In: Surgery of the shoulder. (Eds. Post M, Morrey B F, Hawkins R J). Mosby-Year [503, 902, 891, 990] reference_item 0.85 ["reference content label: Goutallier D, Bernageau J, Patte D. Assessment of the trophi"] reference_item 0.85 reference_zone reference_like citation_line True True
88 5 16 reference_content Goutallier D, Postel J M, Bernageau J, Lavau L, Voisin M C. Fatty degeneration in cuff rupture. Pre- and postoperative evaluation by CT scan. Clin Orthop 1994; 304: 78-83. [502, 996, 891, 1050] reference_item 0.85 ["reference content label: Goutallier D, Postel J M, Bernageau J, Lavau L, Voisin M C. "] reference_item 0.85 reference_zone reference_like citation_line True True
89 5 17 reference_content Harryman D T, Mack L A, Wang K Y, Jackins S E, Richardson M L, Matsen F A. Repairs of the rotator cuff: correlation of functional results with integrity of the cuff. J. Bone Joint Surg (Am) 1991; 73 ( [503, 1054, 891, 1126] reference_item 0.85 ["reference content label: Harryman D T, Mack L A, Wang K Y, Jackins S E, Richardson M "] reference_item 0.85 reference_zone reference_like citation_line True True
90 5 18 reference_content Maughan R J, Watson J S, Weir J. Strength and cross-sectional area of human skeletal muscle. J Physiol 1983; 338: 37-49. [502, 1131, 891, 1184] reference_item 0.85 ["reference content label: Maughan R J, Watson J S, Weir J. Strength and cross-sectiona"] reference_item 0.85 reference_zone reference_like citation_line True True
91 5 19 reference_content Nakagaki K, Osaki J, Tomita Y, Tamai S. Alterations in the supraspinatus belly with rotator cuff tearing: Evaluation with magnetic resonance imaging. J Shoulder Elbow Surg 1994; 3 (2): 88-93. [504, 1189, 892, 1260] reference_item 0.85 ["reference content label: Nakagaki K, Osaki J, Tomita Y, Tamai S. Alterations in the s"] reference_item 0.85 reference_zone reference_like citation_line True True
92 5 20 reference_content Patte D. Classification of rotator cuff lesions. Clin Orthop 1990; 254: 81-6. [504, 1265, 892, 1299] reference_item 0.85 ["reference content label: Patte D. Classification of rotator cuff lesions. Clin Orthop"] reference_item 0.85 reference_zone reference_like citation_line True True

View file

@ -0,0 +1,12 @@
{
"layout": {
"mixed_tail_pages": [5],
"min_reference_items_on_mixed_tail_page": 3
},
"document": {
"notes": [
"5-page mixed_tail fixture. Page 5: 4 body_paragraph + 9 reference_item blocks.",
"Expands coverage for mixed_tail layout classification."
]
}
}

View file

@ -0,0 +1,100 @@
page,block_id,raw_label,content_preview,bbox,role,role_confidence,evidence,seed_role,seed_confidence,zone,style_family,marker_type,render_default,index_default
1,0,header,"J Shoulder Elbow Surg (2015) 24, 1948-1953","[73, 64, 377, 84]",noise,0.9,"[""header label""]",noise,0.9,frontmatter_main_zone,support_like,none,False,False
1,1,header_image,,"[75, 102, 184, 200]",unknown_structural,0.2,"[""unrecognized label 'header_image'""]",unknown_structural,0.2,frontmatter_main_zone,support_like,empty,False,True
1,2,header,ELSEVIER,"[75, 205, 185, 227]",noise,0.9,"[""header label""]",noise,0.9,frontmatter_main_zone,support_like,short_fragment,False,False
1,3,header,"JOURNAL OF
SHOULDER AND
E_LBOW
SURGERY","[915, 79, 1080, 176]",noise,0.9,"[""header label""]",noise,0.9,frontmatter_main_zone,support_like,none,False,False
1,4,header,www.elsevier.com/locate/ymse,"[879, 192, 1083, 211]",noise,0.9,"[""header label""]",noise,0.9,frontmatter_main_zone,support_like,none,False,False
1,5,doc_title,Correlation between glenoid inclination and critical shoulder angle: a radiographic and computed tomography study,"[71, 308, 810, 451]",paper_title,0.8,"[""page-1 zone title_zone: Correlation between glenoid inclination and critical shoulde""]",paper_title,0.8,frontmatter_main_zone,support_like,none,True,True
1,6,image,,"[947, 312, 1083, 367]",media_asset,0.85,"[""media label: image""]",media_asset,0.85,frontmatter_main_zone,support_like,empty,True,True
1,7,text,"Matthew Daggett, DO, MBA*, Birgit Werner, MD, Philipp Collin, MD, Marc-Olivier Gauci, MD, Jean Chaoui, PhD, Gilles Walch, MD","[71, 485, 894, 552]",authors,0.8,"[""page-1 zone author_zone: Matthew Daggett, DO, MBA*, Birgit Werner, MD, Philipp Collin""]",authors,0.8,frontmatter_main_zone,support_like,none,True,True
1,8,text,"Centre Orthopédique Santé, Hôpital Privé Jean Mermoz, Lyon, France","[72, 592, 659, 617]",affiliation,0.8,"[""page-1 zone affiliation_zone: Centre Orthop\u00e9dique Sant\u00e9, H\u00f4pital Priv\u00e9 Jean Mermoz, Lyon, ""]",affiliation,0.8,frontmatter_main_zone,support_like,none,True,True
1,9,abstract,Background: Increased critical shoulder angles consist of both the acromial cover and glenoid inclination and have been found in patients with rotator cuff pathology. The purpose of this study was to ,"[81, 662, 863, 1109]",abstract_body,0.85,"[""abstract label from Paddle OCR""]",abstract_body,0.85,frontmatter_main_zone,support_like,none,True,True
1,10,text,Keywords: Critical shoulder angle; glenoid inclination; rotator cuff tears,"[84, 1104, 625, 1129]",frontmatter_noise,0.7,"[""frontmatter noise text: Keywords: Critical shoulder angle; glenoid inclination; rota""]",frontmatter_noise,0.7,frontmatter_main_zone,support_like,none,False,False
1,11,text,"Originally described by Moor et al, $ ^{12} $ the critical shoulder angle (CSA) has been used to analyze individual, quantitative shoulder anatomy. The CSA is measured by a line connecting the superio","[595, 1215, 1086, 1384]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
1,12,footnote,The Institutional Review Board of the Ethical Committee of Hôpital Privé Jean Mermoz and the Centre Orthopédique Santé approved this study.,"[71, 1284, 557, 1326]",footnote,0.7,"[""footnote label: The Institutional Review Board of the Ethical Committee of H""]",footnote,0.7,body_zone,body_like,none,True,True
1,13,footnote,"*Reprint requests: Matthew Daggett, DO, MBA, Centre Orthopédique Santé, 24 avenue Paul Santé, F-69008 Lyon, France.","[72, 1325, 559, 1364]",footnote,0.7,"[""footnote label: *Reprint requests: Matthew Daggett, DO, MBA, Centre Orthop\u00e9d""]",footnote,0.7,body_zone,body_like,none,True,True
1,14,footnote,E-mail address: matthewdaggett@gmail.com (M. Daggett).,"[94, 1365, 488, 1386]",footnote,0.7,"[""footnote label: E-mail address: matthewdaggett@gmail.com (M. Daggett).""]",footnote,0.7,body_zone,body_like,none,True,True
1,15,footer,1058-2746/$ - see front matter © 2015 Journal of Shoulder and Elbow Surgery Board of Trustees. http://dx.doi.org/10.1016/j.jse.2015.07.013,"[71, 1407, 724, 1448]",noise,0.9,"[""footer label""]",noise,0.9,body_zone,body_like,none,False,False
2,0,header,Correlation of glenoid inclination and CSA,"[104, 64, 455, 88]",noise,0.9,"[""header label""]",noise,0.9,body_zone,body_like,none,False,False
2,1,number,1949,"[1069, 65, 1114, 86]",noise,0.9,"[""page number label""]",noise,0.9,body_zone,body_like,short_fragment,False,False
2,2,text,"and glenoid inclination. Previous investigations have found that the CSA can be a predictor of shoulder pathology, with increased CSA angles amongst individuals with rotator cuff tears and smaller ang","[101, 110, 590, 253]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
2,3,text,"Glenoid inclination, a component of the CSA, has been separately identified as a risk factor for tears of the supraspinatus tendon. $ ^{7,24} $ A reproducible method to measure glenoid inclination, te","[101, 254, 590, 539]",affiliation,0.8,"[""page-1 zone affiliation_zone: Glenoid inclination, a component of the CSA, has been separa""]",affiliation,0.8,body_zone,body_like,none,True,True
2,4,text,The first purpose of this study was to determine whether there is an association between the CSA and glenoid inclination of patients undergoing primary shoulder arthroplasty. We hypothesized that ther,"[101, 541, 590, 829]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
2,5,paragraph_title,Materials and methods,"[103, 869, 346, 894]",section_heading,0.9,"[""explicit scholarly heading: Materials and methods""]",section_heading,0.9,body_zone,heading_like,canonical_section_name,True,True
2,6,text,"The study included 50 shoulders in 48 individuals (27 women, 21 men) who were an average age of 72.9 years (range 61-84 years). The selection of 50 shoulders was made from patients within the clinical","[101, 918, 589, 1074]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
2,7,paragraph_title,Patient groups,"[103, 1097, 249, 1123]",subsection_heading,0.6,"[""unnumbered paragraph_title, inferred level subsection_heading: Patient groups""]",subsection_heading,0.6,body_zone,heading_like,short_fragment,True,True
2,8,text,"Inclusion in the osteoarthritic group required confirmation by both surgeons of A1 classification, as defined by Walch et al., $ ^{22} $ centered head in both coronal and axial plans with minor glenoi","[102, 1147, 590, 1366]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
2,9,text,"For each patient, multiple different imaging measurements were obtained from anteroposterior (AP) radiographs and standard, preoperative CT scans of the shoulder. First, the CSA was","[101, 1368, 590, 1433]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
2,10,image,,"[649, 113, 1095, 408]",media_asset,0.85,"[""media label: image""]",media_asset,0.85,body_zone,unknown_like,empty,True,True
2,11,figure_title,Figure 1 Radiograph shows A1 glenoid.,"[711, 424, 1031, 447]",figure_caption,0.92,"[""figure_title label: Figure 1 Radiograph shows A1 glenoid.""]",figure_caption,0.92,display_zone,legend_like,figure_number,True,True
2,12,image,,"[649, 484, 1095, 1033]",media_asset,0.85,"[""media label: image""]",media_asset,0.85,body_zone,unknown_like,empty,True,True
2,13,figure_title,Figure 2 Radiograph shows E0 glenoid.,"[712, 1051, 1031, 1075]",figure_caption,0.92,"[""figure_title label: Figure 2 Radiograph shows E0 glenoid.""]",figure_caption,0.92,display_zone,legend_like,figure_number,True,True
2,14,text,"obtained via a standard AP shoulder X-ray image with the arm in neutral rotation. Described by Moor et al, $ ^{12} $ the CSA was measured by a line connecting the superior and inferior bony margins of","[626, 1102, 1116, 1234]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
2,15,text,"The second measurement performed on AP radiographs of the shoulder was the β-angle, as described by Maurer et al., $ ^{10} $ which is the angle is formed by the intersection of a line through the floo","[626, 1234, 1116, 1433]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,0,number,1950,"[74, 65, 119, 86]",noise,0.9,"[""page number label""]",noise,0.9,body_zone,body_like,short_fragment,False,False
3,1,header,M. Daggett et al.,"[938, 64, 1081, 88]",noise,0.9,"[""header label""]",noise,0.9,body_zone,body_like,short_fragment,False,False
3,2,image,,"[92, 113, 537, 732]",media_asset,0.85,"[""media label: image""]",media_asset,0.85,body_zone,unknown_like,empty,True,True
3,3,figure_title,Figure 3 Illustration shows the critical shoulder angle (CSA) measurement. The CSA is defined as a line connecting the superior and inferior bony margins of the glenoid and an intersecting line drawn ,"[71, 751, 557, 861]",figure_caption,0.92,"[""figure_title label: Figure 3 Illustration shows the critical shoulder angle (CSA""]",figure_caption,0.92,display_zone,legend_like,figure_number,True,True
3,4,text,"The glenoid inclination was then measured on the CT scan using the validated Glenosys 3-D software. $ ^{23} $ After segmentation and reformatting, the software program determines the planes of the sca","[70, 919, 560, 1117]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,5,paragraph_title,Statistics,"[71, 1141, 167, 1165]",sub_subsection_heading,0.6,"[""unnumbered paragraph_title, inferred level sub_subsection_heading: Statistics""]",sub_subsection_heading,0.6,body_zone,heading_like,short_fragment,True,True
3,6,text,Regression analysis was used to analyze the relationship between CSA and glenoid inclination. The Mann-Whitney U test was used to compare the CSA on AP radiographs between the osteoarthritis (A1 group,"[70, 1192, 560, 1412]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,7,image,,"[618, 112, 1062, 559]",media_asset,0.85,"[""media label: image""]",media_asset,0.85,body_zone,unknown_like,empty,True,True
3,8,figure_title,Figure 4 Measuring glenoid inclination using the $ \beta $-angle on an X-ray image: The $ \beta $-angle is measured from the angle formed by a line along the floor of the supraspinatus fossa and the,"[596, 575, 1084, 663]",figure_caption,0.92,"[""figure_title label: Figure 4 Measuring glenoid inclination using the $ \\beta $-""]",figure_caption,0.92,display_zone,legend_like,figure_number,True,True
3,9,paragraph_title,Results,"[597, 688, 680, 713]",section_heading,0.9,"[""explicit scholarly heading: Results""]",section_heading,0.9,body_zone,heading_like,canonical_section_name,True,True
3,10,text,"A statistically significant correlation was found between the glenoid inclination and the CSA in both groups. Glenoid inclination demonstrated a linear relationship to the CSA ( $ R^{2} = 0.7426 $, P ","[594, 742, 1084, 956]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,11,text,"There were distinct differences in the CSA between the 2 groups. Patients in the A1 group had a mean CSA of 27.7° (standard deviation [SD], 3.4°; range, 17°-33°), whereas patients in the E0 group had ","[594, 958, 1084, 1148]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
3,12,text,"There were also distinct differences of glenoid inclination between the 2 groups. As measured in the 3-D software, the A1 group had a mean glenoid inclination of 4.7° (SD, 5.6°, range, 9° to 13°), an","[594, 1150, 1084, 1414]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
4,0,header,Correlation of glenoid inclination and CSA,"[105, 64, 455, 87]",noise,0.9,"[""header label""]",noise,0.9,body_zone,body_like,none,False,False
4,1,number,1951,"[1069, 65, 1113, 85]",noise,0.9,"[""page number label""]",noise,0.9,body_zone,body_like,short_fragment,False,False
4,2,image,,"[148, 113, 1071, 571]",media_asset,0.85,"[""media label: image""]",media_asset,0.85,body_zone,unknown_like,empty,True,True
4,3,figure_title,"Figure 5 Screenshot shows output provided by the Glenosys (Imascap, Brest, France) 3-dimensional software, which provides anatomic details, including glenoid inclination.","[102, 591, 1113, 637]",figure_caption,0.92,"[""figure_title label: Figure 5 Screenshot shows output provided by the Glenosys (I""]",figure_caption,0.92,display_zone,legend_like,figure_number,True,True
4,4,text,"and the E0 group had a mean glenoid inclination of $ 13.6^{\circ} $ (SD, $ 4.6^{\circ} $; range, $ 7^{\circ}-24^{\circ} $).","[101, 684, 588, 732]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
4,5,paragraph_title,Discussion,"[102, 772, 219, 798]",section_heading,0.9,"[""explicit scholarly heading: Discussion""]",section_heading,0.9,body_zone,heading_like,canonical_section_name,True,True
4,6,text,"Previous studies have investigated the components of the CSA, but to our knowledge, this is the first study to find a correlation between the glenoid inclination and the CSA. The CSA demonstrated a li","[100, 827, 589, 1018]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
4,7,text,The effect of the CSA has been previously examined. Moor et al $ ^{12} $ found an increased CSA (>35°) in patients with rotator cuff tears and decreased CSA (<30°) in patients with osteoarthritis. An ,"[101, 1018, 590, 1377]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
4,8,text,"The massive rotator cuff tear (E0) group had a significant increase in superior glenoid inclination, with a mean glenoid inclination of $ 13.6^{\circ} $, whereas the osteoarthritic (A1)","[100, 1378, 590, 1450]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
4,9,chart,,"[645, 686, 1096, 998]",media_asset,0.85,"[""media label: chart""]",media_asset,0.85,body_zone,unknown_like,empty,True,True
4,10,figure_title,"Figure 6 The critical shoulder angle (CSA) and glenoid inclination, as measured by the B-angle on Glenosys (GI-GL_beta), demonstrated a near linear relationship. MRCT, massive rotator cuff tear; OA, o","[626, 1011, 1115, 1100]",figure_caption,0.92,"[""figure_title label: Figure 6 The critical shoulder angle (CSA) and glenoid incli""]",figure_caption,0.92,display_zone,legend_like,figure_number,True,True
4,11,text,group demonstrated glenoid inclination measurements similar to published anatomic data from Churchill et al $ ^{2} $ of $ 4^{\circ} $ to $ 4.5^{\circ} $. Although rotator cuff disease is likely seco,"[626, 1136, 1116, 1303]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
4,12,text,"Previous studies have demonstrated a potential link between glenoid inclination and rotator cuff tears as well as superior migration of the humeral head $ ^{4,7,8,20,21,24} $; however, others have fai","[626, 1304, 1116, 1449]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,body_zone,body_like,none,True,True
5,0,number,1952,"[74, 65, 118, 86]",noise,0.9,"[""page number label""]",noise,0.9,body_zone,body_like,short_fragment,False,False
5,1,header,M. Daggett et al.,"[938, 65, 1081, 88]",noise,0.9,"[""header label""]",noise,0.9,body_zone,body_like,short_fragment,False,False
5,2,text,"other. $ ^{6,16-18} $ The significant difference in glenoid inclination found between the E0 group and A1 group may partially explain this phenomenon.","[70, 109, 557, 181]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,tail_nonref_hold_zone,unknown_like,none,True,True
5,3,text,Increased glenoid inclination found in the E0 group could support the theory that tears of the rotator cuff are caused by tendon overload rather than by acromial impingement. Gerber et al $ ^{5} $ dem,"[70, 183, 559, 467]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,tail_nonref_hold_zone,unknown_like,none,True,True
5,4,text,"Our study has certain limitations. The potential for misclassification of glenoid subtype exists; however, 2 physicians independently validated the classification on the X-ray images. The interobserve","[70, 469, 558, 636]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,tail_nonref_hold_zone,unknown_like,none,True,True
5,5,text,"Although measurements using AP radiographs showed a less severe difference in glenoid inclination between the 2 groups, this was likely secondary to inadequate visualization of the supraspinatus fossa","[71, 637, 557, 875]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,tail_nonref_hold_zone,unknown_like,none,True,True
5,6,text,Our findings raise many questions about the role vertical orientation of the glenoid has in the causation or evolution of many shoulder disorders. Further investigation into the role superior glenoid ,"[70, 876, 557, 1018]",body_paragraph,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,tail_nonref_hold_zone,unknown_like,none,True,True
5,7,paragraph_title,Conclusion,"[85, 1069, 200, 1094]",structured_insert,0.9,"[""explicit scholarly heading: Conclusion""]",section_heading,0.9,tail_nonref_hold_zone,heading_like,canonical_section_name,False,False
5,8,text,Glenoid inclination is linearly correlated with the critical shoulder angle and is significantly increased in patients with massive rotator cuff tears.,"[82, 1117, 546, 1190]",structured_insert,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,tail_nonref_hold_zone,unknown_like,none,False,False
5,9,paragraph_title,Disclaimer,"[85, 1261, 198, 1287]",structured_insert,0.6,"[""unnumbered paragraph_title, inferred level sub_subsection_heading: Disclaimer""]",sub_subsection_heading,0.6,tail_nonref_hold_zone,heading_like,short_fragment,False,False
5,10,text,"The authors, their immediate families, and any research foundations with which they are affiliated have not received any financial payments or other benefits from any commercial entity related to the ","[82, 1311, 546, 1431]",structured_insert,0.6,"[""default body_paragraph for text label""]",body_paragraph,0.6,tail_nonref_hold_zone,unknown_like,none,False,False
5,11,paragraph_title,References,"[599, 110, 717, 135]",reference_heading,0.9,"[""references heading: References""]",reference_heading,0.9,reference_zone,heading_like,short_fragment,True,True
5,12,reference_content,"1. Bishop JL, Kline SK, Aalderink KJ, Zauel R, Bey MJ. Glenoid inclination: in vivo measures in rotator cuff tear patients and associations with superior glenohumeral joint translation. J Shoulder Elb","[608, 166, 1083, 262]",reference_item,0.85,"[""reference content label: 1. Bishop JL, Kline SK, Aalderink KJ, Zauel R, Bey MJ. Gleno""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,13,reference_content,"2. Churchill RS, Brems J, Kotschi H. Glenoid size, inclination and version: an anatomic study. J Shoulder Elbow Surg 2001;10:327-32.","[607, 266, 1082, 323]",reference_item,0.85,"[""reference content label: 2. Churchill RS, Brems J, Kotschi H. Glenoid size, inclinati""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,14,reference_content,"3. Favard L, Lautmann S, Sirveaux F, Oudet D, Kerjean Y, Huguet D. Hemiarthroplasty versus reverse arthroplasty in the treatment of osteoarthritis with massive rotator cuff tear. In: Walch G, Mole D, ","[608, 325, 1083, 422]",reference_item,0.85,"[""reference content label: 3. Favard L, Lautmann S, Sirveaux F, Oudet D, Kerjean Y, Hug""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,15,reference_content,"4. Flieg N, Gatti C, Doro L, Langenderfer J, Carpenter J, Hughes R. A stochastic analysis of glenoid inclination angle and superior migration of the humeral head. Clin Biomech (Bristol, Avon) 2008;23:","[608, 425, 1082, 503]",reference_item,0.85,"[""reference content label: 4. Flieg N, Gatti C, Doro L, Langenderfer J, Carpenter J, Hu""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,16,reference_content,"5. Gerber C, Snedeker JG, Baumgartner D, Viehöfer AF. Supraspinatus tendon load during abduction is dependent on the size of the critical shoulder angle: a biomechanical analysis. J Orthop Res 2014;32","[608, 506, 1082, 583]",reference_item,0.85,"[""reference content label: 5. Gerber C, Snedeker JG, Baumgartner D, Vieh\u00f6fer AF. Supras""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,17,reference_content,"6. Hamada K, Fukuda H, Mikasa M, Kobayashi Y. Roentgenographic findings in massive rotator cuff tears: a long-term observation. Clin Orthop Relat Res 1990;254:92-6.","[608, 585, 1082, 642]",reference_item,0.85,"[""reference content label: 6. Hamada K, Fukuda H, Mikasa M, Kobayashi Y. Roentgenograph""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,18,reference_content,"7. Hughes RE, Bryant CR, Hall JM, Wening J, Huston LJ, Kuhn JE, et al. Glenoid inclination is associated with full-thickness rotator cuff tears. Clin Orthop Relat Res 2003;407:86-91.","[608, 645, 1081, 702]",reference_item,0.85,"[""reference content label: 7. Hughes RE, Bryant CR, Hall JM, Wening J, Huston LJ, Kuhn ""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,19,reference_content,"8. Itoi E, Motzkin N, Morrey B. Scapular inclination and inferior stability of the shoulder. J Shoulder Elbow Surg 1992;1:131-9.","[607, 704, 1079, 742]",reference_item,0.85,"[""reference content label: 8. Itoi E, Motzkin N, Morrey B. Scapular inclination and inf""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,20,reference_content,"9. Kandemir U, Allaire R, Jolly T, Debski R, McMahon P. The relationship between the orientation of the glenoid and tears of the rotator cuff. J Bone Joint Surg Br 2006;88:1105-9. http://dx.doi.org/10","[607, 745, 1082, 820]",reference_item,0.85,"[""reference content label: 9. Kandemir U, Allaire R, Jolly T, Debski R, McMahon P. The ""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,21,reference_content,"10. Maurer A, Fucentese S, Pfirrmann C, Wirth S, Djahangiri A, Jost B, et al. Assessment of glenoid inclination on routine clinical radiographs and computed tomography examinations of the shoulder. J ","[600, 824, 1083, 901]",reference_item,0.85,"[""reference content label: 10. Maurer A, Fucentese S, Pfirrmann C, Wirth S, Djahangiri ""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,22,reference_content,"11. Minagawa H, Yamamoto N, Abe H. Prevalence of symptomatic and asymptomatic rotator cuff tears in the general population: from mass-screening in one village. J Orthop 2013;10:8-12. http://dx.doi.org","[600, 904, 1082, 980]",reference_item,0.85,"[""reference content label: 11. Minagawa H, Yamamoto N, Abe H. Prevalence of symptomatic""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,23,reference_content,"12. Moor BK, Bouaicha S, Rothenfluh DA, Sukthankar A, Gerber C. Is there an association between the individual anatomy of the scapula and the development of rotator cuff tears or osteoarthritis of the","[601, 984, 1083, 1098]",reference_item,0.85,"[""reference content label: 12. Moor BK, Bouaicha S, Rothenfluh DA, Sukthankar A, Gerber""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,24,reference_content,"13. Moor BK, Röthlisberger M, Müller DA, Zumstein MA, Bouaicha S, Ehlinger M, et al. Age, trauma and the critical shoulder angle accurately predict supraspinatus tendon tears. Orthop Traumatol Surg Re","[600, 1103, 1082, 1180]",reference_item,0.85,"[""reference content label: 13. Moor BK, R\u00f6thlisberger M, M\u00fcller DA, Zumstein MA, Bouaic""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,25,reference_content,"14. Moor BK, Wieser K, Slankamenac K, Gerber C, Bouaicha S. Relationship of individual scapular anatomy and degenerative rotator cuff tears. J Shoulder Elbow Surg 2014;23:536-41. http://dx.doi.org/10.","[600, 1183, 1083, 1259]",reference_item,0.85,"[""reference content label: 14. Moor BK, Wieser K, Slankamenac K, Gerber C, Bouaicha S. ""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,26,reference_content,"15. Nakagawa Y, Hyakuna K, Otani S, Hashitani M, Nakamura T. Epidemiologic study of glenohumeral osteoarthritis with plain radiography. J Shoulder Elbow Surg 1999;8:580-4.","[600, 1262, 1082, 1321]",reference_item,0.85,"[""reference content label: 15. Nakagawa Y, Hyakuna K, Otani S, Hashitani M, Nakamura T.""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,27,reference_content,16. Neer CS 2nd. Replacement arthroplasty for glenohumeral osteoarthritis. J Bone Joint Surg Am 1974;56-A:1-13.,"[600, 1323, 1082, 1360]",reference_item,0.85,"[""reference content label: 16. Neer CS 2nd. Replacement arthroplasty for glenohumeral o""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,28,reference_content,17. Neer CS 2nd. Degenerative lesions of the proximal humeral articular surface. Clin Orthop Relat Res 1961;20:116-25.,"[600, 1362, 1083, 1400]",reference_item,0.85,"[""reference content label: 17. Neer CS 2nd. Degenerative lesions of the proximal humera""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
5,29,reference_content,"18. Neer CS 2nd, Watson KC, Stanton FJ. Recent experience in total shoulder replacement. J Bone Joint Surg Am 1982;64-A:319-37.","[601, 1404, 1083, 1439]",reference_item,0.85,"[""reference content label: 18. Neer CS 2nd, Watson KC, Stanton FJ. Recent experience in""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
6,0,header,Correlation of glenoid inclination and CSA,"[105, 65, 454, 86]",noise,0.9,"[""header label""]",noise,0.9,,unknown_like,none,False,False
6,1,number,1953,"[1070, 66, 1113, 85]",noise,0.9,"[""page number label""]",noise,0.9,,unknown_like,short_fragment,False,False
6,2,reference_content,"19. Nowak DD, Gardner TR, Bigliani LU, Levine WN, Ahmad CS. Interobserver and intraobserver reliability of the Walch classification in primary glenohumeral arthritis. J Shoulder Elbow Surg 2010;19:180","[107, 112, 587, 189]",reference_item,0.85,"[""reference content label: 19. Nowak DD, Gardner TR, Bigliani LU, Levine WN, Ahmad CS. ""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
6,3,reference_content,"20. Terrier A, Reist A, Vogel A, Farron A. Effect of supraspinatus deficiency on humerus translation and glenohumeral contact force during abduction. Clin Biomech (Bristol, Avon) 2007;22:645-51. http:","[106, 192, 586, 268]",reference_item,0.85,"[""reference content label: 20. Terrier A, Reist A, Vogel A, Farron A. Effect of suprasp""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
6,4,reference_content,"21. Tétreault P, Krueger A, Zurakowski D, Gerber C. Glenoid version and rotator cuff tears. J Orthop Res 2004;22:202-7. http://dx.doi.org/10.1016/s0736-0266(03)00116-5","[106, 272, 587, 327]",reference_item,0.85,"[""reference content label: 21. T\u00e9treault P, Krueger A, Zurakowski D, Gerber C. Glenoid ""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
6,5,reference_content,"22. Walch G, Badet R, Boulahia A, Khoury A. Morphologic study of the glenoid in primary glenohumeral osteoarthritis. J Arthroplasty 1999;14:756-60.","[631, 112, 1112, 167]",reference_item,0.85,"[""reference content label: 22. Walch G, Badet R, Boulahia A, Khoury A. Morphologic stud""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
6,6,reference_content,"23. Walch G, Vezeridis PS, Boileau P, Deransart P, Chaoui J. Three-dimensional planning and use of patient-specific guides improve glenoid component position: an in vitro study. J Shoulder Elbow Surg ","[632, 172, 1113, 248]",reference_item,0.85,"[""reference content label: 23. Walch G, Vezeridis PS, Boileau P, Deransart P, Chaoui J.""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
6,7,reference_content,"24. Wong AS, Gallo L, Kuhn JE, Carpenter JE, Hughes RE. The effect of glenoid inclination on superior humeral head migration. J Shoulder Elbow Surg 2003;12:360-4. http://dx.doi.org/10.1016/s1058-2746(","[632, 252, 1114, 326]",reference_item,0.85,"[""reference content label: 24. Wong AS, Gallo L, Kuhn JE, Carpenter JE, Hughes RE. The ""]",reference_item,0.85,reference_zone,reference_like,reference_numeric_dot,True,True
1 page block_id raw_label content_preview bbox role role_confidence evidence seed_role seed_confidence zone style_family marker_type render_default index_default
2 1 0 header J Shoulder Elbow Surg (2015) 24, 1948-1953 [73, 64, 377, 84] noise 0.9 ["header label"] noise 0.9 frontmatter_main_zone support_like none False False
3 1 1 header_image [75, 102, 184, 200] unknown_structural 0.2 ["unrecognized label 'header_image'"] unknown_structural 0.2 frontmatter_main_zone support_like empty False True
4 1 2 header ELSEVIER [75, 205, 185, 227] noise 0.9 ["header label"] noise 0.9 frontmatter_main_zone support_like short_fragment False False
5 1 3 header JOURNAL OF SHOULDER AND E_LBOW SURGERY [915, 79, 1080, 176] noise 0.9 ["header label"] noise 0.9 frontmatter_main_zone support_like none False False
6 1 4 header www.elsevier.com/locate/ymse [879, 192, 1083, 211] noise 0.9 ["header label"] noise 0.9 frontmatter_main_zone support_like none False False
7 1 5 doc_title Correlation between glenoid inclination and critical shoulder angle: a radiographic and computed tomography study [71, 308, 810, 451] paper_title 0.8 ["page-1 zone title_zone: Correlation between glenoid inclination and critical shoulde"] paper_title 0.8 frontmatter_main_zone support_like none True True
8 1 6 image [947, 312, 1083, 367] media_asset 0.85 ["media label: image"] media_asset 0.85 frontmatter_main_zone support_like empty True True
9 1 7 text Matthew Daggett, DO, MBA*, Birgit Werner, MD, Philipp Collin, MD, Marc-Olivier Gauci, MD, Jean Chaoui, PhD, Gilles Walch, MD [71, 485, 894, 552] authors 0.8 ["page-1 zone author_zone: Matthew Daggett, DO, MBA*, Birgit Werner, MD, Philipp Collin"] authors 0.8 frontmatter_main_zone support_like none True True
10 1 8 text Centre Orthopédique Santé, Hôpital Privé Jean Mermoz, Lyon, France [72, 592, 659, 617] affiliation 0.8 ["page-1 zone affiliation_zone: Centre Orthop\u00e9dique Sant\u00e9, H\u00f4pital Priv\u00e9 Jean Mermoz, Lyon, "] affiliation 0.8 frontmatter_main_zone support_like none True True
11 1 9 abstract Background: Increased critical shoulder angles consist of both the acromial cover and glenoid inclination and have been found in patients with rotator cuff pathology. The purpose of this study was to [81, 662, 863, 1109] abstract_body 0.85 ["abstract label from Paddle OCR"] abstract_body 0.85 frontmatter_main_zone support_like none True True
12 1 10 text Keywords: Critical shoulder angle; glenoid inclination; rotator cuff tears [84, 1104, 625, 1129] frontmatter_noise 0.7 ["frontmatter noise text: Keywords: Critical shoulder angle; glenoid inclination; rota"] frontmatter_noise 0.7 frontmatter_main_zone support_like none False False
13 1 11 text Originally described by Moor et al, $ ^{12} $ the critical shoulder angle (CSA) has been used to analyze individual, quantitative shoulder anatomy. The CSA is measured by a line connecting the superio [595, 1215, 1086, 1384] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
14 1 12 footnote The Institutional Review Board of the Ethical Committee of Hôpital Privé Jean Mermoz and the Centre Orthopédique Santé approved this study. [71, 1284, 557, 1326] footnote 0.7 ["footnote label: The Institutional Review Board of the Ethical Committee of H"] footnote 0.7 body_zone body_like none True True
15 1 13 footnote *Reprint requests: Matthew Daggett, DO, MBA, Centre Orthopédique Santé, 24 avenue Paul Santé, F-69008 Lyon, France. [72, 1325, 559, 1364] footnote 0.7 ["footnote label: *Reprint requests: Matthew Daggett, DO, MBA, Centre Orthop\u00e9d"] footnote 0.7 body_zone body_like none True True
16 1 14 footnote E-mail address: matthewdaggett@gmail.com (M. Daggett). [94, 1365, 488, 1386] footnote 0.7 ["footnote label: E-mail address: matthewdaggett@gmail.com (M. Daggett)."] footnote 0.7 body_zone body_like none True True
17 1 15 footer 1058-2746/$ - see front matter © 2015 Journal of Shoulder and Elbow Surgery Board of Trustees. http://dx.doi.org/10.1016/j.jse.2015.07.013 [71, 1407, 724, 1448] noise 0.9 ["footer label"] noise 0.9 body_zone body_like none False False
18 2 0 header Correlation of glenoid inclination and CSA [104, 64, 455, 88] noise 0.9 ["header label"] noise 0.9 body_zone body_like none False False
19 2 1 number 1949 [1069, 65, 1114, 86] noise 0.9 ["page number label"] noise 0.9 body_zone body_like short_fragment False False
20 2 2 text and glenoid inclination. Previous investigations have found that the CSA can be a predictor of shoulder pathology, with increased CSA angles amongst individuals with rotator cuff tears and smaller ang [101, 110, 590, 253] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
21 2 3 text Glenoid inclination, a component of the CSA, has been separately identified as a risk factor for tears of the supraspinatus tendon. $ ^{7,24} $ A reproducible method to measure glenoid inclination, te [101, 254, 590, 539] affiliation 0.8 ["page-1 zone affiliation_zone: Glenoid inclination, a component of the CSA, has been separa"] affiliation 0.8 body_zone body_like none True True
22 2 4 text The first purpose of this study was to determine whether there is an association between the CSA and glenoid inclination of patients undergoing primary shoulder arthroplasty. We hypothesized that ther [101, 541, 590, 829] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
23 2 5 paragraph_title Materials and methods [103, 869, 346, 894] section_heading 0.9 ["explicit scholarly heading: Materials and methods"] section_heading 0.9 body_zone heading_like canonical_section_name True True
24 2 6 text The study included 50 shoulders in 48 individuals (27 women, 21 men) who were an average age of 72.9 years (range 61-84 years). The selection of 50 shoulders was made from patients within the clinical [101, 918, 589, 1074] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
25 2 7 paragraph_title Patient groups [103, 1097, 249, 1123] subsection_heading 0.6 ["unnumbered paragraph_title, inferred level subsection_heading: Patient groups"] subsection_heading 0.6 body_zone heading_like short_fragment True True
26 2 8 text Inclusion in the osteoarthritic group required confirmation by both surgeons of A1 classification, as defined by Walch et al., $ ^{22} $ centered head in both coronal and axial plans with minor glenoi [102, 1147, 590, 1366] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
27 2 9 text For each patient, multiple different imaging measurements were obtained from anteroposterior (AP) radiographs and standard, preoperative CT scans of the shoulder. First, the CSA was [101, 1368, 590, 1433] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
28 2 10 image [649, 113, 1095, 408] media_asset 0.85 ["media label: image"] media_asset 0.85 body_zone unknown_like empty True True
29 2 11 figure_title Figure 1 Radiograph shows A1 glenoid. [711, 424, 1031, 447] figure_caption 0.92 ["figure_title label: Figure 1 Radiograph shows A1 glenoid."] figure_caption 0.92 display_zone legend_like figure_number True True
30 2 12 image [649, 484, 1095, 1033] media_asset 0.85 ["media label: image"] media_asset 0.85 body_zone unknown_like empty True True
31 2 13 figure_title Figure 2 Radiograph shows E0 glenoid. [712, 1051, 1031, 1075] figure_caption 0.92 ["figure_title label: Figure 2 Radiograph shows E0 glenoid."] figure_caption 0.92 display_zone legend_like figure_number True True
32 2 14 text obtained via a standard AP shoulder X-ray image with the arm in neutral rotation. Described by Moor et al, $ ^{12} $ the CSA was measured by a line connecting the superior and inferior bony margins of [626, 1102, 1116, 1234] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
33 2 15 text The second measurement performed on AP radiographs of the shoulder was the β-angle, as described by Maurer et al., $ ^{10} $ which is the angle is formed by the intersection of a line through the floo [626, 1234, 1116, 1433] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
34 3 0 number 1950 [74, 65, 119, 86] noise 0.9 ["page number label"] noise 0.9 body_zone body_like short_fragment False False
35 3 1 header M. Daggett et al. [938, 64, 1081, 88] noise 0.9 ["header label"] noise 0.9 body_zone body_like short_fragment False False
36 3 2 image [92, 113, 537, 732] media_asset 0.85 ["media label: image"] media_asset 0.85 body_zone unknown_like empty True True
37 3 3 figure_title Figure 3 Illustration shows the critical shoulder angle (CSA) measurement. The CSA is defined as a line connecting the superior and inferior bony margins of the glenoid and an intersecting line drawn [71, 751, 557, 861] figure_caption 0.92 ["figure_title label: Figure 3 Illustration shows the critical shoulder angle (CSA"] figure_caption 0.92 display_zone legend_like figure_number True True
38 3 4 text The glenoid inclination was then measured on the CT scan using the validated Glenosys 3-D software. $ ^{23} $ After segmentation and reformatting, the software program determines the planes of the sca [70, 919, 560, 1117] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
39 3 5 paragraph_title Statistics [71, 1141, 167, 1165] sub_subsection_heading 0.6 ["unnumbered paragraph_title, inferred level sub_subsection_heading: Statistics"] sub_subsection_heading 0.6 body_zone heading_like short_fragment True True
40 3 6 text Regression analysis was used to analyze the relationship between CSA and glenoid inclination. The Mann-Whitney U test was used to compare the CSA on AP radiographs between the osteoarthritis (A1 group [70, 1192, 560, 1412] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
41 3 7 image [618, 112, 1062, 559] media_asset 0.85 ["media label: image"] media_asset 0.85 body_zone unknown_like empty True True
42 3 8 figure_title Figure 4 Measuring glenoid inclination using the $ \beta $-angle on an X-ray image: The $ \beta $-angle is measured from the angle formed by a line along the floor of the supraspinatus fossa and the [596, 575, 1084, 663] figure_caption 0.92 ["figure_title label: Figure 4 Measuring glenoid inclination using the $ \\beta $-"] figure_caption 0.92 display_zone legend_like figure_number True True
43 3 9 paragraph_title Results [597, 688, 680, 713] section_heading 0.9 ["explicit scholarly heading: Results"] section_heading 0.9 body_zone heading_like canonical_section_name True True
44 3 10 text A statistically significant correlation was found between the glenoid inclination and the CSA in both groups. Glenoid inclination demonstrated a linear relationship to the CSA ( $ R^{2} = 0.7426 $, P [594, 742, 1084, 956] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
45 3 11 text There were distinct differences in the CSA between the 2 groups. Patients in the A1 group had a mean CSA of 27.7° (standard deviation [SD], 3.4°; range, 17°-33°), whereas patients in the E0 group had [594, 958, 1084, 1148] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
46 3 12 text There were also distinct differences of glenoid inclination between the 2 groups. As measured in the 3-D software, the A1 group had a mean glenoid inclination of 4.7° (SD, 5.6°, range, −9° to 13°), an [594, 1150, 1084, 1414] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
47 4 0 header Correlation of glenoid inclination and CSA [105, 64, 455, 87] noise 0.9 ["header label"] noise 0.9 body_zone body_like none False False
48 4 1 number 1951 [1069, 65, 1113, 85] noise 0.9 ["page number label"] noise 0.9 body_zone body_like short_fragment False False
49 4 2 image [148, 113, 1071, 571] media_asset 0.85 ["media label: image"] media_asset 0.85 body_zone unknown_like empty True True
50 4 3 figure_title Figure 5 Screenshot shows output provided by the Glenosys (Imascap, Brest, France) 3-dimensional software, which provides anatomic details, including glenoid inclination. [102, 591, 1113, 637] figure_caption 0.92 ["figure_title label: Figure 5 Screenshot shows output provided by the Glenosys (I"] figure_caption 0.92 display_zone legend_like figure_number True True
51 4 4 text and the E0 group had a mean glenoid inclination of $ 13.6^{\circ} $ (SD, $ 4.6^{\circ} $; range, $ 7^{\circ}-24^{\circ} $). [101, 684, 588, 732] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
52 4 5 paragraph_title Discussion [102, 772, 219, 798] section_heading 0.9 ["explicit scholarly heading: Discussion"] section_heading 0.9 body_zone heading_like canonical_section_name True True
53 4 6 text Previous studies have investigated the components of the CSA, but to our knowledge, this is the first study to find a correlation between the glenoid inclination and the CSA. The CSA demonstrated a li [100, 827, 589, 1018] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
54 4 7 text The effect of the CSA has been previously examined. Moor et al $ ^{12} $ found an increased CSA (>35°) in patients with rotator cuff tears and decreased CSA (<30°) in patients with osteoarthritis. An [101, 1018, 590, 1377] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
55 4 8 text The massive rotator cuff tear (E0) group had a significant increase in superior glenoid inclination, with a mean glenoid inclination of $ 13.6^{\circ} $, whereas the osteoarthritic (A1) [100, 1378, 590, 1450] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
56 4 9 chart [645, 686, 1096, 998] media_asset 0.85 ["media label: chart"] media_asset 0.85 body_zone unknown_like empty True True
57 4 10 figure_title Figure 6 The critical shoulder angle (CSA) and glenoid inclination, as measured by the B-angle on Glenosys (GI-GL_beta), demonstrated a near linear relationship. MRCT, massive rotator cuff tear; OA, o [626, 1011, 1115, 1100] figure_caption 0.92 ["figure_title label: Figure 6 The critical shoulder angle (CSA) and glenoid incli"] figure_caption 0.92 display_zone legend_like figure_number True True
58 4 11 text group demonstrated glenoid inclination measurements similar to published anatomic data from Churchill et al $ ^{2} $ of $ 4^{\circ} $ to $ 4.5^{\circ} $. Although rotator cuff disease is likely seco [626, 1136, 1116, 1303] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
59 4 12 text Previous studies have demonstrated a potential link between glenoid inclination and rotator cuff tears as well as superior migration of the humeral head $ ^{4,7,8,20,21,24} $; however, others have fai [626, 1304, 1116, 1449] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 body_zone body_like none True True
60 5 0 number 1952 [74, 65, 118, 86] noise 0.9 ["page number label"] noise 0.9 body_zone body_like short_fragment False False
61 5 1 header M. Daggett et al. [938, 65, 1081, 88] noise 0.9 ["header label"] noise 0.9 body_zone body_like short_fragment False False
62 5 2 text other. $ ^{6,16-18} $ The significant difference in glenoid inclination found between the E0 group and A1 group may partially explain this phenomenon. [70, 109, 557, 181] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 tail_nonref_hold_zone unknown_like none True True
63 5 3 text Increased glenoid inclination found in the E0 group could support the theory that tears of the rotator cuff are caused by tendon overload rather than by acromial impingement. Gerber et al $ ^{5} $ dem [70, 183, 559, 467] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 tail_nonref_hold_zone unknown_like none True True
64 5 4 text Our study has certain limitations. The potential for misclassification of glenoid subtype exists; however, 2 physicians independently validated the classification on the X-ray images. The interobserve [70, 469, 558, 636] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 tail_nonref_hold_zone unknown_like none True True
65 5 5 text Although measurements using AP radiographs showed a less severe difference in glenoid inclination between the 2 groups, this was likely secondary to inadequate visualization of the supraspinatus fossa [71, 637, 557, 875] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 tail_nonref_hold_zone unknown_like none True True
66 5 6 text Our findings raise many questions about the role vertical orientation of the glenoid has in the causation or evolution of many shoulder disorders. Further investigation into the role superior glenoid [70, 876, 557, 1018] body_paragraph 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 tail_nonref_hold_zone unknown_like none True True
67 5 7 paragraph_title Conclusion [85, 1069, 200, 1094] structured_insert 0.9 ["explicit scholarly heading: Conclusion"] section_heading 0.9 tail_nonref_hold_zone heading_like canonical_section_name False False
68 5 8 text Glenoid inclination is linearly correlated with the critical shoulder angle and is significantly increased in patients with massive rotator cuff tears. [82, 1117, 546, 1190] structured_insert 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 tail_nonref_hold_zone unknown_like none False False
69 5 9 paragraph_title Disclaimer [85, 1261, 198, 1287] structured_insert 0.6 ["unnumbered paragraph_title, inferred level sub_subsection_heading: Disclaimer"] sub_subsection_heading 0.6 tail_nonref_hold_zone heading_like short_fragment False False
70 5 10 text The authors, their immediate families, and any research foundations with which they are affiliated have not received any financial payments or other benefits from any commercial entity related to the [82, 1311, 546, 1431] structured_insert 0.6 ["default body_paragraph for text label"] body_paragraph 0.6 tail_nonref_hold_zone unknown_like none False False
71 5 11 paragraph_title References [599, 110, 717, 135] reference_heading 0.9 ["references heading: References"] reference_heading 0.9 reference_zone heading_like short_fragment True True
72 5 12 reference_content 1. Bishop JL, Kline SK, Aalderink KJ, Zauel R, Bey MJ. Glenoid inclination: in vivo measures in rotator cuff tear patients and associations with superior glenohumeral joint translation. J Shoulder Elb [608, 166, 1083, 262] reference_item 0.85 ["reference content label: 1. Bishop JL, Kline SK, Aalderink KJ, Zauel R, Bey MJ. Gleno"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
73 5 13 reference_content 2. Churchill RS, Brems J, Kotschi H. Glenoid size, inclination and version: an anatomic study. J Shoulder Elbow Surg 2001;10:327-32. [607, 266, 1082, 323] reference_item 0.85 ["reference content label: 2. Churchill RS, Brems J, Kotschi H. Glenoid size, inclinati"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
74 5 14 reference_content 3. Favard L, Lautmann S, Sirveaux F, Oudet D, Kerjean Y, Huguet D. Hemiarthroplasty versus reverse arthroplasty in the treatment of osteoarthritis with massive rotator cuff tear. In: Walch G, Mole D, [608, 325, 1083, 422] reference_item 0.85 ["reference content label: 3. Favard L, Lautmann S, Sirveaux F, Oudet D, Kerjean Y, Hug"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
75 5 15 reference_content 4. Flieg N, Gatti C, Doro L, Langenderfer J, Carpenter J, Hughes R. A stochastic analysis of glenoid inclination angle and superior migration of the humeral head. Clin Biomech (Bristol, Avon) 2008;23: [608, 425, 1082, 503] reference_item 0.85 ["reference content label: 4. Flieg N, Gatti C, Doro L, Langenderfer J, Carpenter J, Hu"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
76 5 16 reference_content 5. Gerber C, Snedeker JG, Baumgartner D, Viehöfer AF. Supraspinatus tendon load during abduction is dependent on the size of the critical shoulder angle: a biomechanical analysis. J Orthop Res 2014;32 [608, 506, 1082, 583] reference_item 0.85 ["reference content label: 5. Gerber C, Snedeker JG, Baumgartner D, Vieh\u00f6fer AF. Supras"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
77 5 17 reference_content 6. Hamada K, Fukuda H, Mikasa M, Kobayashi Y. Roentgenographic findings in massive rotator cuff tears: a long-term observation. Clin Orthop Relat Res 1990;254:92-6. [608, 585, 1082, 642] reference_item 0.85 ["reference content label: 6. Hamada K, Fukuda H, Mikasa M, Kobayashi Y. Roentgenograph"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
78 5 18 reference_content 7. Hughes RE, Bryant CR, Hall JM, Wening J, Huston LJ, Kuhn JE, et al. Glenoid inclination is associated with full-thickness rotator cuff tears. Clin Orthop Relat Res 2003;407:86-91. [608, 645, 1081, 702] reference_item 0.85 ["reference content label: 7. Hughes RE, Bryant CR, Hall JM, Wening J, Huston LJ, Kuhn "] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
79 5 19 reference_content 8. Itoi E, Motzkin N, Morrey B. Scapular inclination and inferior stability of the shoulder. J Shoulder Elbow Surg 1992;1:131-9. [607, 704, 1079, 742] reference_item 0.85 ["reference content label: 8. Itoi E, Motzkin N, Morrey B. Scapular inclination and inf"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
80 5 20 reference_content 9. Kandemir U, Allaire R, Jolly T, Debski R, McMahon P. The relationship between the orientation of the glenoid and tears of the rotator cuff. J Bone Joint Surg Br 2006;88:1105-9. http://dx.doi.org/10 [607, 745, 1082, 820] reference_item 0.85 ["reference content label: 9. Kandemir U, Allaire R, Jolly T, Debski R, McMahon P. The "] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
81 5 21 reference_content 10. Maurer A, Fucentese S, Pfirrmann C, Wirth S, Djahangiri A, Jost B, et al. Assessment of glenoid inclination on routine clinical radiographs and computed tomography examinations of the shoulder. J [600, 824, 1083, 901] reference_item 0.85 ["reference content label: 10. Maurer A, Fucentese S, Pfirrmann C, Wirth S, Djahangiri "] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
82 5 22 reference_content 11. Minagawa H, Yamamoto N, Abe H. Prevalence of symptomatic and asymptomatic rotator cuff tears in the general population: from mass-screening in one village. J Orthop 2013;10:8-12. http://dx.doi.org [600, 904, 1082, 980] reference_item 0.85 ["reference content label: 11. Minagawa H, Yamamoto N, Abe H. Prevalence of symptomatic"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
83 5 23 reference_content 12. Moor BK, Bouaicha S, Rothenfluh DA, Sukthankar A, Gerber C. Is there an association between the individual anatomy of the scapula and the development of rotator cuff tears or osteoarthritis of the [601, 984, 1083, 1098] reference_item 0.85 ["reference content label: 12. Moor BK, Bouaicha S, Rothenfluh DA, Sukthankar A, Gerber"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
84 5 24 reference_content 13. Moor BK, Röthlisberger M, Müller DA, Zumstein MA, Bouaicha S, Ehlinger M, et al. Age, trauma and the critical shoulder angle accurately predict supraspinatus tendon tears. Orthop Traumatol Surg Re [600, 1103, 1082, 1180] reference_item 0.85 ["reference content label: 13. Moor BK, R\u00f6thlisberger M, M\u00fcller DA, Zumstein MA, Bouaic"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
85 5 25 reference_content 14. Moor BK, Wieser K, Slankamenac K, Gerber C, Bouaicha S. Relationship of individual scapular anatomy and degenerative rotator cuff tears. J Shoulder Elbow Surg 2014;23:536-41. http://dx.doi.org/10. [600, 1183, 1083, 1259] reference_item 0.85 ["reference content label: 14. Moor BK, Wieser K, Slankamenac K, Gerber C, Bouaicha S. "] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
86 5 26 reference_content 15. Nakagawa Y, Hyakuna K, Otani S, Hashitani M, Nakamura T. Epidemiologic study of glenohumeral osteoarthritis with plain radiography. J Shoulder Elbow Surg 1999;8:580-4. [600, 1262, 1082, 1321] reference_item 0.85 ["reference content label: 15. Nakagawa Y, Hyakuna K, Otani S, Hashitani M, Nakamura T."] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
87 5 27 reference_content 16. Neer CS 2nd. Replacement arthroplasty for glenohumeral osteoarthritis. J Bone Joint Surg Am 1974;56-A:1-13. [600, 1323, 1082, 1360] reference_item 0.85 ["reference content label: 16. Neer CS 2nd. Replacement arthroplasty for glenohumeral o"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
88 5 28 reference_content 17. Neer CS 2nd. Degenerative lesions of the proximal humeral articular surface. Clin Orthop Relat Res 1961;20:116-25. [600, 1362, 1083, 1400] reference_item 0.85 ["reference content label: 17. Neer CS 2nd. Degenerative lesions of the proximal humera"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
89 5 29 reference_content 18. Neer CS 2nd, Watson KC, Stanton FJ. Recent experience in total shoulder replacement. J Bone Joint Surg Am 1982;64-A:319-37. [601, 1404, 1083, 1439] reference_item 0.85 ["reference content label: 18. Neer CS 2nd, Watson KC, Stanton FJ. Recent experience in"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
90 6 0 header Correlation of glenoid inclination and CSA [105, 65, 454, 86] noise 0.9 ["header label"] noise 0.9 unknown_like none False False
91 6 1 number 1953 [1070, 66, 1113, 85] noise 0.9 ["page number label"] noise 0.9 unknown_like short_fragment False False
92 6 2 reference_content 19. Nowak DD, Gardner TR, Bigliani LU, Levine WN, Ahmad CS. Interobserver and intraobserver reliability of the Walch classification in primary glenohumeral arthritis. J Shoulder Elbow Surg 2010;19:180 [107, 112, 587, 189] reference_item 0.85 ["reference content label: 19. Nowak DD, Gardner TR, Bigliani LU, Levine WN, Ahmad CS. "] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
93 6 3 reference_content 20. Terrier A, Reist A, Vogel A, Farron A. Effect of supraspinatus deficiency on humerus translation and glenohumeral contact force during abduction. Clin Biomech (Bristol, Avon) 2007;22:645-51. http: [106, 192, 586, 268] reference_item 0.85 ["reference content label: 20. Terrier A, Reist A, Vogel A, Farron A. Effect of suprasp"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
94 6 4 reference_content 21. Tétreault P, Krueger A, Zurakowski D, Gerber C. Glenoid version and rotator cuff tears. J Orthop Res 2004;22:202-7. http://dx.doi.org/10.1016/s0736-0266(03)00116-5 [106, 272, 587, 327] reference_item 0.85 ["reference content label: 21. T\u00e9treault P, Krueger A, Zurakowski D, Gerber C. Glenoid "] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
95 6 5 reference_content 22. Walch G, Badet R, Boulahia A, Khoury A. Morphologic study of the glenoid in primary glenohumeral osteoarthritis. J Arthroplasty 1999;14:756-60. [631, 112, 1112, 167] reference_item 0.85 ["reference content label: 22. Walch G, Badet R, Boulahia A, Khoury A. Morphologic stud"] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
96 6 6 reference_content 23. Walch G, Vezeridis PS, Boileau P, Deransart P, Chaoui J. Three-dimensional planning and use of patient-specific guides improve glenoid component position: an in vitro study. J Shoulder Elbow Surg [632, 172, 1113, 248] reference_item 0.85 ["reference content label: 23. Walch G, Vezeridis PS, Boileau P, Deransart P, Chaoui J."] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True
97 6 7 reference_content 24. Wong AS, Gallo L, Kuhn JE, Carpenter JE, Hughes RE. The effect of glenoid inclination on superior humeral head migration. J Shoulder Elbow Surg 2003;12:360-4. http://dx.doi.org/10.1016/s1058-2746( [632, 252, 1114, 326] reference_item 0.85 ["reference content label: 24. Wong AS, Gallo L, Kuhn JE, Carpenter JE, Hughes RE. The "] reference_item 0.85 reference_zone reference_like reference_numeric_dot True True

View file

@ -0,0 +1,12 @@
{
"layout": {
"mixed_tail_pages": [5],
"min_reference_items_on_mixed_tail_page": 3
},
"document": {
"notes": [
"6-page mixed_tail fixture. Page 5: 5 body_paragraph + 18 reference_item blocks.",
"Expands coverage for mixed_tail layout classification."
]
}
}

View file

@ -5459,4 +5459,100 @@ def test_normalize_document_structure_preserves_heading_page_reference_items() -
assert b.get("role") == "reference_item", \
f"Block {bid} demoted to {b.get('role')}"
assert b.get("zone") == "reference_zone", \
f"Block {bid} zone={b.get('zone')} not reference_zone"
f"Block {bid} zone={b.get('zone')} not reference_zone"
def test_veto_tail_spread_does_not_invert_range() -> None:
"""spread_start must not exceed spread_end when a single-page tail spread
has strong body continuation (3 body_paragraph blocks). Regression for
the bug where _veto_tail_spread_body_continuation pushed spread_start
past spread_end, creating an inverted range."""
from paperforge.worker.ocr_document import TailBoundary, _veto_tail_spread_body_continuation
blocks = [
{"page": 5, "role": "body_paragraph", "seed_role": "body_paragraph",
"text": "Page 5 body continuation para 1. " * 10,
"bbox": [100, 100, 500, 130], "page_width": 1200, "page_height": 1600},
{"page": 5, "role": "body_paragraph", "seed_role": "body_paragraph",
"text": "Page 5 body continuation para 2. " * 10,
"bbox": [100, 140, 500, 170], "page_width": 1200, "page_height": 1600},
{"page": 5, "role": "body_paragraph", "seed_role": "body_paragraph",
"text": "Page 5 body continuation para 3. " * 10,
"bbox": [100, 180, 500, 210], "page_width": 1200, "page_height": 1600},
{"page": 6, "role": "reference_item", "seed_role": "reference_item",
"text": "[1] A reference. Journal. 2024.",
"bbox": [100, 100, 500, 120], "page_width": 1200, "page_height": 1600},
]
boundary = TailBoundary(
body_end_page=4,
backmatter_start=5,
references_start=6,
spread_start=5,
spread_end=5,
is_clean_separated=True,
reason="test: single-page tail spread before veto",
)
result = _veto_tail_spread_body_continuation(boundary, blocks)
assert result.spread_start <= result.spread_end, \
f"spread_start ({result.spread_start}) > spread_end ({result.spread_end})"
assert result.spread_start == 5, \
f"spread_start should remain 5 (capped by spread_end=5), got {result.spread_start}"
def test_frontmatter_support_below_body_start_rescued_by_width() -> None:
"""A frontmatter_support block on page 1 below body start that is narrower
than the body anchor width by 100px should be rescued into
frontmatter_main_zone instead of falling to body_zone. Regression for
CAQNW9Q2 correspondence block (161px width vs ~393px body width)."""
from paperforge.worker.ocr_document import infer_zones
blocks = [
# Paper title — triggers seen_title_or_author
{"block_id": "title", "page": 1,
"role": "paper_title", "seed_role": "paper_title",
"text": "A Study of Important Things",
"bbox": [100, 50, 500, 90],
"marker_signature": {"type": "none"},
"span_signature": {"font_size_median": 16.0},
"layout_signature": {"width": 400, "x_center": 300},
"page_width": 600, "page_height": 900},
# Body paragraph with ≥20 words — triggers body_started
{"block_id": "body1", "page": 1,
"role": "body_paragraph", "seed_role": "body_paragraph",
"text": "This is a body paragraph that has well more than twenty words so that it triggers the first page body start detection function correctly for the test.",
"bbox": [100, 120, 500, 160],
"marker_signature": {"type": "none"},
"span_signature": {"font_size_median": 10.0},
"layout_signature": {"width": 400, "x_center": 300},
"page_width": 600, "page_height": 900},
# Narrow frontmatter_support block below body — should be rescued
{"block_id": "corresp", "page": 1,
"role": "frontmatter_support", "seed_role": "frontmatter_support",
"text": "Correspondence to: Dr. Important Person",
"bbox": [100, 180, 260, 200], # 160px wide, well under 400-100=300
"marker_signature": {"type": "none"},
"span_signature": {"font_size_median": 9.0},
"layout_signature": {"width": 160, "x_center": 180},
"page_width": 600, "page_height": 900},
]
anchors = {
"body_family_anchor": {
"status": "ACCEPT",
"family_name": "body_family",
"sample_pages": [2],
"width_bucket": 400,
},
}
zones = infer_zones(blocks, anchors)
fm_main = zones["frontmatter_main_zone"]
body = zones["body_zone"]
assert "corresp" in fm_main["block_ids"], \
"narrow frontmatter_support block should be in frontmatter_main_zone"
assert "corresp" not in body["block_ids"], \
"narrow frontmatter_support block should NOT be in body_zone"

View file

@ -0,0 +1,17 @@
from __future__ import annotations
from paperforge.worker.ocr_figures import build_figure_inventory_legacy, build_figure_inventory_vnext
from scripts.dev.compare_figure_inventory_legacy_vs_vnext import compare_inventories
def test_compare_inventories_reports_counts_for_same_page_case():
blocks = [
{"block_id": "c1", "page": 1, "role": "figure_caption", "text": "Figure 1. Caption", "bbox": [0, 100, 200, 150]},
{"block_id": "a1", "page": 1, "role": "figure_asset", "bbox": [0, 0, 200, 90], "raw_label": "image"},
]
legacy = build_figure_inventory_legacy(blocks, 1200)
vnext = build_figure_inventory_vnext(blocks, 1200)
diff = compare_inventories(legacy, vnext)
assert diff["vnext_matched_count"] >= 1
assert "vnext_consumed_block_ids" in diff

View file

@ -0,0 +1,27 @@
from __future__ import annotations
from paperforge.worker.ocr_figure_vnext_corpus import FigureCandidateIndex, FigureCorpus
def test_corpus_keeps_raw_facts_and_no_candidate_groups():
blocks = [
{"block_id": "1", "page": 1, "role": "figure_caption", "text": "Figure 1. Caption"},
{"block_id": "2", "page": 1, "role": "figure_asset", "bbox": [0, 0, 10, 10]},
]
corpus = FigureCorpus.from_blocks(blocks, page_width=1200)
assert [b["block_id"] for b in corpus.raw_legends] == ["1"]
assert [b["block_id"] for b in corpus.raw_assets] == ["2"]
assert corpus.page_width == 1200
def test_candidate_index_holds_derived_hypotheses_not_raw_facts():
blocks = [
{"block_id": "1", "page": 1, "role": "figure_caption", "text": "Figure 1. Caption"},
{"block_id": "2", "page": 1, "role": "figure_asset", "bbox": [0, 0, 10, 10]},
]
corpus = FigureCorpus.from_blocks(blocks, page_width=1200)
index = FigureCandidateIndex.from_corpus(corpus)
assert len(index.formal_legends) == 1
assert hasattr(index, "candidate_groups")
assert corpus.raw_legends[0]["block_id"] == "1"
assert index.bundle_source_legend_ids == set()

View file

@ -0,0 +1,160 @@
from __future__ import annotations
from paperforge.worker.ocr_figure_vnext_corpus import FigureCandidateIndex, FigureCorpus
from paperforge.worker.ocr_figure_vnext_passes import PrimarySamePagePass
from paperforge.worker.ocr_figure_vnext_state import FigurePipelineState, OwnershipLedger
def test_primary_same_page_pass_matches_single_safe_group():
blocks = [
{"block_id": "c1", "page": 1, "role": "figure_caption", "text": "Figure 1. Caption", "bbox": [0, 100, 200, 150]},
{"block_id": "a1", "page": 1, "role": "figure_asset", "bbox": [0, 0, 200, 90], "raw_label": "image"},
]
corpus = FigureCorpus.from_blocks(blocks, page_width=1200)
index = FigureCandidateIndex.from_corpus(corpus)
state = FigurePipelineState(corpus=corpus, candidate_index=index, ledger=OwnershipLedger())
report = PrimarySamePagePass().run(state)
assert len(report.accepted) == 1
assert report.accepted[0].claim_type == "match"
assert len(state.matches) == 1
def test_primary_same_page_pass_prefers_higher_score_when_two_legends_compete_for_one_asset(monkeypatch):
blocks = [
{"block_id": "c1", "page": 1, "role": "figure_caption", "text": "Figure 1. Caption", "bbox": [0, 100, 200, 150]},
{"block_id": "c2", "page": 1, "role": "figure_caption", "text": "Figure 2. Caption", "bbox": [0, 160, 200, 210]},
{"block_id": "a1", "page": 1, "role": "figure_asset", "bbox": [0, 0, 200, 90], "raw_label": "image"},
]
corpus = FigureCorpus.from_blocks(blocks, page_width=1200)
index = FigureCandidateIndex.from_corpus(corpus)
if not index.candidate_groups:
index.candidate_groups = [{"group_id": "g1", "page": 1, "asset_block_ids": ["a1"], "media_blocks": [{"block_id": "a1"}], "group_type": "single_asset", "cluster_bbox": [0, 0, 200, 90]}]
scores = [{"score": 0.4, "decision": "matched", "evidence": ["low"]}, {"score": 0.9, "decision": "matched", "evidence": ["high"]}]
def fake_score(*args, **kwargs):
return scores.pop(0)
monkeypatch.setattr("paperforge.worker.ocr_figures._score_legend_to_group", fake_score)
monkeypatch.setattr("paperforge.worker.ocr_figures.score_figure_caption", lambda *a, **k: {"score": 0.9})
state = FigurePipelineState(corpus=corpus, candidate_index=index, ledger=OwnershipLedger())
report = PrimarySamePagePass().run(state)
assert len(report.accepted) == 1
assert report.accepted[0].figure_no == 2
def test_cross_page_reservation_pass_reserves_forward_group_when_same_page_primary_misses():
"""Legend on page 1, group on page 2; same-page pass misses, reservation pass reserves."""
blocks = [
{"block_id": "l1", "page": 1, "role": "figure_caption", "text": "Figure 1. Caption", "bbox": [0, 100, 200, 150]},
{"block_id": "a1", "page": 2, "role": "figure_asset", "bbox": [0, 0, 200, 90], "raw_label": "image"},
]
corpus = FigureCorpus.from_blocks(blocks, page_width=1200)
index = FigureCandidateIndex.from_corpus(corpus)
if not index.candidate_groups:
index.candidate_groups = [{"group_id": "g2", "page": 2, "asset_block_ids": ["a1"], "media_blocks": [{"block_id": "a1"}], "group_type": "single_asset", "cluster_bbox": [0, 0, 200, 90]}]
state = FigurePipelineState(corpus=corpus, candidate_index=index, ledger=OwnershipLedger())
# Same-page pass should NOT match — group is on a different page
primary_report = PrimarySamePagePass().run(state)
assert len(primary_report.accepted) == 0
# Cross-page reservation pass should reserve the forward group
report = CrossPageReservationPass().run(state)
assert len(report.accepted) == 1
assert report.accepted[0].claim_type == "reserve"
assert len(state.reservations) == 1
# The reserved group should be marked as reserved in the ledger
group_ref = ResourceRef(kind="group", page=2, block_id=None, group_id="g2")
assert state.ledger.can_claim_group(group_ref) is False
def test_cross_page_reservation_pass_does_not_reserve_group_already_claimed_same_page(monkeypatch):
"""Legend already matched by same-page pass should be skipped by reservation pass."""
blocks = [
{"block_id": "l1", "page": 1, "role": "figure_caption", "text": "Figure 1. Caption", "bbox": [0, 100, 200, 150]},
{"block_id": "a1", "page": 1, "role": "figure_asset", "bbox": [0, 0, 200, 90], "raw_label": "image"},
]
corpus = FigureCorpus.from_blocks(blocks, page_width=1200)
index = FigureCandidateIndex.from_corpus(corpus)
if not index.candidate_groups:
index.candidate_groups = [{"group_id": "g1", "page": 1, "asset_block_ids": ["a1"], "media_blocks": [{"block_id": "a1"}], "group_type": "single_asset", "cluster_bbox": [0, 0, 200, 90]}]
monkeypatch.setattr("paperforge.worker.ocr_figures._score_legend_to_group", lambda *a, **k: {"score": 0.9, "decision": "matched", "evidence": [""]})
monkeypatch.setattr("paperforge.worker.ocr_figures.score_figure_caption", lambda *a, **k: {"score": 0.9})
state = FigurePipelineState(corpus=corpus, candidate_index=index, ledger=OwnershipLedger())
# Same-page pass should match
primary_report = PrimarySamePagePass().run(state)
assert len(primary_report.accepted) == 1
assert len(state.matches) == 1
# Cross-page reservation pass should skip already-matched legend
report = CrossPageReservationPass().run(state)
assert len(report.accepted) == 0
assert len(state.reservations) == 0
def test_cross_page_reservation_pass_reserves_forward_group_when_same_page_primary_misses():
"""Legend on page 1, group on page 2; same-page pass misses, reservation pass reserves."""
blocks = [
{"block_id": "l1", "page": 1, "role": "figure_caption", "text": "Figure 1. Caption", "bbox": [0, 100, 200, 150]},
{"block_id": "a1", "page": 2, "role": "figure_asset", "bbox": [0, 0, 200, 90], "raw_label": "image"},
]
corpus = FigureCorpus.from_blocks(blocks, page_width=1200)
index = FigureCandidateIndex.from_corpus(corpus)
if not index.candidate_groups:
index.candidate_groups = [{"group_id": "g2", "page": 2, "asset_block_ids": ["a1"], "media_blocks": [{"block_id": "a1"}], "group_type": "single_asset", "cluster_bbox": [0, 0, 200, 90]}]
state = FigurePipelineState(corpus=corpus, candidate_index=index, ledger=OwnershipLedger())
# Same-page pass should NOT match — group is on a different page
primary_report = PrimarySamePagePass().run(state)
assert len(primary_report.accepted) == 0
# Cross-page reservation pass should reserve the forward group
report = CrossPageReservationPass().run(state)
assert len(report.accepted) == 1
assert report.accepted[0].claim_type == "reserve"
assert len(state.reservations) == 1
# The reserved group should be marked as reserved in the ledger
group_ref = ResourceRef(kind="group", page=2, block_id=None, group_id="g2")
assert state.ledger.can_claim_group(group_ref) is False
def test_cross_page_reservation_pass_does_not_reserve_group_already_claimed_same_page(monkeypatch):
"""Legend already matched by same-page pass should be skipped by reservation pass."""
blocks = [
{"block_id": "l1", "page": 1, "role": "figure_caption", "text": "Figure 1. Caption", "bbox": [0, 100, 200, 150]},
{"block_id": "a1", "page": 1, "role": "figure_asset", "bbox": [0, 0, 200, 90], "raw_label": "image"},
]
corpus = FigureCorpus.from_blocks(blocks, page_width=1200)
index = FigureCandidateIndex.from_corpus(corpus)
if not index.candidate_groups:
index.candidate_groups = [{"group_id": "g1", "page": 1, "asset_block_ids": ["a1"], "media_blocks": [{"block_id": "a1"}], "group_type": "single_asset", "cluster_bbox": [0, 0, 200, 90]}]
monkeypatch.setattr("paperforge.worker.ocr_figures._score_legend_to_group", lambda *a, **k: {"score": 0.9, "decision": "matched", "evidence": [""]})
monkeypatch.setattr("paperforge.worker.ocr_figures.score_figure_caption", lambda *a, **k: {"score": 0.9})
state = FigurePipelineState(corpus=corpus, candidate_index=index, ledger=OwnershipLedger())
# Same-page pass should match
primary_report = PrimarySamePagePass().run(state)
assert len(primary_report.accepted) == 1
assert len(state.matches) == 1
# Cross-page reservation pass should skip already-matched legend
report = CrossPageReservationPass().run(state)
assert len(report.accepted) == 0
assert len(state.reservations) == 0

View file

@ -0,0 +1,40 @@
from __future__ import annotations
from paperforge.worker.ocr_figure_vnext_types import ClaimProposal, ResourceRef
from paperforge.worker.ocr_figure_vnext_state import FigurePipelineState, OwnershipLedger
def test_ledger_rejects_double_ownership_for_same_asset_and_records_conflict():
ledger = OwnershipLedger()
asset = ResourceRef(kind="asset", page=3, block_id="42")
legend_a = ResourceRef(kind="legend", page=3, block_id="77", figure_no=1)
legend_b = ResourceRef(kind="legend", page=3, block_id="78", figure_no=2)
ledger.claim_assets([asset], owner=legend_a, reason="same_page_primary")
conflict = ledger.try_claim_assets([asset], owner=legend_b, reason="conflicting_match")
assert conflict is not None
assert conflict.resource == asset
assert conflict.current_owner == legend_a
assert any(entry["action"] == "conflict" for entry in ledger.snapshot())
def test_pipeline_state_accept_match_records_diagnostic():
state = FigurePipelineState(corpus=None, candidate_index=None, ledger=OwnershipLedger())
proposal = ClaimProposal(
pass_name="primary_same_page",
figure_no=1,
claim_type="match",
legends=[ResourceRef(kind="legend", page=1, block_id="c1", figure_no=1)],
assets=[ResourceRef(kind="asset", page=1, block_id="a1")],
groups=[],
confidence=0.9,
evidence_rank=1,
reason="same_page_primary",
diagnostics={"evidence": ["test"]},
)
state.accept_match(proposal, {"figure_id": "Figure 1"})
assert state.matches == [{"figure_id": "Figure 1"}]
assert state.diagnostics[-1]["event"] == "match_accepted"

View file

@ -0,0 +1,19 @@
from __future__ import annotations
import pytest
from paperforge.worker.ocr_figure_vnext_types import ResourceRef
def test_resource_ref_rejects_page_agnostic_asset():
with pytest.raises(ValueError):
ResourceRef(kind="asset", page=None, block_id="42")
def test_resource_ref_normalizes_block_id_type():
assert ResourceRef(kind="asset", page=1, block_id=42) == ResourceRef(kind="asset", page=1, block_id="42")
def test_resource_ref_rejects_group_without_group_id():
with pytest.raises(ValueError):
ResourceRef(kind="group", page=1, block_id=None)

View file

@ -6424,3 +6424,75 @@ def test_recover_internal_figure_number_inside_overlap_gate():
# Line that covers >15% of asset -> edge_band_score = 0
large_line = [100, 100, 500, 600] # covers large area
assert _asset_edge_band_score(large_line, asset) == 0.0
def test_recover_figure_heading_prefix_multi_column():
"""Column-aware guard: skip same-y interfering lines from other columns."""
from paperforge.worker.ocr_figures import _recover_figure_heading_prefix
from paperforge.worker.ocr_document import PageLayoutProfile
block = {
"page": 1,
"text": "fraction of time that the system remains idle before processing the next request.",
}
# 2-column layout: boundary at x=600
# Heading at left column, interfering body line at right column (i+1 in pure y-order),
# real caption line further below in left column
page_pdf_lines = {
1: [
{"page": 1, "text": "FIGURE 3.", "bbox": [100, 100, 300, 120]},
# Interfering line — right column, shares ≥15 char prefix, would be i+1 in y-order
{"page": 1, "text": "fraction of time that", "bbox": [700, 130, 850, 150]},
# Real caption line — left column, below heading
{"page": 1, "text": "fraction of time that the system", "bbox": [100, 160, 350, 180]},
],
}
page_layouts = {
1: PageLayoutProfile(column_count=2, column_boundaries=[0, 600, 1200]),
}
# Multi-column mode: should skip the right-column line and find the left-column caption
result = _recover_figure_heading_prefix(block, page_pdf_lines, page_layouts)
assert result is not None, "Expected prefix recovery in 2-column mode"
assert result.startswith("FIGURE 3.")
assert "fraction of time that the system remains idle" in result
# Single-column mode (no page_layouts): should hit the interfering line (false positive)
result_single = _recover_figure_heading_prefix(block, page_pdf_lines)
assert result_single is not None, "False positive expected in single-column mode"
# The recovery is wrong — it prepends FIGURE 3. to the interfering line's text,
# not the full caption
assert result_single is not None
def test_build_figure_inventory_wrapper_stays_legacy_path(monkeypatch):
from paperforge.worker import ocr_figures
called = {"legacy": 0, "vnext": 0}
def fake_legacy(blocks, page_width=1200, page_pdf_lines_by_page=None):
called["legacy"] += 1
return {"source": "legacy"}
def fake_vnext(blocks, page_width=1200):
called["vnext"] += 1
return {"source": "vnext"}
monkeypatch.setattr(ocr_figures, "build_figure_inventory_legacy", fake_legacy)
monkeypatch.setattr(ocr_figures, "build_figure_inventory_vnext", fake_vnext)
result = ocr_figures.build_figure_inventory([], 1200)
assert result == {"source": "legacy"}
assert called == {"legacy": 1, "vnext": 0}
def test_vnext_entrypoint_is_callable_without_cutover():
from paperforge.worker.ocr_figures import build_figure_inventory_vnext
result = build_figure_inventory_vnext([], 1200)
assert isinstance(result, dict)
assert result.get("pipeline_mode") == "vnext"
assert result.get("matched_figures") == []

View file

@ -1,5 +1,6 @@
"""Production-path regression gate for OCR-v2 real papers.
Primary: fixture-backed deterministic replay via build_raw_blocks_for_result_lines
-> build_structured_blocks -> figure/table inventory -> render_fulltext_markdown.
@ -10,6 +11,8 @@ Secondary: env-driven audit tests (marker: @pytest.mark.audit, skip if
from __future__ import annotations
import json
import csv
import os
import re
from pathlib import Path
@ -1192,3 +1195,44 @@ def test_3fdt9652_multi_column_figures_not_merged(tmp_path: Path) -> None:
assert fig2[0].get("legend_block_id") != fig3[0].get("legend_block_id"), (
"Fig 2 and Fig 3 share the same legend_block_id"
)
def _load_trace_csv(key: str) -> list[dict]:
"""Load block_trace.csv from fixture directory."""
path = FIXTURE_ROOT / key / "block_trace.csv"
if not path.exists():
pytest.skip(f"block_trace.csv not found for {key}")
with open(path, newline="", encoding="utf-8-sig") as f:
return list(csv.DictReader(f))
def _load_layout_expectations(key: str) -> dict:
"""Load 'layout' section from expectations.json."""
exp = _load_expectations(key)
return exp.get("layout", {})
MIXED_TAIL_FIXTURES = ["9TW98JH8", "MZC482YI"]
@pytest.mark.parametrize("key", MIXED_TAIL_FIXTURES)
def test_mixed_tail_fixture_layout_classification(key: str) -> None:
"""Assert fixtures classified as mixed_tail have reference_item (≥3)
and body_paragraph blocks on the same page."""
trace = _load_trace_csv(key)
layout_exp = _load_layout_expectations(key)
mixed_tail_pages = layout_exp.get("mixed_tail_pages", [])
min_refs = layout_exp.get("min_reference_items_on_mixed_tail_page", 3)
assert mixed_tail_pages, f"No mixed_tail_pages defined for {key}"
for page in mixed_tail_pages:
page_blocks = [b for b in trace if int(b.get("page", 0)) == page]
roles = [b.get("role", "") for b in page_blocks]
ref_count = roles.count("reference_item")
body_count = roles.count("body_paragraph")
assert body_count >= 1, \
f"{key} page {page}: expected body_paragraph blocks, got {body_count}"
assert ref_count >= min_refs, \
f"{key} page {page}: expected ≥{min_refs} reference_item blocks, got {ref_count}"