mirror of
https://github.com/lllin000/PaperForge.git
synced 2026-07-22 06:50:53 +00:00
merge: ocr pairing framework
# Conflicts: # paperforge/worker/ocr_tables.py # tests/test_ocr_tables.py
This commit is contained in:
LLLin000
commit
fa734f6fea
30 changed files with 6533 additions and 1834 deletions
|
|
@ -1,13 +1,13 @@
|
|||
# OCR-v2 Project Management Log
|
||||
|
||||
> **Branch:** `feat/figure-pipeline-vnext` | **Last Updated:** 2026-07-03
|
||||
> **Active work:** Figure pipeline vnext cutover completed in worktree. Wrapper now points to vnext by default. 346 tests pass. Next: merge branch back to `master`.
|
||||
> **Branch:** `feat/ocr-pairing-framework` | **Last Updated:** 2026-07-03
|
||||
> **Active work:** Generic OCR pairing framework extracted; figure and table now both run on the shared pairing core in worktree. Targeted merge-unblock verification passed: 357 tests green, touched-file `ruff check`/`ruff format --check` green, 6 runnable real-paper table fixtures validated. Next: merge branch back to `master`.
|
||||
|
||||
---
|
||||
|
||||
## 0. Executive Summary
|
||||
|
||||
**Current state:** Figure pipeline vnext is now fully implemented (11 passes incl. accounting), cutover corpus reviewed on 5 papers with 0 consumed-asset regressions, and the public wrapper has been switched to vnext on this branch. 346 tests pass (`59` vnext + `258` figure + `29` render). Next: merge `feat/figure-pipeline-vnext` back to `master`.
|
||||
**Current state:** The OCR pairing framework branch is merge-ready. Figure and table pipelines now share `ocr_pairing_types.py` / `ocr_pairing_state.py` / `ocr_pairing_framework.py`, while domain-specific logic stays in `ocr_figure_domain.py` and `ocr_table_domain.py`. The last merge blockers were cleared by (1) moving figure-only rotation enrichment out of generic state into a figure-domain hook, (2) hardening table semantic parity validation across all runnable real-paper fixtures including `37LK5T97`, and (3) cleaning touched-file lint/format issues. Verification: 357 targeted tests passed. Next: merge `feat/ocr-pairing-framework` back to `master`.
|
||||
---
|
||||
|
||||
## 1. Architecture
|
||||
|
|
@ -46,31 +46,25 @@ raw observations → structural signatures → stable anchors/families → zone
|
|||
|
||||
## 2. Current Status
|
||||
|
||||
### 2.1 Test Suite
|
||||
### 2.1 Test / Verification Status
|
||||
</br>
|
||||
| Suite | Result |
|
||||
|-------|--------|
|
||||
| Figure stack (figures + reader + containment + backmatter boundary) | **286 passed** ✅ |
|
||||
| Document + roles + render + gate + rebuild + state machine + trace | **330 passed, 0 failed** ✅ |
|
||||
| Author bio detection (ocr_bio) | **37 passed** ✅ |
|
||||
| Spec contracts | All passed ✅ |
|
||||
| Real-paper regressions | All passed ✅ |
|
||||
| **Total OCR tests** | **1018 passed, 275 skipped (fixture unavailable), 0 failed** ✅ |
|
||||
| Pairing framework cutover suites (`test_ocr_figures` + `test_ocr_rebuild` + `test_ocr_tables` + `test_ocr_pairing_framework` + `test_ocr_table_pairing_framework`) | **357 passed, 0 failed** ✅ |
|
||||
| Real-paper table parity fixtures | **6 runnable fixtures checked** ✅ |
|
||||
| Touched-file lint (`ruff check`) | **OK** ✅ |
|
||||
| Touched-file format (`ruff format --check`) | **OK** ✅ |
|
||||
</br>
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| Structural gate | Installed |
|
||||
| Role assignment (seed only) | ✅ |
|
||||
| Zone inference + fallback | ✅ |
|
||||
| Figure inventory | Global distance clustering — correct |
|
||||
| Backmatter boundary | ✅ Ref-anchored partition committed |
|
||||
| Pre-ref tail zone | ✅ Fixed |
|
||||
| Abstract detection | ✅ Fixed |
|
||||
| Frontmatter heading normalization | ✅ Fixed — no text matching |
|
||||
| non_body_insert clustering | ✅ Fixed — page-1 guard threshold |
|
||||
| Render (backmatter headings) | ✅ Correct heading rendering |
|
||||
| State machine | ✅ Accepts done_degraded as terminal |
|
||||
| Rebuild (span backfill skip) | ✅ Version match check fixed |
|
||||
| Author bio detection (Passes B+C) | ✅ P0: post-ref text-only bios as backmatter_body. P1: figure-residual portrait assets as author_bio_asset + figure_caption support |
|
||||
| Figure pipeline vnext | ✅ Default wrapper switched; 11-pass pipeline active; cutover corpus reviewed (5 papers, 0 regressions) |
|
||||
| Figure pipeline vnext | ✅ Shared pairing core + figure-only pre-enricher hook |
|
||||
| Table pipeline vnext | ✅ Shared pairing core + semantic parity validation |
|
||||
| Pairing framework core | ✅ `ocr_pairing_*` modules active for both figure and table |
|
||||
| Rebuild/orchestrator seam | ✅ Public wrappers unchanged; callers still use `build_figure_inventory(...)` / `build_table_inventory(...)` |
|
||||
| Cross-domain figure/table conflict resolution | ✅ Still external to pairing core |
|
||||
|
||||
### 2.3 Fix Status
|
||||
|
||||
|
|
@ -95,48 +89,40 @@ raw observations → structural signatures → stable anchors/families → zone
|
|||
| 18 | — | **Issue 2**: Demoted-caption figure inner-text leakage | Container bbox regions | Validated `_container_bbox` regions in `tag_figure_contained_text` via 3 helpers + containment-only integration | `0e4ecbc` |
|
||||
| 19 | — | **Issue 5**: Cross-column page_assets groups falsely accepted | Column-homogeneity gate | `_column_band_id` + rejection in `_is_safe_page_assets_group` | `4ab227e` |
|
||||
| 20 | — | **Issue 6**: Same asset consumed by figure AND table | Post-hoc arbitration | `resolve_media_asset_conflicts` resolves asymmetric cases; weak/weak stays in `ownership_conflicts` | `4ab227e` |
|
||||
| 21 | — | Pairing framework extraction + figure migration | Refactor | Extracted shared `ocr_pairing_*` core from figure vnext; preserved public seams and figure behavior | `6229f6c`, `7cfbb5f`, `32541cf` |
|
||||
| 22 | — | Table vnext on pairing framework + public cutover | Refactor/feature | Added `ocr_table_domain.py` + `ocr_table_passes.py`, preserved resolver-consumed fields, switched `build_table_inventory(...)` to vnext | `db01518`, `ea6a1f0`, `a9e68ac`, `a2e5788` |
|
||||
| 23 | 37LK5T97 + cutover corpus | Merge-unblock hardening | Moved figure-only rotation enrichment out of generic state, validated 6 runnable real-paper table fixtures semantically, fixed touched-file lint/format issues | working tree (pre-merge session) |
|
||||
#### P2#1a — Previous-page legend locator bridge (✅ Fixed `3f61f4a`)
|
||||
WV2FF4NV Fig 10: locator "See legend on previous page" on p16 not bridged to full legend (in rejected_legends, misclassified as body_paragraph)
|
||||
|
||||
#### Remaining P2
|
||||
#### Remaining P2 / P3
|
||||
|
||||
1. **Figure containment gaps** — `cluster_bbox` containment only runs on matched figures; composite figures with demoted caption upstream never enter containment. Inner-text detection misses `vision_footnote`/`paragraph_title` raw_labels.
|
||||
2. **Short "Table N" caption matching** — bare `"Table 1."` labels miss table_asset in ownership pipeline. Existing code partially handles it; residual cases remain.
|
||||
3. **Page-1 body paragraph OCR text extraction failure** (was: width check) — W33NLVJ2 right col Introduction block: OCR detected text region (raw_label=text) but extracted empty string. Pipeline previously mis-signaled as `unknown_structural`. Fixed: now `ocr_text_missing`.
|
||||
4. **Body text backfill overlap** — `backfill_missing_text_from_pdf` uses `get_text("words", clip=expanded)` returning words beyond block's bbox; can cause text duplication at render level.
|
||||
5. **2HEUD5P9 reference ordering** — ✅ Fixed by F12 (raw_label=reference_content signal)
|
||||
6. **Mixed-column tail page zone absorption** — ✅ Fixed: tighten year-period regex
|
||||
|
||||
### P3 (Boundary / Edge Cases)
|
||||
|
||||
1. **DW biography page mismatch** — pages 32-34 vs expectations 33-34.
|
||||
2. **AJR side-caption recovery** — deliberately deferred; not part of generic inventory refactor.
|
||||
3. **Chinese Windows encoding** — non-ASCII PDF filenames garbled via GBK codepage. Glob fallback added; residual `meta.json` corruption from earlier runs.
|
||||
4. **`backfill_missing_text_from_pdf` bbox-exact filtering** — need decision on render-level dedup vs backfill-level bbox clamp.
|
||||
5. **Multi-caption page column collapse** — `page_assets` group on multi-column pages merges separate figures in different columns.
|
||||
6. **Figure/table shared-consumption registry** — no shared consumed registry for ambiguous image-like blocks.
|
||||
7. **Short papers (<3 pages) health falsely red** — no headings/abstract in Letter/Editor formats.
|
||||
1. **Compatibility naming debt** — `figure_no`, `legend`, and `FigurePipelineState` remain backwards-compat names inside the shared pairing core. They are intentionally deferred cleanup, not release blockers.
|
||||
2. **37LK5T97 legacy consumption bug** — legacy table inventory drops caption `block_id=0` from `consumed_block_ids` because `if bid` treats `0` as falsy; vnext keeps `'0'`. Validation now treats this as a known legacy bug, not a vnext regression.
|
||||
3. **Figure containment gaps** — `cluster_bbox` containment only runs on matched figures; composite figures with demoted caption upstream still do not enter containment.
|
||||
4. **Short "Table N" caption matching** — bare `"Table 1."` labels still have residual weak cases outside the validated fixture set.
|
||||
5. **Chinese Windows encoding** — non-ASCII PDF filenames can still surface GBK path issues in older artifacts.
|
||||
6. **Shared consumed registry for ambiguous image-like blocks** — figure/table post-hoc arbitration exists, but there is still no first-class shared consumed registry for unresolved image-like assets.
|
||||
|
||||
## 4. Active Queue
|
||||
|
||||
1. ✅ P1 backmatter boundary (ref-anchored partition)
|
||||
2. ✅ Pre-ref tail zone fix (4KCHGV2Z)
|
||||
3. ✅ Gate 5 frontmatter fix series (24YKLTHQ)
|
||||
4. ✅ Stale trace-vs-expectation fixtures cleared (10 assertion updates)
|
||||
5. ✅ All stale test expectations reconciled (non_body_insert, caffard, legend_like, structural gate, render, state machine, rebuild, truth docs)
|
||||
6. ✅ P0 author bio detection (post_ref_bio_cleanup for reference_item)
|
||||
7. ✅ P1 author bio detection (residual_author_bio_pass + figure_caption support + tag_figure_contained_text protection)
|
||||
8. **NEXT: Merge `feat/figure-pipeline-vnext` → `master`**
|
||||
1. ✅ Pairing framework extraction complete
|
||||
2. ✅ Figure migration onto shared pairing core complete
|
||||
3. ✅ Table migration onto shared pairing core complete
|
||||
4. ✅ Generic-state impurity removed (`_match_pre_enricher` hook at figure boundary)
|
||||
5. ✅ Real-paper table semantic parity validation hardened (6 runnable fixtures)
|
||||
6. ✅ Touched-file lint / format cleanup complete
|
||||
7. **NEXT: Merge `feat/ocr-pairing-framework` → `master`**
|
||||
|
||||
### 4.1 Immediate Next Steps
|
||||
|
||||
- [x] Phase 0-4 vnext implementation complete
|
||||
- [x] Phase 5 cutover evaluation complete
|
||||
- [x] Cutover corpus: 5 papers, all 9 spec categories covered
|
||||
- [x] Cutover diff review: improvement=2, equivalent=2, parity=1, regression=0
|
||||
- [x] Wrapper switched to vnext in branch
|
||||
- [x] Full verification: 346 passed
|
||||
- [ ] Merge `feat/figure-pipeline-vnext` into `master`
|
||||
- [x] Shared `ocr_pairing_*` core active for figure + table
|
||||
- [x] Public wrappers preserved
|
||||
- [x] Table parity checks upgraded from smoke shape to semantic comparison
|
||||
- [x] `37LK5T97` made runnable under `tests/fixtures/ocr_vnext_real_papers/`
|
||||
- [x] Generic state made domain-neutral for match enrichment
|
||||
- [x] Targeted verification: 357 passed
|
||||
- [ ] Merge `feat/ocr-pairing-framework` into `master`
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -236,6 +222,9 @@ WV2FF4NV Fig 10: locator "See legend on previous page" on p16 not bridged to ful
|
|||
| 2026-07-01 | Broaden recovery gate to handle normal prematch unknown figures | Synthetic-figure gate (`bbox_only_asset` flag) excluded `figure_unknown_NNN` from normal rotated prematch path. Gate now allows figure_unknown figures without synthetic flags. |
|
||||
| 2026-07-03 | Cutover uses evidence gates, not code confidence | VNext matched or improved on the full cutover corpus with identical consumed asset sets; wrapper switch became a release decision only after diff review + gate verification. |
|
||||
| 2026-07-03 | Legacy schema tests must be upgraded before wrapper switch | Real-paper behavior was cutover-ready earlier, but `test_ocr_figures.py` still asserted legacy-only inventory keys. Updating the test contract was required to make wrapper switch honest. |
|
||||
| 2026-07-03 | Generic state uses a domain hook for figure rotation enrichment | Figure rotation metadata had to stay behaviorally intact without polluting the shared pairing state used by table passes. A pre-match enricher hook keeps the core generic and the figure path exact. |
|
||||
| 2026-07-03 | Table parity comparison normalizes benign storage drift only | int/str block IDs, ordering differences, and `None` vs empty unmatched asset IDs are storage-level drift already tolerated downstream. Validation now compares semantic fields rather than raw serialization artifacts. |
|
||||
| 2026-07-03 | Legacy `block_id=0` drop is treated as a legacy bug, not a parity target | `37LK5T97` showed legacy dropping consumed caption id `0` via falsy filtering. The branch keeps vnext truth and documents the legacy defect instead of reproducing it. |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -327,6 +316,7 @@ python -m ruff check paperforge/worker/ocr_*.py
|
|||
| 2026-07-02 | Round 2 truth audit + 3 targeted bug fixes for 37LK5T97 | Batch-audited 10 fresh papers (5 GREEN / 4 YELLOW / 1 RED). 37LK5T97 found with Figure 1 broken (sidecar caption demoted) + 6 unmatched rotated tables. Fixes: (1) `_is_sidecar_candidate` guard in candidate_resolution, (2) `adjacent_x`+`y_overlap` in score_table_match for rotated captions, (3) rotated table render bbox+270° correction. Also: rotated figure crop quality fix (4x zoom + coordinate normalization in `_crop_asset_from_pdf`). Commits: `59cd01a`, `bd3f3b6`, `86e0d14`. 428 regression tests pass. | §9.20 |
|
||||
| 2026-07-02 | Zone/role robustness completion — Figure caption prefix recovery + inline table fix + table matching audit | Figure caption prefix recovery from PDF text layer (`_recover_figure_heading_prefix`): 5S7UI34M 4→9 matched figures, 33→1 unmatched. Inline `<table>` HTML role fix: 650 blocks now `table_html` after rebuild (priority bug: raw_label=table fired before `<table>` check). Table matching audit: 620 remaining `media_asset` pending full rebuild. 585 figure/table/role tests pass. | §9.21 |
|
||||
| 2026-07-03 | Figure pipeline vnext cutover completed | Implemented all remaining vnext passes (composite parent, group/classic sequential, unresolved consolidation, accounting), expanded compare harness, curated 5-paper cutover corpus covering all 9 spec categories, generated diff review (improvement=2 / equivalent=2 / parity=1 / regression=0), updated legacy figure tests for vnext contract, and switched `build_figure_inventory(...)` wrapper to vnext on branch `feat/figure-pipeline-vnext`. Verification: 346 tests passed. | §9.22 |
|
||||
| 2026-07-03 | OCR pairing framework merge-unblock pass | Cleared the remaining merge blockers on `feat/ocr-pairing-framework`: moved figure-only rotation enrichment out of generic state, upgraded table cutover validation to semantic parity across 6 runnable real-paper fixtures including `37LK5T97`, and cleaned touched-file lint/format issues. Verification: 357 targeted tests passed; merge-ready. | §9.23 |
|
||||
|
||||
## 9. Historical Detail Archive
|
||||
|
||||
|
|
@ -596,3 +586,43 @@ Full-day debugging session across 8 gold papers. 98 bug annotations, 8 pipeline
|
|||
- `paperforge/worker/ocr_roles.py` — inline `<table>` before raw_label=table
|
||||
- `paperforge/worker/ocr_document.py` — removed `table_html_candidate` dead path
|
||||
- `paperforge/worker/ocr_structural_gate.py` — `table_html` verifier
|
||||
|
||||
### 9.22 Plan A: OCR Pairing Framework Extraction (2026-07-03)
|
||||
|
||||
**Goal:** Extract generic OCR pairing mechanics from figure vnext and migrate figure onto the framework with no behavior change. Table vnext deferred to Plan B.
|
||||
|
||||
**Architecture (Option B):**
|
||||
- Generic framework: `ocr_pairing_types.py`, `ocr_pairing_state.py`, `ocr_pairing_framework.py`
|
||||
- Figure domain: `ocr_figure_domain.py` (FigureCorpus, FigureCandidateIndex), 8 pass files (import paths only)
|
||||
- Compatibility shims: `ocr_figure_vnext_types.py`, `ocr_figure_vnext_state.py`, `ocr_figure_vnext_corpus.py` each re-export from framework/domain
|
||||
- `ocr_figures.py`: orchestration loop replaced with `run_pairing_passes(state, pass_classes)` from framework
|
||||
|
||||
**Key decisions:**
|
||||
- Keep `figure_no` and `FigurePipelineState` names in Plan A (rename deferred until migration stable)
|
||||
- Plan A extracts pass orchestration only, not full framework-owned arbitration
|
||||
- Table vnext deferred — no table file changes
|
||||
|
||||
**Commits (branch `feat/ocr-pairing-framework`, 6 commits over master):**
|
||||
| # | Commit | Description |
|
||||
|---|--------|-------------|
|
||||
| 1 | `0f94123` | docs: add Plan A implementation document |
|
||||
| 2 | `96a5ddd` | fix(tests): update stale span_backfill_version constants |
|
||||
| 3 | `1cc44b5` | test(ocr): lock figure vnext extraction baseline |
|
||||
| 4 | `6229f6c` | refactor(ocr): extract generic pairing types and state |
|
||||
| 5 | `7cfbb5f` | refactor(ocr): add pairing pass runner and figure domain module |
|
||||
| 6 | `32541cf` | test(ocr): prove rebuild compatibility with pairing framework |
|
||||
|
||||
**Files changed:** 21 files, +1546 -364
|
||||
|
||||
**Tests:** 288 pass in figure + rebuild + pairing suites (0 failed). Pre-existing `paperforge.resources` ModuleNotFoundError in test_ocr_document.py unrelated.
|
||||
|
||||
**Verification gates:**
|
||||
- Pre-existing rebuild test fixed (stale version constant)
|
||||
- `build_figure_inventory` → `build_figure_inventory_vnext` delegation test passes
|
||||
- Shim identity tests prove re-exports are the same class (not copies)
|
||||
- Rebuild compatibility test calls `run_derived_rebuild_for_keys()` for real and asserts `build_figure_inventory` is invoked
|
||||
- No table file changes in diff
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-07-03-ocr-pairing-framework-design.md`
|
||||
**Plan:** `docs/superpowers/plans/2026-07-03-ocr-pairing-framework-plan-a.md`
|
||||
**Branch:** `feat/ocr-pairing-framework` (in worktree `.worktrees/feat-ocr-pairing-framework/`)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,903 @@
|
|||
# OCR Pairing Framework Plan A Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Extract a generic OCR pairing framework from the current figure vnext implementation and migrate figure pairing onto it with no intended behavior change.
|
||||
|
||||
**Architecture:** Keep the external seam unchanged: `build_figure_inventory(...)` continues to return the same inventory contract and `ocr_rebuild.py` / OCR post-processing continue calling it the same way. Extract only the generic pairing mechanics into `ocr_pairing_*` modules, keep figure facts/hypotheses/assembly in figure-domain modules, and do not implement table vnext in this plan.
|
||||
|
||||
**Tech Stack:** Python 3.x, pytest, OCR worker modules under `paperforge/worker/`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- This plan is **Plan A only**.
|
||||
- Do **not** implement table vnext in this plan.
|
||||
- Do **not** change table heuristics in this plan.
|
||||
- Do **not** redesign OCR roles, layout, zones, or document structure.
|
||||
- Treat figure migration as a **no-behavior-change refactor**.
|
||||
- Preserve the external contract of `build_figure_inventory(...)` and `build_figure_inventory_vnext(...)`.
|
||||
- Preserve current pass order inside figure vnext.
|
||||
- In Plan A, extract **pass orchestration only** into `ocr_pairing_framework.py`; do **not** claim this plan centralizes proposal arbitration.
|
||||
- Keep passes using existing `OwnershipLedger` and state accept methods where they already do; do **not** introduce new direct ownership bypasses in Plan A.
|
||||
- If compatibility shims are needed for old `ocr_figure_vnext_types.py` / `ocr_figure_vnext_state.py`, make them explicit and temporary.
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
- `paperforge/worker/ocr_figures.py` — figure entrypoints and vnext orchestration
|
||||
- `paperforge/worker/ocr_figure_vnext_types.py` — current generic pairing types to extract
|
||||
- `paperforge/worker/ocr_figure_vnext_state.py` — current ledger/state to extract
|
||||
- `paperforge/worker/ocr_figure_vnext_corpus.py` — current figure corpus/index, target of domain migration
|
||||
- `paperforge/worker/ocr_figure_vnext_passes.py` — primary figure passes
|
||||
- `paperforge/worker/ocr_figure_vnext_sidecar_pass.py` — figure sidecar pass
|
||||
- `paperforge/worker/ocr_figure_vnext_locator_pass.py` — figure locator pass
|
||||
- `paperforge/worker/ocr_figure_vnext_group_seq_pass.py` — figure group-sequential pass
|
||||
- `paperforge/worker/ocr_figure_vnext_composite_pass.py` — figure composite pass
|
||||
- `paperforge/worker/ocr_figure_vnext_classic_seq_pass.py` — figure classic sequential / unresolved-cluster passes
|
||||
- `paperforge/worker/ocr_figure_vnext_bundle_pass.py` — figure legend bundle pass
|
||||
- `paperforge/worker/ocr_figure_vnext_accounting_pass.py` — final figure accounting pass
|
||||
- `paperforge/worker/ocr_rebuild.py` — rebuild caller that must remain contract-compatible
|
||||
- `tests/test_ocr_figures.py` — main figure regression and unit coverage
|
||||
- `tests/test_ocr_rebuild.py` — rebuild smoke coverage if needed for compatibility proof
|
||||
|
||||
Target new files:
|
||||
|
||||
- `paperforge/worker/ocr_pairing_types.py`
|
||||
- `paperforge/worker/ocr_pairing_state.py`
|
||||
- `paperforge/worker/ocr_pairing_framework.py`
|
||||
- `paperforge/worker/ocr_figure_domain.py`
|
||||
- `tests/test_ocr_pairing_framework.py`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Establish the no-behavior-change baseline
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/test_ocr_figures.py`
|
||||
- Create: `tests/test_ocr_pairing_framework.py`
|
||||
- Optional Create: `scripts/dev/compare_figure_inventory_vnext_baseline.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `build_figure_inventory(structured_blocks, page_width=1200, page_pdf_lines_by_page=None) -> dict[str, Any]`
|
||||
- Produces:
|
||||
- `test_build_figure_inventory_delegates_to_vnext()`
|
||||
- `test_vnext_types_module_is_currently_generic_surface()`
|
||||
- `test_vnext_state_module_is_currently_generic_surface()`
|
||||
- optional baseline helper: `compare_vnext_inventory_baseline(...) -> int`
|
||||
|
||||
- [ ] **Step 1: Write the failing baseline tests**
|
||||
|
||||
Add to `tests/test_ocr_figures.py`:
|
||||
|
||||
```python
|
||||
def test_build_figure_inventory_delegates_to_vnext(monkeypatch):
|
||||
from paperforge.worker import ocr_figures
|
||||
|
||||
called = {}
|
||||
|
||||
def fake_vnext(structured_blocks, page_width=1200, page_pdf_lines_by_page=None):
|
||||
called["args"] = (structured_blocks, page_width, page_pdf_lines_by_page)
|
||||
return {"pipeline_mode": "vnext", "matched_figures": []}
|
||||
|
||||
monkeypatch.setattr(ocr_figures, "build_figure_inventory_vnext", fake_vnext)
|
||||
|
||||
blocks = [{"block_id": "b1", "page": 1, "role": "body_text", "text": "x"}]
|
||||
result = ocr_figures.build_figure_inventory(blocks, page_width=777, page_pdf_lines_by_page={1: []})
|
||||
|
||||
assert result["pipeline_mode"] == "vnext"
|
||||
assert called["args"] == (blocks, 777, {1: []})
|
||||
```
|
||||
|
||||
Add to `tests/test_ocr_pairing_framework.py`:
|
||||
|
||||
```python
|
||||
def test_vnext_types_module_is_currently_generic_surface():
|
||||
from paperforge.worker.ocr_figure_vnext_types import ClaimProposal, PassReport, ResourceRef
|
||||
|
||||
assert ResourceRef.__name__ == "ResourceRef"
|
||||
assert ClaimProposal.__name__ == "ClaimProposal"
|
||||
assert PassReport.__name__ == "PassReport"
|
||||
|
||||
|
||||
def test_vnext_state_module_is_currently_generic_surface():
|
||||
from paperforge.worker.ocr_figure_vnext_state import FigurePipelineState, OwnershipLedger
|
||||
|
||||
state = FigurePipelineState(corpus=None, candidate_index=None, ledger=OwnershipLedger())
|
||||
|
||||
assert state.matches == []
|
||||
assert state.reservations == []
|
||||
assert state.unresolved == []
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify the baseline is real**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest \
|
||||
tests/test_ocr_figures.py::test_build_figure_inventory_delegates_to_vnext \
|
||||
tests/test_ocr_pairing_framework.py::test_vnext_types_module_is_currently_generic_surface \
|
||||
tests/test_ocr_pairing_framework.py::test_vnext_state_module_is_currently_generic_surface -q
|
||||
```
|
||||
|
||||
Expected: PASS. If any test fails, stop and fix the baseline before extraction.
|
||||
|
||||
- [ ] **Step 3: Add a representative output-baseline smoke helper**
|
||||
|
||||
Create `scripts/dev/compare_figure_inventory_vnext_baseline.py`:
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from paperforge.worker.ocr_figures import build_figure_inventory
|
||||
|
||||
|
||||
def compare_vnext_inventory_baseline(fixture_root: Path) -> dict[str, object]:
|
||||
structured_path = fixture_root / "structure" / "blocks.structured.jsonl"
|
||||
blocks = [json.loads(line) for line in structured_path.read_text(encoding="utf-8").splitlines() if line.strip()]
|
||||
inventory = build_figure_inventory(blocks)
|
||||
return {
|
||||
"pipeline_mode": inventory.get("pipeline_mode"),
|
||||
"matched_ids": [m.get("figure_id") for m in inventory.get("matched_figures", [])],
|
||||
"match_count": len(inventory.get("matched_figures", [])),
|
||||
}
|
||||
```
|
||||
|
||||
Do not overbuild this script. It exists only to capture a pre-extraction baseline on representative fixtures.
|
||||
|
||||
- [ ] **Step 4: Run the baseline helper on one representative real-paper fixture**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -c "from pathlib import Path; from scripts.dev.compare_figure_inventory_vnext_baseline import compare_vnext_inventory_baseline; print(compare_vnext_inventory_baseline(Path('tests/fixtures/ocr_real_papers/6QNRHRKX')))"
|
||||
```
|
||||
|
||||
Expected: a small dict showing `pipeline_mode='vnext'` and stable figure IDs / count.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
tests/test_ocr_figures.py \
|
||||
tests/test_ocr_pairing_framework.py \
|
||||
scripts/dev/compare_figure_inventory_vnext_baseline.py
|
||||
git commit -m "test(ocr): lock figure vnext extraction baseline"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Extract generic pairing types into `ocr_pairing_types.py`
|
||||
|
||||
**Files:**
|
||||
- Create: `paperforge/worker/ocr_pairing_types.py`
|
||||
- Modify: `paperforge/worker/ocr_figure_vnext_types.py`
|
||||
- Modify: `tests/test_ocr_pairing_framework.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: current `ResourceRef`, `OwnershipConflict`, `ClaimProposal`, `PassReport`
|
||||
- Produces:
|
||||
- `paperforge.worker.ocr_pairing_types.ResourceRef`
|
||||
- `paperforge.worker.ocr_pairing_types.OwnershipConflict`
|
||||
- `paperforge.worker.ocr_pairing_types.ClaimProposal`
|
||||
- `paperforge.worker.ocr_pairing_types.PassReport`
|
||||
- compatibility shim behavior in `ocr_figure_vnext_types.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing extraction tests**
|
||||
|
||||
Add to `tests/test_ocr_pairing_framework.py`:
|
||||
|
||||
```python
|
||||
def test_pairing_types_module_exports_generic_types():
|
||||
from paperforge.worker.ocr_pairing_types import ClaimProposal, PassReport, ResourceRef
|
||||
|
||||
ref = ResourceRef(kind="legend", page=1, block_id="b1")
|
||||
proposal = ClaimProposal(
|
||||
pass_name="p",
|
||||
figure_no=1,
|
||||
claim_type="match",
|
||||
legends=[ref],
|
||||
assets=[],
|
||||
groups=[],
|
||||
confidence=0.9,
|
||||
evidence_rank=1,
|
||||
reason="test",
|
||||
)
|
||||
report = PassReport(pass_name="p")
|
||||
|
||||
assert proposal.pass_name == "p"
|
||||
assert report.pass_name == "p"
|
||||
assert ref.page == 1
|
||||
|
||||
|
||||
def test_figure_vnext_types_module_reexports_framework_types():
|
||||
from paperforge.worker.ocr_pairing_types import ResourceRef as FrameworkResourceRef
|
||||
from paperforge.worker.ocr_figure_vnext_types import ResourceRef as FigureModuleResourceRef
|
||||
|
||||
assert FigureModuleResourceRef is FrameworkResourceRef
|
||||
|
||||
|
||||
def test_resource_ref_rejects_page_agnostic_asset():
|
||||
import pytest
|
||||
|
||||
from paperforge.worker.ocr_pairing_types import ResourceRef
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
ResourceRef(kind="asset", page=None, block_id="a1")
|
||||
|
||||
|
||||
def test_resource_ref_normalizes_block_id_type():
|
||||
from paperforge.worker.ocr_pairing_types import ResourceRef
|
||||
|
||||
assert ResourceRef(kind="asset", page=1, block_id=1) == ResourceRef(kind="asset", page=1, block_id="1")
|
||||
|
||||
|
||||
def test_resource_ref_rejects_group_without_group_id():
|
||||
import pytest
|
||||
|
||||
from paperforge.worker.ocr_pairing_types import ResourceRef
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
ResourceRef(kind="group", page=1, block_id=None)
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest \
|
||||
tests/test_ocr_pairing_framework.py::test_pairing_types_module_exports_generic_types \
|
||||
tests/test_ocr_pairing_framework.py::test_figure_vnext_types_module_reexports_framework_types \
|
||||
tests/test_ocr_pairing_framework.py::test_resource_ref_rejects_page_agnostic_asset \
|
||||
tests/test_ocr_pairing_framework.py::test_resource_ref_normalizes_block_id_type \
|
||||
tests/test_ocr_pairing_framework.py::test_resource_ref_rejects_group_without_group_id -q
|
||||
```
|
||||
|
||||
Expected: FAIL because `ocr_pairing_types.py` does not exist yet.
|
||||
|
||||
- [ ] **Step 3: Write the minimal extraction**
|
||||
|
||||
Create `paperforge/worker/ocr_pairing_types.py` with the current dataclasses moved verbatim first:
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Literal
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResourceRef:
|
||||
kind: Literal["legend", "asset", "group"]
|
||||
page: int | None
|
||||
block_id: str | None
|
||||
group_id: str | None = None
|
||||
figure_no: int | None = field(default=None, compare=False)
|
||||
origin: str | None = field(default=None, compare=False)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.page is not None:
|
||||
object.__setattr__(self, "page", int(self.page))
|
||||
if self.block_id is not None:
|
||||
object.__setattr__(self, "block_id", str(self.block_id))
|
||||
if self.group_id is not None:
|
||||
object.__setattr__(self, "group_id", str(self.group_id))
|
||||
|
||||
if self.kind == "asset" and (self.page is None or self.block_id is None):
|
||||
raise ValueError("asset ResourceRef requires page + block_id")
|
||||
if self.kind == "legend" and (self.page is None or self.block_id is None):
|
||||
raise ValueError("legend ResourceRef requires page + block_id")
|
||||
if self.kind == "group" and (self.page is None or self.group_id is None):
|
||||
raise ValueError("group ResourceRef requires page + group_id")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class OwnershipConflict:
|
||||
resource: ResourceRef
|
||||
current_owner: ResourceRef | None
|
||||
attempted_owner: ResourceRef | None
|
||||
reason: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class ClaimProposal:
|
||||
pass_name: str
|
||||
figure_no: int | None
|
||||
claim_type: Literal["match", "reserve", "block", "unresolved_cluster", "composite_parent"]
|
||||
legends: list[ResourceRef]
|
||||
assets: list[ResourceRef]
|
||||
groups: list[ResourceRef]
|
||||
confidence: float
|
||||
evidence_rank: int
|
||||
reason: str
|
||||
diagnostics: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class PassReport:
|
||||
pass_name: str
|
||||
proposals: list[ClaimProposal] = field(default_factory=list)
|
||||
accepted: list[ClaimProposal] = field(default_factory=list)
|
||||
rejected: list[ClaimProposal] = field(default_factory=list)
|
||||
conflicts: list[OwnershipConflict] = field(default_factory=list)
|
||||
invariant_errors: list[str] = field(default_factory=list)
|
||||
```
|
||||
|
||||
Then replace `paperforge/worker/ocr_figure_vnext_types.py` with a compatibility shim:
|
||||
|
||||
```python
|
||||
from .ocr_pairing_types import ClaimProposal, OwnershipConflict, PassReport, ResourceRef
|
||||
|
||||
__all__ = ["ResourceRef", "OwnershipConflict", "ClaimProposal", "PassReport"]
|
||||
```
|
||||
|
||||
Do not rename `figure_no` to `entity_no` in Plan A. That belongs to a later compatibility-tightening task only after figure migration is stable.
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run the command from Step 2.
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
paperforge/worker/ocr_pairing_types.py \
|
||||
paperforge/worker/ocr_figure_vnext_types.py \
|
||||
tests/test_ocr_pairing_framework.py
|
||||
git commit -m "refactor(ocr): extract generic pairing types"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Extract generic state and ledger into `ocr_pairing_state.py`
|
||||
|
||||
**Files:**
|
||||
- Create: `paperforge/worker/ocr_pairing_state.py`
|
||||
- Modify: `paperforge/worker/ocr_figure_vnext_state.py`
|
||||
- Modify: `tests/test_ocr_pairing_framework.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `ResourceRef`, `ClaimProposal`, `OwnershipConflict`
|
||||
- Produces:
|
||||
- `paperforge.worker.ocr_pairing_state.OwnershipLedger`
|
||||
- `paperforge.worker.ocr_pairing_state.FigurePipelineState` (keep existing class name in Plan A for compatibility)
|
||||
- compatibility shim behavior in `ocr_figure_vnext_state.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing ledger / reexport tests**
|
||||
|
||||
Add to `tests/test_ocr_pairing_framework.py`:
|
||||
|
||||
```python
|
||||
def test_pairing_state_module_exports_ownership_ledger():
|
||||
from paperforge.worker.ocr_pairing_state import OwnershipLedger
|
||||
from paperforge.worker.ocr_pairing_types import ResourceRef
|
||||
|
||||
ledger = OwnershipLedger()
|
||||
owner = ResourceRef(kind="legend", page=1, block_id="cap")
|
||||
asset = ResourceRef(kind="asset", page=1, block_id="asset")
|
||||
|
||||
assert ledger.try_claim_assets([asset], owner=owner, reason="test") is None
|
||||
assert ledger.owner_of_asset(page=1, block_id="asset") == owner
|
||||
|
||||
|
||||
def test_figure_vnext_state_module_reexports_framework_ledger():
|
||||
from paperforge.worker.ocr_pairing_state import OwnershipLedger as FrameworkLedger
|
||||
from paperforge.worker.ocr_figure_vnext_state import OwnershipLedger as FigureModuleLedger
|
||||
|
||||
assert FigureModuleLedger is FrameworkLedger
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest \
|
||||
tests/test_ocr_pairing_framework.py::test_pairing_state_module_exports_ownership_ledger \
|
||||
tests/test_ocr_pairing_framework.py::test_figure_vnext_state_module_reexports_framework_ledger -q
|
||||
```
|
||||
|
||||
Expected: FAIL because `ocr_pairing_state.py` does not exist yet.
|
||||
|
||||
- [ ] **Step 3: Write the minimal extraction**
|
||||
|
||||
Create `paperforge/worker/ocr_pairing_state.py` by moving the current `OwnershipLedger` and `FigurePipelineState` verbatim first, only changing imports:
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from .ocr_pairing_types import ClaimProposal, OwnershipConflict, ResourceRef
|
||||
|
||||
|
||||
class OwnershipLedger:
|
||||
...
|
||||
|
||||
|
||||
@dataclass
|
||||
class FigurePipelineState:
|
||||
corpus: object | None
|
||||
candidate_index: object | None
|
||||
ledger: OwnershipLedger
|
||||
matches: list[dict] = field(default_factory=list)
|
||||
unresolved: list[dict] = field(default_factory=list)
|
||||
hypotheses: list[dict] = field(default_factory=list)
|
||||
diagnostics: list[dict] = field(default_factory=list)
|
||||
reservations: list[dict] = field(default_factory=list)
|
||||
completeness: dict = field(default_factory=dict)
|
||||
|
||||
def accept_match(self, proposal: ClaimProposal, match_record: dict) -> None:
|
||||
...
|
||||
|
||||
def accept_reservation(self, proposal: ClaimProposal) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
Then replace `paperforge/worker/ocr_figure_vnext_state.py` with a compatibility shim:
|
||||
|
||||
```python
|
||||
from .ocr_pairing_state import FigurePipelineState, OwnershipLedger
|
||||
|
||||
__all__ = ["OwnershipLedger", "FigurePipelineState"]
|
||||
```
|
||||
|
||||
Do not rename `FigurePipelineState` yet. That compatibility cleanup belongs after the framework extraction is proven stable.
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run the command from Step 2.
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
paperforge/worker/ocr_pairing_state.py \
|
||||
paperforge/worker/ocr_figure_vnext_state.py \
|
||||
tests/test_ocr_pairing_framework.py
|
||||
git commit -m "refactor(ocr): extract generic pairing state and ledger"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Add `ocr_pairing_framework.py` and migrate figure orchestration imports
|
||||
|
||||
**Files:**
|
||||
- Create: `paperforge/worker/ocr_pairing_framework.py`
|
||||
- Modify: `paperforge/worker/ocr_figures.py`
|
||||
- Modify: all figure pass files under `paperforge/worker/ocr_figure_vnext_*.py`
|
||||
- Modify: `tests/test_ocr_figures.py`
|
||||
- Modify: `tests/test_ocr_pairing_framework.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `FigurePipelineState`, `PassReport`, existing figure pass classes
|
||||
- Produces:
|
||||
- `run_pairing_passes(state, pass_classes) -> list[PassReport]`
|
||||
- figure pass imports now sourced from `ocr_pairing_types` / `ocr_pairing_state`
|
||||
- no change to per-pass arbitration behavior in Plan A
|
||||
- [ ] **Step 1: Write the failing runner test**
|
||||
|
||||
Add to `tests/test_ocr_pairing_framework.py`:
|
||||
|
||||
```python
|
||||
def test_run_pairing_passes_executes_in_declared_order():
|
||||
from paperforge.worker.ocr_pairing_framework import run_pairing_passes
|
||||
from paperforge.worker.ocr_pairing_state import FigurePipelineState, OwnershipLedger
|
||||
from paperforge.worker.ocr_pairing_types import PassReport
|
||||
|
||||
seen = []
|
||||
|
||||
class FirstPass:
|
||||
name = "first"
|
||||
def run(self, state):
|
||||
seen.append(self.name)
|
||||
return PassReport(pass_name=self.name)
|
||||
|
||||
class SecondPass:
|
||||
name = "second"
|
||||
def run(self, state):
|
||||
seen.append(self.name)
|
||||
return PassReport(pass_name=self.name)
|
||||
|
||||
state = FigurePipelineState(corpus=None, candidate_index=None, ledger=OwnershipLedger())
|
||||
reports = run_pairing_passes(state, [FirstPass, SecondPass])
|
||||
|
||||
assert seen == ["first", "second"]
|
||||
assert [r.pass_name for r in reports] == ["first", "second"]
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/test_ocr_pairing_framework.py::test_run_pairing_passes_executes_in_declared_order -q
|
||||
```
|
||||
|
||||
Expected: FAIL because `ocr_pairing_framework.py` does not exist yet.
|
||||
|
||||
- [ ] **Step 3: Write the minimal framework runner and migrate imports**
|
||||
|
||||
Create `paperforge/worker/ocr_pairing_framework.py`:
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
|
||||
def run_pairing_passes(state, pass_classes: Sequence[type]) -> list:
|
||||
reports = []
|
||||
for pass_cls in pass_classes:
|
||||
reports.append(pass_cls().run(state))
|
||||
return reports
|
||||
```
|
||||
|
||||
In `paperforge/worker/ocr_figures.py`, replace the local orchestration loop inside `build_figure_inventory_vnext(...)`:
|
||||
|
||||
```python
|
||||
reports = []
|
||||
for pass_cls in (
|
||||
PrimarySamePagePass,
|
||||
CompositeParentPass,
|
||||
SidecarPass,
|
||||
LocatorBridgePass,
|
||||
CrossPageReservationPass,
|
||||
CrossPageSettlementPass,
|
||||
LegendBundlePass,
|
||||
GroupSequentialPass,
|
||||
ClassicSequentialPass,
|
||||
UnresolvedClusterConsolidation,
|
||||
FinalAccountingPass,
|
||||
):
|
||||
reports.append(pass_cls().run(state))
|
||||
```
|
||||
|
||||
with:
|
||||
|
||||
```python
|
||||
from .ocr_pairing_framework import run_pairing_passes
|
||||
|
||||
reports = run_pairing_passes(
|
||||
state,
|
||||
[
|
||||
PrimarySamePagePass,
|
||||
CompositeParentPass,
|
||||
SidecarPass,
|
||||
LocatorBridgePass,
|
||||
CrossPageReservationPass,
|
||||
CrossPageSettlementPass,
|
||||
LegendBundlePass,
|
||||
GroupSequentialPass,
|
||||
ClassicSequentialPass,
|
||||
UnresolvedClusterConsolidation,
|
||||
FinalAccountingPass,
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
Then update all figure pass files to import from the extracted modules, for example:
|
||||
|
||||
```python
|
||||
from .ocr_pairing_types import ClaimProposal, PassReport, ResourceRef
|
||||
from .ocr_pairing_state import FigurePipelineState
|
||||
```
|
||||
|
||||
Keep pass logic unchanged in Plan A. This task only extracts **orchestration**, not framework-owned arbitration. Do not refactor passes to centralize proposal commit logic in this plan.
|
||||
|
||||
- [ ] **Step 4: Run focused tests to verify the runner is wired correctly**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest \
|
||||
tests/test_ocr_pairing_framework.py::test_run_pairing_passes_executes_in_declared_order \
|
||||
tests/test_ocr_figures.py::test_build_figure_inventory_delegates_to_vnext -q
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
paperforge/worker/ocr_pairing_framework.py \
|
||||
paperforge/worker/ocr_figures.py \
|
||||
paperforge/worker/ocr_figure_vnext_*.py \
|
||||
tests/test_ocr_pairing_framework.py \
|
||||
tests/test_ocr_figures.py
|
||||
git commit -m "refactor(ocr): add pairing pass runner for figure vnext"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Move figure facts and hypotheses into `ocr_figure_domain.py`
|
||||
|
||||
**Files:**
|
||||
- Create: `paperforge/worker/ocr_figure_domain.py`
|
||||
- Modify: `paperforge/worker/ocr_figure_vnext_corpus.py`
|
||||
- Modify: `paperforge/worker/ocr_figures.py`
|
||||
- Modify: `tests/test_ocr_pairing_framework.py`
|
||||
- Modify: `tests/test_ocr_figures.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: current `FigureCorpus`, `FigureCandidateIndex`
|
||||
- Produces:
|
||||
- `paperforge.worker.ocr_figure_domain.FigureCorpus`
|
||||
- `paperforge.worker.ocr_figure_domain.FigureCandidateIndex`
|
||||
- compatibility shim behavior in `ocr_figure_vnext_corpus.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing domain-module reexport tests**
|
||||
|
||||
Add to `tests/test_ocr_pairing_framework.py`:
|
||||
|
||||
```python
|
||||
def test_figure_domain_module_exports_corpus_and_candidate_index():
|
||||
from paperforge.worker.ocr_figure_domain import FigureCandidateIndex, FigureCorpus
|
||||
|
||||
assert FigureCorpus.__name__ == "FigureCorpus"
|
||||
assert FigureCandidateIndex.__name__ == "FigureCandidateIndex"
|
||||
|
||||
|
||||
def test_legacy_vnext_corpus_module_reexports_figure_domain_types():
|
||||
from paperforge.worker.ocr_figure_domain import FigureCorpus as DomainFigureCorpus
|
||||
from paperforge.worker.ocr_figure_vnext_corpus import FigureCorpus as LegacyModuleFigureCorpus
|
||||
|
||||
assert LegacyModuleFigureCorpus is DomainFigureCorpus
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest \
|
||||
tests/test_ocr_pairing_framework.py::test_figure_domain_module_exports_corpus_and_candidate_index \
|
||||
tests/test_ocr_pairing_framework.py::test_legacy_vnext_corpus_module_reexports_figure_domain_types -q
|
||||
```
|
||||
|
||||
Expected: FAIL because `ocr_figure_domain.py` does not exist yet.
|
||||
|
||||
- [ ] **Step 3: Write the minimal domain extraction**
|
||||
|
||||
Create `paperforge/worker/ocr_figure_domain.py` by moving `FigureCorpus` and `FigureCandidateIndex` from `ocr_figure_vnext_corpus.py` first.
|
||||
|
||||
If the moved code needs helpers from `ocr_figures.py`, keep those imports lazy inside methods or classmethods. Preferred pattern:
|
||||
|
||||
```python
|
||||
class FigureCandidateIndex:
|
||||
@classmethod
|
||||
def from_corpus(cls, corpus):
|
||||
from . import ocr_figures
|
||||
...
|
||||
```
|
||||
|
||||
Do **not** add a new top-level import cycle between `ocr_figures.py` and `ocr_figure_domain.py`.
|
||||
|
||||
Then replace `paperforge/worker/ocr_figure_vnext_corpus.py` with:
|
||||
|
||||
```python
|
||||
from .ocr_figure_domain import FigureCandidateIndex, FigureCorpus
|
||||
|
||||
__all__ = ["FigureCorpus", "FigureCandidateIndex"]
|
||||
```
|
||||
|
||||
In `paperforge/worker/ocr_figures.py`, change:
|
||||
|
||||
```python
|
||||
from .ocr_figure_vnext_corpus import FigureCandidateIndex, FigureCorpus
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```python
|
||||
from .ocr_figure_domain import FigureCandidateIndex, FigureCorpus
|
||||
```
|
||||
|
||||
Do not edit corpus/index behavior in this task beyond lazy-import adjustments needed to avoid circular imports.
|
||||
|
||||
- [ ] **Step 4: Run focused tests and one representative figure regression**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest \
|
||||
tests/test_ocr_pairing_framework.py::test_figure_domain_module_exports_corpus_and_candidate_index \
|
||||
tests/test_ocr_pairing_framework.py::test_legacy_vnext_corpus_module_reexports_figure_domain_types \
|
||||
tests/test_ocr_figures.py -q
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
paperforge/worker/ocr_figure_domain.py \
|
||||
paperforge/worker/ocr_figure_vnext_corpus.py \
|
||||
paperforge/worker/ocr_figures.py \
|
||||
tests/test_ocr_pairing_framework.py \
|
||||
tests/test_ocr_figures.py
|
||||
git commit -m "refactor(ocr): move figure corpus and candidate index into domain module"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Prove rebuild compatibility and close Plan A
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/test_ocr_rebuild.py`
|
||||
- Modify: `tests/test_ocr_pairing_framework.py`
|
||||
- Optional Modify: `paperforge/worker/ocr_rebuild.py` only if imports need compatibility cleanup
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: extracted framework modules, unchanged `build_figure_inventory(...)`
|
||||
- Produces:
|
||||
- rebuild-level compatibility test
|
||||
- final baseline comparison results proving no intended figure behavior change
|
||||
|
||||
- [ ] **Step 1: Write the rebuild compatibility test against the real rebuild path**
|
||||
|
||||
Add to `tests/test_ocr_rebuild.py`:
|
||||
|
||||
```python
|
||||
def test_run_derived_rebuild_for_keys_still_uses_public_build_figure_inventory(tmp_path: Path, monkeypatch) -> None:
|
||||
import json
|
||||
|
||||
from paperforge.worker.ocr_rebuild import run_derived_rebuild_for_keys
|
||||
|
||||
key = "TESTKEY1"
|
||||
paper_root = tmp_path / "System" / "PaperForge" / "ocr" / key
|
||||
(paper_root / "canonical").mkdir(parents=True)
|
||||
(paper_root / "raw").mkdir(parents=True)
|
||||
(paper_root / "structure").mkdir(parents=True)
|
||||
|
||||
(paper_root / "canonical" / "blocks.raw.jsonl").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"paper_id": key,
|
||||
"page": 1,
|
||||
"block_id": "p1_b1",
|
||||
"raw_label": "text",
|
||||
"raw_order": 0,
|
||||
"text": "Example text",
|
||||
"bbox": [10, 10, 100, 40],
|
||||
"page_width": 600,
|
||||
"page_height": 800,
|
||||
}
|
||||
)
|
||||
+ "\\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(paper_root / "raw" / "source_metadata.json").write_text(
|
||||
json.dumps({"title": "Example Title"}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(paper_root / "meta.json").write_text(json.dumps({"source_pdf": ""}), encoding="utf-8")
|
||||
|
||||
called = {"count": 0}
|
||||
|
||||
def fake_build_figure_inventory(structured_blocks, page_width=1200, page_pdf_lines_by_page=None):
|
||||
called["count"] += 1
|
||||
return {
|
||||
"pipeline_mode": "vnext",
|
||||
"matched_figures": [],
|
||||
"ambiguous_figures": [],
|
||||
"unmatched_legends": [],
|
||||
"unmatched_assets": [],
|
||||
"unresolved_clusters": [],
|
||||
"held_figures": [],
|
||||
"rejected_legends": [],
|
||||
"page_ledger": {},
|
||||
"residual_ledger": {},
|
||||
"local_pairing_hypotheses": [],
|
||||
"pass_reports": [],
|
||||
"completeness": {"total_numbered_legends": 0, "accounted_for": 0, "details": []},
|
||||
}
|
||||
|
||||
monkeypatch.setattr("paperforge.worker.ocr_figures.build_figure_inventory", fake_build_figure_inventory)
|
||||
monkeypatch.setattr("paperforge.worker.ocr_blocks.write_structured_blocks_jsonl", lambda *args, **kwargs: None)
|
||||
monkeypatch.setattr(
|
||||
"paperforge.worker.ocr_metadata.extract_frontmatter_candidates_from_blocks",
|
||||
lambda structured: {"title": "Example Title", "authors_text": None, "doi_candidates": []},
|
||||
)
|
||||
|
||||
result = run_derived_rebuild_for_keys(tmp_path, [key])
|
||||
|
||||
assert result["rebuild_count"] == 1
|
||||
assert called["count"] > 0
|
||||
```
|
||||
|
||||
This test must call `run_derived_rebuild_for_keys(...)` for real. Do not replace it with a hand-call to the fake function.
|
||||
|
||||
- [ ] **Step 2: Run the focused rebuild and framework tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest \
|
||||
tests/test_ocr_pairing_framework.py \
|
||||
tests/test_ocr_rebuild.py::test_run_derived_rebuild_for_keys_still_uses_public_build_figure_inventory -q
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 3: Run representative figure regression proof**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/test_ocr_figures.py -q
|
||||
python -c "from pathlib import Path; from scripts.dev.compare_figure_inventory_vnext_baseline import compare_vnext_inventory_baseline; print(compare_vnext_inventory_baseline(Path('tests/fixtures/ocr_real_papers/6QNRHRKX')))"
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- all figure tests PASS
|
||||
- baseline helper output remains unchanged from Task 1
|
||||
|
||||
- [ ] **Step 4: Remove any accidental table changes from the diff**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
git diff --name-only -- paperforge/worker/ocr_tables.py tests/test_ocr_tables.py
|
||||
```
|
||||
|
||||
Expected: no output. If there is output, revert it before committing.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
tests/test_ocr_rebuild.py \
|
||||
tests/test_ocr_pairing_framework.py \
|
||||
scripts/dev/compare_figure_inventory_vnext_baseline.py \
|
||||
paperforge/worker/ocr_pairing_types.py \
|
||||
paperforge/worker/ocr_pairing_state.py \
|
||||
paperforge/worker/ocr_pairing_framework.py \
|
||||
paperforge/worker/ocr_figure_domain.py \
|
||||
paperforge/worker/ocr_figure_vnext_types.py \
|
||||
paperforge/worker/ocr_figure_vnext_state.py \
|
||||
paperforge/worker/ocr_figure_vnext_corpus.py \
|
||||
paperforge/worker/ocr_figure_vnext_*.py \
|
||||
paperforge/worker/ocr_figures.py
|
||||
git commit -m "refactor(ocr): migrate figure vnext onto pairing framework"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review Checklist
|
||||
|
||||
- Spec coverage:
|
||||
- Preflight baseline gate -> Task 1
|
||||
- Framework type extraction -> Task 2
|
||||
- Framework state/ledger extraction -> Task 3
|
||||
- Framework runner / arbitration seam -> Task 4
|
||||
- Figure-domain migration -> Task 5
|
||||
- Rebuild compatibility and no-behavior proof -> Task 6
|
||||
- Table vnext intentionally excluded -> enforced in Global Constraints and Task 6 Step 4
|
||||
- Placeholder scan:
|
||||
- No `TODO` / `TBD` / “similar to Task N” placeholders remain.
|
||||
- One note in Task 6 explicitly tells the implementer to reuse existing rebuild fixture helpers in-file instead of inventing new infrastructure; this is intentional scoping guidance, not a missing design detail.
|
||||
- Type consistency:
|
||||
- Plan A keeps `figure_no` and `FigurePipelineState` names for compatibility.
|
||||
- Framework extraction is import-path-first; neutral renames like `entity_no` and `anchor` are deferred until after migration stability is proven.
|
||||
|
||||
## Execution Handoff
|
||||
|
||||
**Plan complete and saved to `docs/superpowers/plans/2026-07-03-ocr-pairing-framework-plan-a.md`. Two execution options:**
|
||||
|
||||
**1. Subagent-Driven (recommended)** - I dispatch a fresh subagent per task, review between tasks, fast iteration
|
||||
|
||||
**2. Inline Execution** - Execute tasks in this session using executing-plans, batch execution with checkpoints
|
||||
|
||||
**Which approach?**
|
||||
128
paperforge/worker/ocr_figure_domain.py
Normal file
128
paperforge/worker/ocr_figure_domain.py
Normal file
|
|
@ -0,0 +1,128 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass
|
||||
class FigureCorpus:
|
||||
"""Immutable facts extracted from structured blocks.
|
||||
|
||||
This dataclass stores raw, unfiltered data plus computed layout profiles.
|
||||
No derived hypotheses or candidate logic lives here.
|
||||
"""
|
||||
|
||||
blocks: list[dict]
|
||||
page_width: float
|
||||
raw_legends: list[dict] = field(default_factory=list)
|
||||
raw_assets: list[dict] = field(default_factory=list)
|
||||
locator_candidates: list[dict] = field(default_factory=list)
|
||||
page_layouts: dict[int, Any] = field(default_factory=dict)
|
||||
|
||||
@classmethod
|
||||
def from_blocks(cls, blocks: list[dict], page_width: float = 1200) -> FigureCorpus:
|
||||
from paperforge.worker.ocr_document import _build_page_layout_profiles
|
||||
|
||||
from . import ocr_figures
|
||||
|
||||
raw_legends = [
|
||||
b for b in blocks if b.get("role") in {"figure_caption", "figure_caption_candidate"}
|
||||
]
|
||||
raw_assets = [
|
||||
b for b in blocks if b.get("role") in {"figure_asset", "media_asset"}
|
||||
]
|
||||
locator_candidates = [
|
||||
b for b in raw_legends if ocr_figures._is_previous_page_legend_locator(b)
|
||||
]
|
||||
return cls(
|
||||
blocks=list(blocks),
|
||||
page_width=page_width,
|
||||
raw_legends=raw_legends,
|
||||
raw_assets=raw_assets,
|
||||
locator_candidates=locator_candidates,
|
||||
page_layouts=_build_page_layout_profiles(blocks),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class FigureCandidateIndex:
|
||||
"""Derived hypotheses and candidates built from a FigureCorpus.
|
||||
|
||||
These are derived interpretations — lists of formal legends, deduped legends
|
||||
(currently same as formal_legends — dedup over groups may be added later),
|
||||
rejected legends that didn't meet criteria for formal status, candidate
|
||||
groups built from assets, and per-page maps for sidecar/competition detection.
|
||||
"""
|
||||
|
||||
formal_legends: list[dict]
|
||||
held_legends: list[dict]
|
||||
rejected_legends: list[dict]
|
||||
deduped_legends: list[dict]
|
||||
candidate_groups: list[dict]
|
||||
competing_caption_pages: set[int]
|
||||
sidecar_candidates: dict[int, list[dict]]
|
||||
bundle_source_legend_ids: set[str]
|
||||
locator_candidates: list[dict]
|
||||
composite_parent_candidates: list[dict] = field(default_factory=list)
|
||||
|
||||
@classmethod
|
||||
def from_corpus(cls, corpus: FigureCorpus) -> FigureCandidateIndex:
|
||||
from . import ocr_figures
|
||||
|
||||
formal_legends = [
|
||||
b
|
||||
for b in corpus.raw_legends
|
||||
if ocr_figures._is_formal_legend(
|
||||
str(b.get("text", "")), b, corpus.page_width
|
||||
)
|
||||
]
|
||||
rejected_legends = [b for b in corpus.raw_legends if b not in formal_legends]
|
||||
candidate_groups = ocr_figures._build_candidate_figure_groups_from_assets(
|
||||
corpus.raw_assets,
|
||||
corpus.blocks,
|
||||
formal_legends,
|
||||
page_width=corpus.page_width,
|
||||
)
|
||||
competing_caption_pages = {
|
||||
int(leg.get("page"))
|
||||
for leg in formal_legends
|
||||
if leg.get("page") is not None
|
||||
and ocr_figures._extract_figure_number(str(leg.get("text", "")))
|
||||
is not None
|
||||
}
|
||||
# Populate sidecar_candidates: pages with >=2 aligned narrow captions.
|
||||
sidecar_candidates: dict[int, list[dict]] = {}
|
||||
_legends_by_page: dict[int, list[dict]] = {}
|
||||
for leg in formal_legends:
|
||||
_legends_by_page.setdefault(int(leg.get("page", 0) or 0), []).append(leg)
|
||||
for sp, spl in _legends_by_page.items():
|
||||
narrow_set = ocr_figures._same_page_narrow_caption_column(spl, corpus.page_width)
|
||||
if len(narrow_set) >= 2:
|
||||
sidecar_candidates[sp] = narrow_set
|
||||
|
||||
# Populate bundle_source_legend_ids: legends on pages with >=3 numbered legends
|
||||
# and zero assets on that page.
|
||||
bundle_source_legend_ids = ocr_figures._identify_bundle_source_legend_ids(
|
||||
formal_legends, corpus.raw_assets
|
||||
)
|
||||
|
||||
# Populate composite_parent_candidates: same-page atomic groups with
|
||||
# horizontal alignment + vertical adjacency (2x2 grid, columnar stacks).
|
||||
composite_parent_candidates = ocr_figures._build_composite_parent_figure_groups_visual_only(
|
||||
candidate_groups,
|
||||
corpus.raw_assets,
|
||||
corpus.blocks,
|
||||
corpus.page_width,
|
||||
)
|
||||
return cls(
|
||||
formal_legends=formal_legends,
|
||||
held_legends=[],
|
||||
rejected_legends=rejected_legends,
|
||||
deduped_legends=formal_legends,
|
||||
candidate_groups=candidate_groups,
|
||||
competing_caption_pages=competing_caption_pages,
|
||||
sidecar_candidates=sidecar_candidates,
|
||||
bundle_source_legend_ids=bundle_source_legend_ids,
|
||||
locator_candidates=corpus.locator_candidates,
|
||||
composite_parent_candidates=composite_parent_candidates,
|
||||
)
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from .ocr_figure_vnext_types import PassReport, ResourceRef
|
||||
from .ocr_pairing_types import PassReport, ResourceRef
|
||||
|
||||
|
||||
class FinalAccountingPass:
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from .ocr_figure_vnext_types import ClaimProposal, PassReport, ResourceRef
|
||||
from .ocr_pairing_types import ClaimProposal, PassReport, ResourceRef
|
||||
|
||||
_NON_PURE_ROLES = frozenset({
|
||||
"body_paragraph",
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from .ocr_figure_vnext_types import ClaimProposal, PassReport, ResourceRef
|
||||
from .ocr_pairing_types import ClaimProposal, PassReport, ResourceRef
|
||||
|
||||
|
||||
def _resource_page(block: dict) -> int | None:
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from .ocr_figure_vnext_types import ClaimProposal, PassReport, ResourceRef
|
||||
from .ocr_pairing_types import ClaimProposal, PassReport, ResourceRef
|
||||
|
||||
|
||||
def _resource_page(block: dict) -> int | None:
|
||||
|
|
|
|||
|
|
@ -1,127 +1,3 @@
|
|||
from __future__ import annotations
|
||||
from .ocr_figure_domain import FigureCandidateIndex, FigureCorpus
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass
|
||||
class FigureCorpus:
|
||||
"""Immutable facts extracted from structured blocks.
|
||||
|
||||
This dataclass stores raw, unfiltered data plus computed layout profiles.
|
||||
No derived hypotheses or candidate logic lives here.
|
||||
"""
|
||||
|
||||
blocks: list[dict]
|
||||
page_width: float
|
||||
raw_legends: list[dict] = field(default_factory=list)
|
||||
raw_assets: list[dict] = field(default_factory=list)
|
||||
locator_candidates: list[dict] = field(default_factory=list)
|
||||
page_layouts: dict[int, Any] = field(default_factory=dict)
|
||||
|
||||
@classmethod
|
||||
def from_blocks(cls, blocks: list[dict], page_width: float = 1200) -> FigureCorpus:
|
||||
from paperforge.worker.ocr_document import _build_page_layout_profiles
|
||||
|
||||
from . import ocr_figures
|
||||
|
||||
raw_legends = [
|
||||
b for b in blocks if b.get("role") in {"figure_caption", "figure_caption_candidate"}
|
||||
]
|
||||
raw_assets = [
|
||||
b for b in blocks if b.get("role") in {"figure_asset", "media_asset"}
|
||||
]
|
||||
locator_candidates = [
|
||||
b for b in raw_legends if ocr_figures._is_previous_page_legend_locator(b)
|
||||
]
|
||||
return cls(
|
||||
blocks=list(blocks),
|
||||
page_width=page_width,
|
||||
raw_legends=raw_legends,
|
||||
raw_assets=raw_assets,
|
||||
locator_candidates=locator_candidates,
|
||||
page_layouts=_build_page_layout_profiles(blocks),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class FigureCandidateIndex:
|
||||
"""Derived hypotheses and candidates built from a FigureCorpus.
|
||||
|
||||
Every field here is a computed / filtered view of the raw corpus facts.
|
||||
No raw data is duplicated — references point back into the corpus.
|
||||
"""
|
||||
|
||||
formal_legends: list[dict]
|
||||
held_legends: list[dict]
|
||||
rejected_legends: list[dict]
|
||||
deduped_legends: list[dict]
|
||||
candidate_groups: list[dict]
|
||||
competing_caption_pages: set[int]
|
||||
sidecar_candidates: dict[int, list[dict]]
|
||||
bundle_source_legend_ids: set[str]
|
||||
locator_candidates: list[dict]
|
||||
composite_parent_candidates: list[dict] = field(default_factory=list)
|
||||
|
||||
@classmethod
|
||||
def from_corpus(cls, corpus: FigureCorpus) -> FigureCandidateIndex:
|
||||
from . import ocr_figures
|
||||
|
||||
formal_legends = [
|
||||
b
|
||||
for b in corpus.raw_legends
|
||||
if ocr_figures._is_formal_legend(
|
||||
str(b.get("text", "")), b, corpus.page_width
|
||||
)
|
||||
]
|
||||
rejected_legends = [b for b in corpus.raw_legends if b not in formal_legends]
|
||||
candidate_groups = ocr_figures._build_candidate_figure_groups_from_assets(
|
||||
corpus.raw_assets,
|
||||
corpus.blocks,
|
||||
formal_legends,
|
||||
page_width=corpus.page_width,
|
||||
)
|
||||
competing_caption_pages = {
|
||||
int(leg.get("page"))
|
||||
for leg in formal_legends
|
||||
if leg.get("page") is not None
|
||||
and ocr_figures._extract_figure_number(str(leg.get("text", "")))
|
||||
is not None
|
||||
}
|
||||
# Populate sidecar_candidates: pages with ≥2 aligned narrow captions.
|
||||
sidecar_candidates: dict[int, list[dict]] = {}
|
||||
_legends_by_page: dict[int, list[dict]] = {}
|
||||
for leg in formal_legends:
|
||||
_legends_by_page.setdefault(int(leg.get("page", 0) or 0), []).append(leg)
|
||||
for sp, spl in _legends_by_page.items():
|
||||
narrow_set = ocr_figures._same_page_narrow_caption_column(spl, corpus.page_width)
|
||||
if len(narrow_set) >= 2:
|
||||
sidecar_candidates[sp] = narrow_set
|
||||
|
||||
# Populate bundle_source_legend_ids: legends on pages with ≥3 numbered legends
|
||||
# and zero assets on that page.
|
||||
bundle_source_legend_ids = ocr_figures._identify_bundle_source_legend_ids(
|
||||
formal_legends, corpus.raw_assets
|
||||
)
|
||||
|
||||
|
||||
# Populate composite_parent_candidates: same-page atomic groups with
|
||||
# horizontal alignment + vertical adjacency (2x2 grid, columnar stacks).
|
||||
composite_parent_candidates = ocr_figures._build_composite_parent_figure_groups_visual_only(
|
||||
candidate_groups,
|
||||
corpus.raw_assets,
|
||||
corpus.blocks,
|
||||
corpus.page_width,
|
||||
)
|
||||
return cls(
|
||||
formal_legends=formal_legends,
|
||||
held_legends=[],
|
||||
rejected_legends=rejected_legends,
|
||||
deduped_legends=formal_legends,
|
||||
candidate_groups=candidate_groups,
|
||||
competing_caption_pages=competing_caption_pages,
|
||||
sidecar_candidates=sidecar_candidates,
|
||||
bundle_source_legend_ids=bundle_source_legend_ids,
|
||||
locator_candidates=corpus.locator_candidates,
|
||||
composite_parent_candidates=composite_parent_candidates,
|
||||
)
|
||||
__all__ = ["FigureCorpus", "FigureCandidateIndex"]
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ next-page, then previous-page (with guard) order.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
from .ocr_figure_vnext_types import ClaimProposal, PassReport, ResourceRef
|
||||
from .ocr_pairing_types import ClaimProposal, PassReport, ResourceRef
|
||||
|
||||
|
||||
def _resource_page(block: dict) -> int | None:
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ Run AFTER PrimarySamePagePass and cross-page passes but BEFORE generic fallback.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
from .ocr_figure_vnext_types import ClaimProposal, PassReport, ResourceRef
|
||||
from .ocr_pairing_types import ClaimProposal, PassReport, ResourceRef
|
||||
|
||||
|
||||
def _resource_page(block: dict) -> int | None:
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from .ocr_figure_vnext_types import ClaimProposal, PassReport, ResourceRef
|
||||
from .ocr_pairing_types import ClaimProposal, PassReport, ResourceRef
|
||||
|
||||
|
||||
def _resource_page(block: dict) -> int | None:
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from .ocr_figure_vnext_types import ClaimProposal, PassReport, ResourceRef
|
||||
from .ocr_pairing_types import ClaimProposal, PassReport, ResourceRef
|
||||
|
||||
|
||||
def _resource_page(block: dict) -> int | None:
|
||||
|
|
|
|||
|
|
@ -1,145 +1,3 @@
|
|||
from __future__ import annotations
|
||||
from .ocr_pairing_state import FigurePipelineState, OwnershipLedger
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from .ocr_figure_vnext_types import ClaimProposal, OwnershipConflict, ResourceRef
|
||||
|
||||
|
||||
class OwnershipLedger:
|
||||
def __init__(self) -> None:
|
||||
self._reserved_groups: set[ResourceRef] = set()
|
||||
self._owners: dict[ResourceRef, ResourceRef] = {}
|
||||
self._journal: list[dict[str, object]] = []
|
||||
|
||||
def claim_assets(self, assets: list[ResourceRef], *, owner: ResourceRef, reason: str) -> None:
|
||||
conflict = self.try_claim_assets(assets, owner=owner, reason=reason)
|
||||
if conflict is not None:
|
||||
raise ValueError(f"asset already owned: {conflict.resource}")
|
||||
|
||||
def try_claim_assets(
|
||||
self, assets: list[ResourceRef], *, owner: ResourceRef, reason: str
|
||||
) -> OwnershipConflict | None:
|
||||
for asset in assets:
|
||||
current = self._owners.get(asset)
|
||||
if current is not None and current != owner:
|
||||
conflict = OwnershipConflict(
|
||||
resource=asset,
|
||||
current_owner=current,
|
||||
attempted_owner=owner,
|
||||
reason=reason,
|
||||
)
|
||||
self._journal.append({
|
||||
"action": "conflict",
|
||||
"resource": asset,
|
||||
"current_owner": current,
|
||||
"attempted_owner": owner,
|
||||
"reason": reason,
|
||||
})
|
||||
return conflict
|
||||
for asset in assets:
|
||||
self._owners[asset] = owner
|
||||
self._journal.append({"action": "claim", "resource": asset, "owner": owner, "reason": reason})
|
||||
return None
|
||||
|
||||
def owner_of(self, resource: ResourceRef) -> ResourceRef | None:
|
||||
return self._owners.get(resource)
|
||||
|
||||
def owner_of_asset(self, *, page: int, block_id: int | str) -> ResourceRef | None:
|
||||
return self.owner_of(ResourceRef(kind="asset", page=page, block_id=block_id))
|
||||
|
||||
def reserve_group(self, group: ResourceRef, *, reason: str) -> None:
|
||||
self._reserved_groups.add(group)
|
||||
self._journal.append({"action": "reserve_group", "resource": group, "reason": reason})
|
||||
|
||||
def can_claim_group(self, group: ResourceRef) -> bool:
|
||||
return group not in self._reserved_groups
|
||||
|
||||
def transition_reserved_group_to_claimed(
|
||||
self, group: ResourceRef, *, owner: ResourceRef, reason: str
|
||||
) -> None:
|
||||
self._reserved_groups.discard(group)
|
||||
self._journal.append({
|
||||
"action": "transition_to_claimed",
|
||||
"resource": group,
|
||||
"owner": owner,
|
||||
"reason": reason,
|
||||
})
|
||||
|
||||
def snapshot(self) -> list[dict[str, object]]:
|
||||
return list(self._journal)
|
||||
|
||||
|
||||
@dataclass
|
||||
class FigurePipelineState:
|
||||
corpus: object | None
|
||||
candidate_index: object | None
|
||||
ledger: OwnershipLedger
|
||||
matches: list[dict] = field(default_factory=list)
|
||||
unresolved: list[dict] = field(default_factory=list)
|
||||
hypotheses: list[dict] = field(default_factory=list)
|
||||
diagnostics: list[dict] = field(default_factory=list)
|
||||
reservations: list[dict] = field(default_factory=list)
|
||||
completeness: dict = field(default_factory=dict)
|
||||
|
||||
def accept_match(self, proposal: ClaimProposal, match_record: dict) -> None:
|
||||
# Enrich match_record with rotation metadata if the legend has rotated text.
|
||||
self._enrich_rotation(proposal, match_record)
|
||||
self.matches.append(match_record)
|
||||
self.diagnostics.append({
|
||||
"event": "match_accepted",
|
||||
"pass_name": proposal.pass_name,
|
||||
"figure_no": proposal.figure_no,
|
||||
"reason": proposal.reason,
|
||||
"resources": {
|
||||
"legends": proposal.legends,
|
||||
"assets": proposal.assets,
|
||||
"groups": proposal.groups,
|
||||
},
|
||||
})
|
||||
|
||||
def _enrich_rotation(self, proposal: ClaimProposal, match_record: dict) -> None:
|
||||
"""Detect rotated caption and add rotation_correction_deg + cluster_bbox."""
|
||||
from .ocr_figures import _caption_has_rotated_text, _prepare_rotated_caption_normalization
|
||||
|
||||
if not self.corpus or not proposal.legends:
|
||||
return
|
||||
legend_ref = proposal.legends[0]
|
||||
# Find the legend block in the corpus
|
||||
legend_block = None
|
||||
for b in self.corpus.blocks:
|
||||
if str(b.get("block_id", "")) == str(legend_ref.block_id) and b.get("page") == legend_ref.page:
|
||||
legend_block = b
|
||||
break
|
||||
if legend_block is None or not _caption_has_rotated_text(legend_block):
|
||||
return
|
||||
# Find the first asset block from the proposal
|
||||
if not proposal.assets or not self.corpus.raw_assets:
|
||||
return
|
||||
asset_ref = proposal.assets[0]
|
||||
asset_block = None
|
||||
for a in self.corpus.raw_assets:
|
||||
if str(a.get("block_id", "")) == str(asset_ref.block_id) and a.get("page") == asset_ref.page:
|
||||
asset_block = a
|
||||
break
|
||||
if asset_block is None:
|
||||
return
|
||||
normalized = _prepare_rotated_caption_normalization(legend_block, asset_block)
|
||||
if normalized is not None:
|
||||
match_record["rotation_correction_deg"] = normalized["rotation_correction_deg"]
|
||||
match_record["cluster_bbox"] = normalized["rotation_union_bbox"]
|
||||
|
||||
def accept_reservation(self, proposal: ClaimProposal) -> None:
|
||||
self.reservations.append({
|
||||
"pass_name": proposal.pass_name,
|
||||
"figure_no": proposal.figure_no,
|
||||
"reason": proposal.reason,
|
||||
"legends": list(proposal.legends),
|
||||
"groups": list(proposal.groups),
|
||||
})
|
||||
self.diagnostics.append({
|
||||
"event": "reservation_accepted",
|
||||
"pass_name": proposal.pass_name,
|
||||
"figure_no": proposal.figure_no,
|
||||
"reason": proposal.reason,
|
||||
"resources": {"groups": proposal.groups},
|
||||
})
|
||||
__all__ = ["OwnershipLedger", "FigurePipelineState"]
|
||||
|
|
|
|||
|
|
@ -1,61 +1,3 @@
|
|||
from __future__ import annotations
|
||||
from .ocr_pairing_types import ClaimProposal, OwnershipConflict, PassReport, ResourceRef
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Literal
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResourceRef:
|
||||
kind: Literal["legend", "asset", "group"]
|
||||
page: int | None
|
||||
block_id: str | None
|
||||
group_id: str | None = None
|
||||
figure_no: int | None = field(default=None, compare=False)
|
||||
origin: str | None = field(default=None, compare=False)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.page is not None:
|
||||
object.__setattr__(self, "page", int(self.page))
|
||||
if self.block_id is not None:
|
||||
object.__setattr__(self, "block_id", str(self.block_id))
|
||||
if self.group_id is not None:
|
||||
object.__setattr__(self, "group_id", str(self.group_id))
|
||||
|
||||
if self.kind == "asset" and (self.page is None or self.block_id is None):
|
||||
raise ValueError("asset ResourceRef requires page + block_id")
|
||||
if self.kind == "legend" and (self.page is None or self.block_id is None):
|
||||
raise ValueError("legend ResourceRef requires page + block_id")
|
||||
if self.kind == "group" and (self.page is None or self.group_id is None):
|
||||
raise ValueError("group ResourceRef requires page + group_id")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class OwnershipConflict:
|
||||
resource: ResourceRef
|
||||
current_owner: ResourceRef | None
|
||||
attempted_owner: ResourceRef | None
|
||||
reason: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class ClaimProposal:
|
||||
pass_name: str
|
||||
figure_no: int | None
|
||||
claim_type: Literal["match", "reserve", "block", "unresolved_cluster", "composite_parent"]
|
||||
legends: list[ResourceRef]
|
||||
assets: list[ResourceRef]
|
||||
groups: list[ResourceRef]
|
||||
confidence: float
|
||||
evidence_rank: int
|
||||
reason: str
|
||||
diagnostics: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class PassReport:
|
||||
pass_name: str
|
||||
proposals: list[ClaimProposal] = field(default_factory=list)
|
||||
accepted: list[ClaimProposal] = field(default_factory=list)
|
||||
rejected: list[ClaimProposal] = field(default_factory=list)
|
||||
conflicts: list[OwnershipConflict] = field(default_factory=list)
|
||||
invariant_errors: list[str] = field(default_factory=list)
|
||||
__all__ = ["ResourceRef", "OwnershipConflict", "ClaimProposal", "PassReport"]
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
10
paperforge/worker/ocr_pairing_framework.py
Normal file
10
paperforge/worker/ocr_pairing_framework.py
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
|
||||
def run_pairing_passes(state, pass_classes: Sequence[type]) -> list:
|
||||
reports = []
|
||||
for pass_cls in pass_classes:
|
||||
reports.append(pass_cls().run(state))
|
||||
return reports
|
||||
146
paperforge/worker/ocr_pairing_state.py
Normal file
146
paperforge/worker/ocr_pairing_state.py
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from .ocr_pairing_types import ClaimProposal, OwnershipConflict, ResourceRef
|
||||
|
||||
|
||||
class OwnershipLedger:
|
||||
def __init__(self) -> None:
|
||||
self._reserved_groups: set[ResourceRef] = set()
|
||||
self._owners: dict[ResourceRef, ResourceRef] = {}
|
||||
self._journal: list[dict[str, object]] = []
|
||||
|
||||
def claim_assets(self, assets: list[ResourceRef], *, owner: ResourceRef, reason: str) -> None:
|
||||
conflict = self.try_claim_assets(assets, owner=owner, reason=reason)
|
||||
if conflict is not None:
|
||||
raise ValueError(f"asset already owned: {conflict.resource}")
|
||||
|
||||
def try_claim_assets(
|
||||
self, assets: list[ResourceRef], *, owner: ResourceRef, reason: str
|
||||
) -> OwnershipConflict | None:
|
||||
for asset in assets:
|
||||
current = self._owners.get(asset)
|
||||
if current is not None and current != owner:
|
||||
conflict = OwnershipConflict(
|
||||
resource=asset,
|
||||
current_owner=current,
|
||||
attempted_owner=owner,
|
||||
reason=reason,
|
||||
)
|
||||
self._journal.append(
|
||||
{
|
||||
"action": "conflict",
|
||||
"resource": asset,
|
||||
"current_owner": current,
|
||||
"attempted_owner": owner,
|
||||
"reason": reason,
|
||||
}
|
||||
)
|
||||
return conflict
|
||||
for asset in assets:
|
||||
self._owners[asset] = owner
|
||||
self._journal.append({"action": "claim", "resource": asset, "owner": owner, "reason": reason})
|
||||
return None
|
||||
|
||||
def owner_of(self, resource: ResourceRef) -> ResourceRef | None:
|
||||
return self._owners.get(resource)
|
||||
|
||||
def owner_of_asset(self, *, page: int, block_id: int | str) -> ResourceRef | None:
|
||||
return self.owner_of(ResourceRef(kind="asset", page=page, block_id=block_id))
|
||||
|
||||
def reserve_group(self, group: ResourceRef, *, reason: str) -> None:
|
||||
self._reserved_groups.add(group)
|
||||
self._journal.append({"action": "reserve_group", "resource": group, "reason": reason})
|
||||
|
||||
def can_claim_group(self, group: ResourceRef) -> bool:
|
||||
return group not in self._reserved_groups
|
||||
|
||||
def transition_reserved_group_to_claimed(self, group: ResourceRef, *, owner: ResourceRef, reason: str) -> None:
|
||||
self._reserved_groups.discard(group)
|
||||
self._journal.append(
|
||||
{
|
||||
"action": "transition_to_claimed",
|
||||
"resource": group,
|
||||
"owner": owner,
|
||||
"reason": reason,
|
||||
}
|
||||
)
|
||||
|
||||
def snapshot(self) -> list[dict[str, object]]:
|
||||
return list(self._journal)
|
||||
|
||||
def journal_text_attachment(self, texts: list[ResourceRef], *, owner: ResourceRef, reason: str) -> None:
|
||||
for text in texts:
|
||||
self._journal.append(
|
||||
{
|
||||
"action": "attach_text",
|
||||
"resource": text,
|
||||
"owner": owner,
|
||||
"reason": reason,
|
||||
}
|
||||
)
|
||||
|
||||
def text_attachments_for(self, owner: ResourceRef) -> list[ResourceRef]:
|
||||
return [
|
||||
entry["resource"]
|
||||
for entry in self._journal
|
||||
if entry.get("action") == "attach_text" and entry.get("owner") == owner
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class FigurePipelineState:
|
||||
corpus: object | None
|
||||
candidate_index: object | None
|
||||
ledger: OwnershipLedger
|
||||
matches: list[dict] = field(default_factory=list)
|
||||
unresolved: list[dict] = field(default_factory=list)
|
||||
hypotheses: list[dict] = field(default_factory=list)
|
||||
diagnostics: list[dict] = field(default_factory=list)
|
||||
reservations: list[dict] = field(default_factory=list)
|
||||
completeness: dict = field(default_factory=dict)
|
||||
# Hook set by domain code (e.g., figures) to enrich match_records before storing.
|
||||
_match_pre_enricher: Callable | None = None
|
||||
|
||||
def accept_match(self, proposal: ClaimProposal, match_record: dict) -> None:
|
||||
if self._match_pre_enricher is not None:
|
||||
self._match_pre_enricher(proposal, match_record, self)
|
||||
self.matches.append(match_record)
|
||||
self.diagnostics.append(
|
||||
{
|
||||
"event": "match_accepted",
|
||||
"pass_name": proposal.pass_name,
|
||||
"figure_no": proposal.figure_no,
|
||||
"reason": proposal.reason,
|
||||
"resources": {
|
||||
"legends": proposal.legends,
|
||||
"assets": proposal.assets,
|
||||
"groups": proposal.groups,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
def accept_reservation(self, proposal: ClaimProposal) -> None:
|
||||
self.reservations.append(
|
||||
{
|
||||
"pass_name": proposal.pass_name,
|
||||
"figure_no": proposal.figure_no,
|
||||
"reason": proposal.reason,
|
||||
"legends": list(proposal.legends),
|
||||
"groups": list(proposal.groups),
|
||||
}
|
||||
)
|
||||
self.diagnostics.append(
|
||||
{
|
||||
"event": "reservation_accepted",
|
||||
"pass_name": proposal.pass_name,
|
||||
"figure_no": proposal.figure_no,
|
||||
"reason": proposal.reason,
|
||||
"resources": {"groups": proposal.groups},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
PipelineState = FigurePipelineState
|
||||
71
paperforge/worker/ocr_pairing_types.py
Normal file
71
paperforge/worker/ocr_pairing_types.py
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Literal
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResourceRef:
|
||||
kind: Literal["legend", "asset", "group", "text"]
|
||||
page: int | None
|
||||
block_id: str | None
|
||||
group_id: str | None = None
|
||||
figure_no: int | None = field(default=None, compare=False)
|
||||
origin: str | None = field(default=None, compare=False)
|
||||
role: str | None = field(default=None, compare=False)
|
||||
@property
|
||||
def entity_no(self) -> int | None:
|
||||
return self.figure_no
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.page is not None:
|
||||
object.__setattr__(self, "page", int(self.page))
|
||||
if self.block_id is not None:
|
||||
object.__setattr__(self, "block_id", str(self.block_id))
|
||||
if self.group_id is not None:
|
||||
object.__setattr__(self, "group_id", str(self.group_id))
|
||||
|
||||
if self.kind == "asset" and (self.page is None or self.block_id is None):
|
||||
raise ValueError("asset ResourceRef requires page + block_id")
|
||||
if self.kind == "legend" and (self.page is None or self.block_id is None):
|
||||
raise ValueError("legend ResourceRef requires page + block_id")
|
||||
if self.kind == "group" and (self.page is None or self.group_id is None):
|
||||
raise ValueError("group ResourceRef requires page + group_id")
|
||||
if self.kind == "text" and (self.page is None or self.block_id is None):
|
||||
raise ValueError("text ResourceRef requires page + block_id")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class OwnershipConflict:
|
||||
resource: ResourceRef
|
||||
current_owner: ResourceRef | None
|
||||
attempted_owner: ResourceRef | None
|
||||
reason: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class ClaimProposal:
|
||||
pass_name: str
|
||||
figure_no: int | None
|
||||
claim_type: Literal["match", "reserve", "block", "unresolved_cluster", "composite_parent", "attach_text"]
|
||||
legends: list[ResourceRef]
|
||||
assets: list[ResourceRef]
|
||||
groups: list[ResourceRef]
|
||||
confidence: float
|
||||
evidence_rank: int
|
||||
reason: str
|
||||
diagnostics: dict[str, Any] = field(default_factory=dict)
|
||||
texts: list[ResourceRef] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def entity_no(self) -> int | None:
|
||||
return self.figure_no
|
||||
|
||||
@dataclass
|
||||
class PassReport:
|
||||
pass_name: str
|
||||
proposals: list[ClaimProposal] = field(default_factory=list)
|
||||
accepted: list[ClaimProposal] = field(default_factory=list)
|
||||
rejected: list[ClaimProposal] = field(default_factory=list)
|
||||
conflicts: list[OwnershipConflict] = field(default_factory=list)
|
||||
invariant_errors: list[str] = field(default_factory=list)
|
||||
177
paperforge/worker/ocr_table_domain.py
Normal file
177
paperforge/worker/ocr_table_domain.py
Normal file
|
|
@ -0,0 +1,177 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass
|
||||
class TableCorpus:
|
||||
blocks: list[dict]
|
||||
raw_captions: list[dict] = field(default_factory=list)
|
||||
raw_assets: list[dict] = field(default_factory=list)
|
||||
page_footnote_prior: dict[int, float] = field(default_factory=dict)
|
||||
page_max_y: dict[int, float] = field(default_factory=dict)
|
||||
|
||||
@classmethod
|
||||
def from_blocks(cls, blocks: list[dict]) -> TableCorpus:
|
||||
from . import ocr_tables
|
||||
|
||||
raw_captions = [
|
||||
b
|
||||
for b in blocks
|
||||
if b.get("role") in {"table_caption", "table_caption_candidate"}
|
||||
or ocr_tables._is_validation_first_table_candidate(b)
|
||||
]
|
||||
raw_assets = []
|
||||
for block in blocks:
|
||||
role = block.get("role", "")
|
||||
raw_label = str(block.get("raw_label", "") or "").strip()
|
||||
if role not in ("table_asset", "table_html", "media_asset", "figure_asset"):
|
||||
continue
|
||||
if role == "figure_asset" and raw_label != "table":
|
||||
continue
|
||||
raw_assets.append(block)
|
||||
|
||||
page_footnote_prior = ocr_tables._collect_page_footnote_prior(blocks)
|
||||
page_max_y: dict[int, float] = {}
|
||||
for block in blocks:
|
||||
bbox = block.get("bbox") or [0, 0, 0, 0]
|
||||
if len(bbox) >= 4:
|
||||
page = int(block.get("page", 0) or 0)
|
||||
page_max_y[page] = max(page_max_y.get(page, 0.0), float(bbox[3]))
|
||||
|
||||
return cls(
|
||||
blocks=blocks,
|
||||
raw_captions=raw_captions,
|
||||
raw_assets=raw_assets,
|
||||
page_footnote_prior=page_footnote_prior,
|
||||
page_max_y=page_max_y,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class TableCandidateIndex:
|
||||
caption_records: list[dict]
|
||||
assets_by_page: dict[int, list[tuple[int, dict]]]
|
||||
|
||||
@classmethod
|
||||
def from_corpus(cls, corpus: TableCorpus) -> TableCandidateIndex:
|
||||
from . import ocr_tables
|
||||
|
||||
captions = sorted(
|
||||
corpus.raw_captions,
|
||||
key=lambda block: (
|
||||
int(block.get("page", 0) or 0),
|
||||
str(block.get("block_id", "")),
|
||||
),
|
||||
)
|
||||
caption_records = []
|
||||
for caption in captions:
|
||||
text = str(caption.get("text", "") or "")
|
||||
caption_records.append(
|
||||
{
|
||||
"caption": caption,
|
||||
"caption_block_id": str(caption.get("block_id", "")),
|
||||
"caption_text": text,
|
||||
"table_number": ocr_tables._extract_table_number(text),
|
||||
"formal_table_number": ocr_tables._extract_base_table_number(text),
|
||||
"is_continuation": ocr_tables._is_continuation_caption(text),
|
||||
"is_validation_first_candidate": ocr_tables._is_validation_first_table_candidate(
|
||||
caption
|
||||
),
|
||||
"is_weak_truncated": ocr_tables._is_insufficient_table_caption_evidence(
|
||||
caption
|
||||
),
|
||||
"is_weak_explicit_caption": ocr_tables._is_weak_explicit_table_caption(
|
||||
caption
|
||||
),
|
||||
"continuation_ids": [],
|
||||
"status": "pending",
|
||||
"candidate_assets": [],
|
||||
}
|
||||
)
|
||||
|
||||
assets_by_page: dict[int, list[tuple[int, dict]]] = {}
|
||||
for idx, asset in enumerate(corpus.raw_assets):
|
||||
page = int(asset.get("page", 0) or 0)
|
||||
assets_by_page.setdefault(page, []).append((idx, asset))
|
||||
|
||||
return cls(caption_records=caption_records, assets_by_page=assets_by_page)
|
||||
|
||||
|
||||
def assemble_table_inventory(
|
||||
state, candidate_index: TableCandidateIndex
|
||||
) -> dict[str, Any]:
|
||||
matched_caption_ids = {
|
||||
t.get("caption_block_id", "")
|
||||
for t in state.matches
|
||||
if t.get("has_asset")
|
||||
}
|
||||
used_asset_ids = {
|
||||
str(t.get("asset_block_id", ""))
|
||||
for t in state.matches
|
||||
if t.get("has_asset") and t.get("asset_block_id")
|
||||
}
|
||||
held_tables = [
|
||||
record["held_table"]
|
||||
for record in candidate_index.caption_records
|
||||
if record.get("status") == "held" and "held_table" in record
|
||||
]
|
||||
unmatched_captions = [
|
||||
record["caption"]
|
||||
for record in candidate_index.caption_records
|
||||
if record["caption_block_id"] not in matched_caption_ids
|
||||
and record.get("status") != "held"
|
||||
]
|
||||
# Create has_asset=False table entries for unmatched captions (legacy-compat)
|
||||
unmatched_table_entries: list[dict] = []
|
||||
for record in candidate_index.caption_records:
|
||||
if record["caption_block_id"] in matched_caption_ids or record.get("status") == "held":
|
||||
continue
|
||||
caption = record["caption"]
|
||||
unmatched_table_entries.append(
|
||||
{
|
||||
"caption_block_id": record["caption_block_id"],
|
||||
"page": caption.get("page", 0),
|
||||
"caption_text": record["caption_text"],
|
||||
"table_number": record["table_number"],
|
||||
"formal_table_number": record["formal_table_number"],
|
||||
"asset_block_id": None,
|
||||
"asset_bbox": [],
|
||||
"assistive_text": "",
|
||||
"truth_source": "image",
|
||||
"has_asset": False,
|
||||
"segments": [],
|
||||
"note_block_ids": [],
|
||||
"note_texts": [],
|
||||
"note_bboxes": [],
|
||||
"note_band_bbox": [],
|
||||
"note_match_reason": "",
|
||||
"note_confidence": 0.0,
|
||||
"bridge_block_ids": [],
|
||||
"consumed_block_ids": [],
|
||||
"is_continuation": record["is_continuation"],
|
||||
"continuation_of": None,
|
||||
"match_score": {"decision": "unmatched", "evidence": [], "matched_asset_id": "", "score": 0.0},
|
||||
"match_status": record.get("status", "unmatched_caption"),
|
||||
"candidate_assets": record["candidate_assets"],
|
||||
"render_bbox": None,
|
||||
"render_rotation_deg": 0,
|
||||
}
|
||||
)
|
||||
|
||||
unmatched_assets = [
|
||||
asset
|
||||
for page_assets in candidate_index.assets_by_page.values()
|
||||
for _, asset in page_assets
|
||||
if str(asset.get("block_id", "")) not in used_asset_ids
|
||||
]
|
||||
return {
|
||||
"tables": list(state.matches) + unmatched_table_entries,
|
||||
"held_tables": held_tables,
|
||||
"unmatched_captions": unmatched_captions,
|
||||
"unmatched_assets": unmatched_assets,
|
||||
"official_table_count": len(
|
||||
[t for t in state.matches if t.get("has_asset") and not t.get("is_continuation")]
|
||||
),
|
||||
}
|
||||
387
paperforge/worker/ocr_table_passes.py
Normal file
387
paperforge/worker/ocr_table_passes.py
Normal file
|
|
@ -0,0 +1,387 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from .ocr_pairing_types import ClaimProposal, PassReport, ResourceRef
|
||||
|
||||
|
||||
class TableWeakCaptionRecoveryPass:
|
||||
name = "table_weak_caption_recovery"
|
||||
|
||||
def run(self, state):
|
||||
from . import ocr_tables
|
||||
|
||||
report = PassReport(pass_name=self.name)
|
||||
for record in state.candidate_index.caption_records:
|
||||
if not record["is_weak_truncated"]:
|
||||
continue
|
||||
caption = record["caption"]
|
||||
continuation = ocr_tables._find_table_caption_continuation(caption, state.corpus.blocks)
|
||||
materialized, continuation_ids = ocr_tables._materialize_table_caption(caption, continuation)
|
||||
record["caption"] = materialized
|
||||
record["caption_text"] = str(materialized.get("text", "") or "")
|
||||
record["continuation_ids"] = continuation_ids
|
||||
if record["is_validation_first_candidate"]:
|
||||
same_page_assets = state.candidate_index.assets_by_page.get(int(caption.get("page", 0) or 0), [])
|
||||
if not same_page_assets:
|
||||
record["status"] = "held"
|
||||
record["held_table"] = {
|
||||
"table_id": f"held_table_{len([r for r in state.candidate_index.caption_records if r.get('status') == 'held']) + 1:03d}",
|
||||
"caption_block_id": record["caption_block_id"],
|
||||
"page": caption.get("page", 0),
|
||||
"caption_text": record["caption_text"],
|
||||
"table_number": record["table_number"],
|
||||
"formal_table_number": record["formal_table_number"],
|
||||
"hold_reason": "insufficient_caption_evidence",
|
||||
"zone": caption.get("zone", ""),
|
||||
"style_family": caption.get("style_family", ""),
|
||||
"marker_signature": caption.get("marker_signature", {}),
|
||||
}
|
||||
return report
|
||||
|
||||
|
||||
class TableSamePagePass:
|
||||
name = "table_same_page"
|
||||
|
||||
def run(self, state):
|
||||
from . import ocr_tables
|
||||
|
||||
report = PassReport(pass_name=self.name)
|
||||
for record in state.candidate_index.caption_records:
|
||||
if record.get("status") != "pending":
|
||||
continue
|
||||
caption = record["caption"]
|
||||
caption_page = int(caption.get("page", 0) or 0)
|
||||
page_assets = [
|
||||
(idx, asset)
|
||||
for idx, asset in state.candidate_index.assets_by_page.get(caption_page, [])
|
||||
if state.ledger.owner_of_asset(page=caption_page, block_id=asset.get("block_id")) is None
|
||||
]
|
||||
scored = ocr_tables._score_candidate_assets(page_assets, caption, is_continuation=record["is_continuation"])
|
||||
scored.sort(key=lambda item: item[2].get("score", 0.0), reverse=True)
|
||||
record["candidate_assets"] = [
|
||||
{"asset_block_id": asset.get("block_id", ""), "match_score": score}
|
||||
for _, asset, score in scored[:3]
|
||||
]
|
||||
if not scored:
|
||||
continue
|
||||
top_idx, top_asset, top_score = scored[0]
|
||||
second_score = scored[1][2].get("score", 0.0) if len(scored) > 1 else -1.0
|
||||
top_score_val = top_score.get("score", 0.0)
|
||||
if top_score_val < 0.4:
|
||||
continue
|
||||
if top_score_val - second_score < 0.15:
|
||||
record["status"] = "ambiguous"
|
||||
continue
|
||||
owner = ResourceRef(kind="legend", page=caption_page, block_id=record["caption_block_id"], figure_no=record["formal_table_number"])
|
||||
asset_ref = ResourceRef(kind="asset", page=int(top_asset.get("page", 0) or 0), block_id=top_asset.get("block_id"))
|
||||
conflict = state.ledger.try_claim_assets([asset_ref], owner=owner, reason=self.name)
|
||||
if conflict is not None:
|
||||
report.conflicts.append(conflict)
|
||||
continue
|
||||
match_status = "matched" if top_score.get("score", 0.0) >= 0.6 else "matched_low_confidence"
|
||||
state.accept_match(
|
||||
ClaimProposal(
|
||||
pass_name=self.name,
|
||||
figure_no=record["formal_table_number"],
|
||||
claim_type="match",
|
||||
legends=[owner],
|
||||
assets=[asset_ref],
|
||||
groups=[],
|
||||
texts=[],
|
||||
confidence=float(top_score.get("score", 0.0)),
|
||||
evidence_rank=0,
|
||||
reason=self.name,
|
||||
),
|
||||
{
|
||||
"caption_block_id": record["caption_block_id"],
|
||||
"page": caption_page,
|
||||
"caption_text": record["caption_text"],
|
||||
"table_number": record["table_number"],
|
||||
"formal_table_number": record["formal_table_number"],
|
||||
"asset_block_id": top_asset.get("block_id", ""),
|
||||
"asset_bbox": top_asset.get("bbox", [0, 0, 0, 0]),
|
||||
"assistive_text": str(top_asset.get("text", "") or ""),
|
||||
"truth_source": "image",
|
||||
"has_asset": True,
|
||||
"segments": [
|
||||
{
|
||||
"page": top_asset.get("page", 0),
|
||||
"asset_block_id": top_asset.get("block_id", ""),
|
||||
"asset_bbox": top_asset.get("bbox", [0, 0, 0, 0]),
|
||||
"is_continuation": record["is_continuation"],
|
||||
}
|
||||
],
|
||||
"note_block_ids": [],
|
||||
"note_texts": [],
|
||||
"note_bboxes": [],
|
||||
"note_band_bbox": [],
|
||||
"note_match_reason": "",
|
||||
"note_confidence": 0.0,
|
||||
"bridge_block_ids": [],
|
||||
"consumed_block_ids": [record["caption_block_id"], top_asset.get("block_id", ""), *record.get("continuation_ids", [])],
|
||||
"is_continuation": record["is_continuation"],
|
||||
"continuation_of": None,
|
||||
"match_status": match_status,
|
||||
"candidate_assets": record["candidate_assets"],
|
||||
"match_score": top_score,
|
||||
"render_bbox": None,
|
||||
"render_rotation_deg": 0,
|
||||
"asset_family_hint": top_asset.get("asset_family_hint"),
|
||||
"asset_family_confidence": top_asset.get("asset_family_confidence"),
|
||||
"asset_family_evidence": top_asset.get("asset_family_evidence"),
|
||||
},
|
||||
)
|
||||
record["status"] = "matched"
|
||||
return report
|
||||
|
||||
|
||||
class TableAdjacentPagePass:
|
||||
name = "table_adjacent_page"
|
||||
|
||||
def run(self, state):
|
||||
from . import ocr_tables
|
||||
|
||||
report = PassReport(pass_name=self.name)
|
||||
for record in state.candidate_index.caption_records:
|
||||
if record.get("status") != "pending":
|
||||
continue
|
||||
caption = record["caption"]
|
||||
caption_page = int(caption.get("page", 0) or 0)
|
||||
candidate_pages = [caption_page - 1, caption_page, caption_page + 1]
|
||||
all_candidates = []
|
||||
for page in candidate_pages:
|
||||
if page < 1:
|
||||
continue
|
||||
page_assets = [
|
||||
(idx, asset)
|
||||
for idx, asset in state.candidate_index.assets_by_page.get(page, [])
|
||||
if state.ledger.owner_of_asset(page=page, block_id=asset.get("block_id")) is None
|
||||
]
|
||||
all_candidates.extend(ocr_tables._score_candidate_assets(page_assets, caption, is_continuation=record["is_continuation"]))
|
||||
|
||||
for _, asset, score_dict in all_candidates:
|
||||
a_page = int(asset.get("page", 0) or 0)
|
||||
if a_page == caption_page - 1:
|
||||
ab = asset.get("bbox") or [0, 0, 0, 0]
|
||||
cb = caption.get("bbox") or [0, 0, 0, 0]
|
||||
if len(ab) >= 4 and len(cb) >= 4:
|
||||
x_ratio = (min(cb[2], ab[2]) - max(cb[0], ab[0])) / max(1.0, min(cb[2] - cb[0], ab[2] - ab[0]))
|
||||
page_h = max(state.corpus.page_max_y.values()) if state.corpus.page_max_y else 1.0
|
||||
if x_ratio >= 0.5 and float(ab[3]) >= page_h * 0.85 and float(cb[1]) <= page_h * 0.15:
|
||||
score_dict["score"] = min(score_dict.get("score", 0.0) + 0.15, 1.0)
|
||||
score_dict.setdefault("evidence", []).append("continuation_geometry_elevation")
|
||||
|
||||
all_candidates.sort(key=lambda item: item[2].get("score", 0.0), reverse=True)
|
||||
record["candidate_assets"] = [
|
||||
{"asset_block_id": asset.get("block_id", ""), "match_score": score}
|
||||
for _, asset, score in all_candidates[:3]
|
||||
]
|
||||
if not all_candidates:
|
||||
record["status"] = "unmatched_caption"
|
||||
continue
|
||||
top_idx, top_asset, top_score = all_candidates[0]
|
||||
second_score = all_candidates[1][2].get("score", 0.0) if len(all_candidates) > 1 else -1.0
|
||||
if top_score.get("score", 0.0) < 0.4:
|
||||
record["status"] = "unmatched_caption"
|
||||
continue
|
||||
if top_score.get("score", 0.0) - second_score < 0.15:
|
||||
record["status"] = "ambiguous"
|
||||
continue
|
||||
owner = ResourceRef(kind="legend", page=caption_page, block_id=record["caption_block_id"], figure_no=record["formal_table_number"])
|
||||
asset_ref = ResourceRef(kind="asset", page=int(top_asset.get("page", 0) or 0), block_id=top_asset.get("block_id"))
|
||||
conflict = state.ledger.try_claim_assets([asset_ref], owner=owner, reason=self.name)
|
||||
if conflict is not None:
|
||||
report.conflicts.append(conflict)
|
||||
continue
|
||||
match_status = "matched" if top_score.get("score", 0.0) >= 0.6 else "matched_low_confidence"
|
||||
continuation_of = None
|
||||
if record["is_continuation"] and record["formal_table_number"] is not None:
|
||||
for existing in state.matches:
|
||||
if existing.get("formal_table_number") == record["formal_table_number"] and not existing.get("is_continuation"):
|
||||
continuation_of = record["formal_table_number"]
|
||||
break
|
||||
state.accept_match(
|
||||
ClaimProposal(
|
||||
pass_name=self.name,
|
||||
figure_no=record["formal_table_number"],
|
||||
claim_type="match",
|
||||
legends=[owner],
|
||||
assets=[asset_ref],
|
||||
groups=[],
|
||||
texts=[],
|
||||
confidence=float(top_score.get("score", 0.0)),
|
||||
evidence_rank=1,
|
||||
reason=self.name,
|
||||
),
|
||||
{
|
||||
"caption_block_id": record["caption_block_id"],
|
||||
"page": caption_page,
|
||||
"caption_text": record["caption_text"],
|
||||
"table_number": record["table_number"],
|
||||
"formal_table_number": record["formal_table_number"],
|
||||
"asset_block_id": top_asset.get("block_id", ""),
|
||||
"asset_bbox": top_asset.get("bbox", [0, 0, 0, 0]),
|
||||
"assistive_text": str(top_asset.get("text", "") or ""),
|
||||
"truth_source": "image",
|
||||
"has_asset": True,
|
||||
"segments": [
|
||||
{
|
||||
"page": top_asset.get("page", 0),
|
||||
"asset_block_id": top_asset.get("block_id", ""),
|
||||
"asset_bbox": top_asset.get("bbox", [0, 0, 0, 0]),
|
||||
"is_continuation": record["is_continuation"],
|
||||
}
|
||||
],
|
||||
"note_block_ids": [],
|
||||
"note_texts": [],
|
||||
"note_bboxes": [],
|
||||
"note_band_bbox": [],
|
||||
"note_match_reason": "",
|
||||
"note_confidence": 0.0,
|
||||
"bridge_block_ids": [],
|
||||
"consumed_block_ids": [record["caption_block_id"], top_asset.get("block_id", ""), *record.get("continuation_ids", [])],
|
||||
"is_continuation": record["is_continuation"],
|
||||
"continuation_of": continuation_of,
|
||||
"match_status": match_status,
|
||||
"candidate_assets": record["candidate_assets"],
|
||||
"match_score": top_score,
|
||||
"render_bbox": None,
|
||||
"render_rotation_deg": 0,
|
||||
"asset_family_hint": top_asset.get("asset_family_hint"),
|
||||
"asset_family_confidence": top_asset.get("asset_family_confidence"),
|
||||
"asset_family_evidence": top_asset.get("asset_family_evidence"),
|
||||
},
|
||||
)
|
||||
record["status"] = "matched"
|
||||
return report
|
||||
|
||||
|
||||
class TableNotesAttachmentPass:
|
||||
name = "table_notes_attachment"
|
||||
|
||||
def run(self, state):
|
||||
from . import ocr_tables
|
||||
|
||||
report = PassReport(pass_name=self.name)
|
||||
for table in state.matches:
|
||||
if not table.get("has_asset"):
|
||||
table.setdefault("note_block_ids", [])
|
||||
table.setdefault("note_texts", [])
|
||||
table.setdefault("note_bboxes", [])
|
||||
table.setdefault("note_band_bbox", [])
|
||||
table.setdefault("note_match_reason", "")
|
||||
table.setdefault("note_confidence", 0.0)
|
||||
table.setdefault("bridge_block_ids", [])
|
||||
continue
|
||||
|
||||
asset_page = int(table.get("page", 0) or 0)
|
||||
asset_bbox = table.get("asset_bbox", [0, 0, 0, 0])
|
||||
asset_bottom = asset_bbox[3] if len(asset_bbox) >= 4 else 0
|
||||
|
||||
candidates = []
|
||||
note_match_reason = ""
|
||||
for block in state.corpus.blocks:
|
||||
if int(block.get("page", 0) or 0) != asset_page:
|
||||
continue
|
||||
brole = str(block.get("role", "") or "")
|
||||
braw_label = str(block.get("raw_label", "") or "").strip()
|
||||
btext = str(block.get("text", "") or "").strip()
|
||||
is_note = (
|
||||
brole == "footnote"
|
||||
or braw_label == "vision_footnote"
|
||||
or (
|
||||
0 < len(btext) < 120
|
||||
and brole not in {
|
||||
"noise", "page_footer", "page_header", "frontmatter_noise",
|
||||
"table_caption", "table_caption_candidate",
|
||||
"table_asset", "media_asset", "figure_caption",
|
||||
"section_heading", "subsection_heading", "reference_heading",
|
||||
}
|
||||
)
|
||||
)
|
||||
if not is_note:
|
||||
continue
|
||||
bbbox = block.get("bbox") or [0, 0, 0, 0]
|
||||
if len(bbbox) < 4:
|
||||
note_match_reason = "invalid_bbox"
|
||||
continue
|
||||
if bbbox[1] < asset_bottom or bbbox[1] > asset_bottom + 100:
|
||||
note_match_reason = "outside_vertical_range"
|
||||
continue
|
||||
if ocr_tables._table_note_falls_into_page_footnote_prior(bbbox, asset_page, state.corpus.page_footnote_prior):
|
||||
note_match_reason = "page_footnote_prior_rejected"
|
||||
continue
|
||||
if ocr_tables._looks_like_body_text_below_table(block, asset_bbox):
|
||||
note_match_reason = "body_text_like_excluded"
|
||||
continue
|
||||
candidates.append(block)
|
||||
|
||||
if candidates:
|
||||
candidates.sort(key=lambda b: (b.get("bbox") or [0, 0, 0, 0])[1])
|
||||
table["note_block_ids"] = [str(b.get("block_id", "")) for b in candidates if b.get("block_id")]
|
||||
table["note_texts"] = [str(b.get("text", "") or "").strip() for b in candidates if str(b.get("text", "") or "").strip()]
|
||||
table["note_bboxes"] = [b.get("bbox", [0, 0, 0, 0]) for b in candidates]
|
||||
table["note_band_bbox"] = [
|
||||
min(bb[0] for bb in table["note_bboxes"]),
|
||||
min(bb[1] for bb in table["note_bboxes"]),
|
||||
max(bb[2] for bb in table["note_bboxes"]),
|
||||
max(bb[3] for bb in table["note_bboxes"]),
|
||||
]
|
||||
table["note_match_reason"] = "note_band_geometry_match"
|
||||
table["note_confidence"] = 0.85
|
||||
else:
|
||||
table.setdefault("note_block_ids", [])
|
||||
table.setdefault("note_texts", [])
|
||||
table.setdefault("note_bboxes", [])
|
||||
table.setdefault("note_band_bbox", [])
|
||||
table["note_match_reason"] = note_match_reason or "no_footnote_role"
|
||||
table.setdefault("note_confidence", 0.0)
|
||||
|
||||
asset_block = next(
|
||||
(a for a in state.corpus.raw_assets if str(a.get("block_id", "")) == str(table.get("asset_block_id", "")) and int(a.get("page", 0) or 0) == asset_page),
|
||||
None,
|
||||
)
|
||||
if asset_block is not None:
|
||||
rot = ocr_tables._table_has_rotated_content(asset_block)
|
||||
if rot:
|
||||
ab = table.get("asset_bbox", [])
|
||||
caption = next(
|
||||
(r["caption"] for r in state.candidate_index.caption_records if r["caption_block_id"] == table.get("caption_block_id")),
|
||||
None,
|
||||
)
|
||||
cb = (caption.get("bbox") or caption.get("block_bbox") or []) if caption else []
|
||||
if len(ab) >= 4 and len(cb) >= 4:
|
||||
table["render_bbox"] = [min(cb[0], ab[0]), min(cb[1], ab[1]), max(cb[2], ab[2]), max(cb[3], ab[3])]
|
||||
table["render_rotation_deg"] = rot
|
||||
|
||||
# Bridge block detection
|
||||
table["bridge_block_ids"] = [
|
||||
str(block.get("block_id") or "")
|
||||
for block in state.corpus.blocks
|
||||
if int(block.get("page", 0) or 0) == asset_page
|
||||
and block.get("bridge_eligible")
|
||||
and str(block.get("layout_region") or "") == "display_zone"
|
||||
and block.get("block_id")
|
||||
]
|
||||
|
||||
# Consumed block IDs include notes
|
||||
existing_ids = set(table.get("consumed_block_ids") or [])
|
||||
for bid in table.get("note_block_ids", []):
|
||||
if bid not in existing_ids:
|
||||
table.setdefault("consumed_block_ids", []).append(bid)
|
||||
return report
|
||||
|
||||
|
||||
class TableFinalAccountingPass:
|
||||
name = "table_final_accounting"
|
||||
|
||||
def run(self, state):
|
||||
report = PassReport(pass_name=self.name)
|
||||
state.completeness = {
|
||||
"total_numbered_tables": len([r for r in state.candidate_index.caption_records if r.get("formal_table_number") is not None]),
|
||||
"accounted_for": len([r for r in state.candidate_index.caption_records if r.get("status") in {"matched", "held"}]),
|
||||
"details": [
|
||||
{"caption_block_id": r["caption_block_id"], "status": r.get("status", "pending")}
|
||||
for r in state.candidate_index.caption_records
|
||||
],
|
||||
}
|
||||
return report
|
||||
|
|
@ -1,6 +1,7 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
|
@ -114,7 +115,6 @@ def _is_weak_explicit_table_caption(block: dict) -> bool:
|
|||
def _find_table_caption_continuation(caption: dict, structured_blocks: list[dict]) -> dict | None:
|
||||
"""Find the block immediately after a weak-truncated table caption that
|
||||
looks like a continuation (stolen as figure_caption or similar)."""
|
||||
caption_page = caption.get("page", 0)
|
||||
caption_bbox = caption.get("bbox") or [0, 0, 0, 0]
|
||||
next_idx = None
|
||||
for i, block in enumerate(structured_blocks):
|
||||
|
|
@ -233,7 +233,7 @@ def _table_note_falls_into_page_footnote_prior(
|
|||
return float(note_bbox[1]) >= float(prior_by_page[page])
|
||||
|
||||
def build_table_inventory(structured_blocks: list[dict]) -> dict[str, Any]:
|
||||
return build_table_inventory_legacy(structured_blocks)
|
||||
return build_table_inventory_vnext(structured_blocks)
|
||||
|
||||
|
||||
def build_table_inventory_legacy(structured_blocks: list[dict]) -> dict[str, Any]:
|
||||
|
|
@ -637,3 +637,33 @@ def write_back_table_roles(inventory: dict, structured_blocks: list[dict]) -> No
|
|||
|
||||
def write_table_inventory(dst: Path, inventory: dict[str, Any]) -> None:
|
||||
write_json(dst, inventory)
|
||||
|
||||
|
||||
def build_table_inventory_vnext(structured_blocks: list[dict]) -> dict[str, Any]:
|
||||
from .ocr_pairing_framework import run_pairing_passes
|
||||
from .ocr_pairing_state import OwnershipLedger, PipelineState
|
||||
from .ocr_table_domain import TableCandidateIndex, TableCorpus, assemble_table_inventory
|
||||
from .ocr_table_passes import (
|
||||
TableAdjacentPagePass,
|
||||
TableFinalAccountingPass,
|
||||
TableNotesAttachmentPass,
|
||||
TableSamePagePass,
|
||||
TableWeakCaptionRecoveryPass,
|
||||
)
|
||||
|
||||
corpus = TableCorpus.from_blocks(structured_blocks)
|
||||
candidate_index = TableCandidateIndex.from_corpus(corpus)
|
||||
state = PipelineState(corpus=corpus, candidate_index=candidate_index, ledger=OwnershipLedger())
|
||||
reports = run_pairing_passes(
|
||||
state,
|
||||
[
|
||||
TableWeakCaptionRecoveryPass,
|
||||
TableSamePagePass,
|
||||
TableAdjacentPagePass,
|
||||
TableNotesAttachmentPass,
|
||||
TableFinalAccountingPass,
|
||||
],
|
||||
)
|
||||
inventory = assemble_table_inventory(state, candidate_index)
|
||||
inventory["pass_reports"] = [asdict(r) for r in reports]
|
||||
return inventory
|
||||
|
|
|
|||
|
|
@ -1,27 +1,30 @@
|
|||
# OCR-v2 Active Queue
|
||||
> Status: ACTIVE QUEUE — Round 2 truth audit + 3 targeted bug fixes. 428 regression tests pass.
|
||||
> Last updated: 2026-07-02
|
||||
> Scope: post-audit cleanup
|
||||
> Status: ACTIVE QUEUE — OCR pairing framework branch is merge-ready after merge-unblock verification. 357 targeted tests pass.
|
||||
> Last updated: 2026-07-03
|
||||
> Scope: pairing framework integration / post-merge follow-up
|
||||
|
||||
## Current Priorities
|
||||
1. **Monitor production OCR** after rotated-figure prematch, asset-internal recovery, sidecar/table fixes
|
||||
2. **Archive stale `project/current/` files**
|
||||
3. **Group-first figure inventory refactor** (deferred)
|
||||
1. **Merge `feat/ocr-pairing-framework`** back to `master`
|
||||
2. **Monitor production OCR** after pairing-core unification (figure + table on shared framework)
|
||||
3. **Archive stale `project/current/` files**
|
||||
4. **Compatibility naming cleanup** (`figure_no` / `legend` / `FigurePipelineState`) — deferred
|
||||
|
||||
## Completed This Session (cumulative)
|
||||
- **Round 2 truth audit**: 10 new papers batch-audited via ocr_truth_audit.py (high-risk mode)
|
||||
- 5 GREEN / 4 YELLOW / 1 RED (37LK5T97 — Figure 1 broken, tables unmatched)
|
||||
- 2 vision agents dispatched for RED papers
|
||||
- Findings: `docs/superpowers/specs/2026-07-02-ocr-truth-audit-round2-findings.md`
|
||||
- **37LK5T97 Figure 1 sidecar fix** (ocr_document.py)
|
||||
- Add `_is_sidecar_candidate` guard in candidate_resolution demotion
|
||||
- Sidecar caption in adjacent column no longer demoted as narrative prose
|
||||
- Figure 1 matched: `figure_001`, asset=block_id=9
|
||||
- **37LK5T97 rotated table matching fix** (ocr_scores.py)
|
||||
- Add `adjacent_x` + `y_overlap_with_asset` checks in score_table_match for rotated captions
|
||||
- All 6 tables matched with has_asset=true
|
||||
- **37LK5T97 rotated table render fix** (ocr_tables.py, ocr_objects.py)
|
||||
- `_table_has_rotated_content` detects dir=[0,-1], computes union render_bbox + rotation_deg=270
|
||||
- Tables 1-5 rendered at correct orientation (1908×2858 → 2858×1908)
|
||||
- Same approach as rotated figure rendering
|
||||
- Git: commits `16f9403` (recovery), `59cd01a` (figure quality), `bd3f3b6` (sidecar+table match), `86e0d14` (table render)
|
||||
- **Pairing framework extraction**:
|
||||
- Shared `ocr_pairing_types.py`, `ocr_pairing_state.py`, `ocr_pairing_framework.py`
|
||||
- Figure pipeline migrated onto shared core without changing public seam
|
||||
- **Table pipeline migration**:
|
||||
- Added `ocr_table_domain.py` + ordered `ocr_table_passes.py`
|
||||
- `build_table_inventory(...)` now routes to table vnext on shared core
|
||||
- **Merge-unblock hardening**:
|
||||
- Generic-state figure rotation enrichment moved behind figure-only `_match_pre_enricher`
|
||||
- Table semantic parity validation upgraded across 6 runnable real-paper fixtures
|
||||
- Added runnable `tests/fixtures/ocr_vnext_real_papers/37LK5T97/blocks.structured.jsonl`
|
||||
- Touched-file `ruff check` / `ruff format --check` green
|
||||
- **Verification**:
|
||||
- `tests/test_ocr_figures.py`
|
||||
- `tests/test_ocr_rebuild.py`
|
||||
- `tests/test_ocr_tables.py`
|
||||
- `tests/test_ocr_pairing_framework.py`
|
||||
- `tests/test_ocr_table_pairing_framework.py`
|
||||
- Result: **357 passed, 0 failed**
|
||||
|
|
|
|||
17
scripts/dev/compare_figure_inventory_vnext_baseline.py
Normal file
17
scripts/dev/compare_figure_inventory_vnext_baseline.py
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from paperforge.worker.ocr_figures import build_figure_inventory
|
||||
|
||||
|
||||
def compare_vnext_inventory_baseline(fixture_root: Path) -> dict[str, object]:
|
||||
structured_path = fixture_root / "structure" / "blocks.structured.jsonl"
|
||||
blocks = [json.loads(line) for line in structured_path.read_text(encoding="utf-8").splitlines() if line.strip()]
|
||||
inventory = build_figure_inventory(blocks)
|
||||
return {
|
||||
"pipeline_mode": inventory.get("pipeline_mode"),
|
||||
"matched_ids": [m.get("figure_id") for m in inventory.get("matched_figures", [])],
|
||||
"match_count": len(inventory.get("matched_figures", [])),
|
||||
}
|
||||
74
scripts/dev/compare_table_inventory_legacy_vs_vnext.py
Normal file
74
scripts/dev/compare_table_inventory_legacy_vs_vnext.py
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from paperforge.worker.ocr_tables import build_table_inventory_legacy, build_table_inventory_vnext
|
||||
|
||||
|
||||
def _load_blocks(fixture_root: Path) -> list[dict]:
|
||||
# Check both fixture_root/structure/blocks.structured.jsonl and fixture_root/blocks.structured.jsonl
|
||||
candidate = fixture_root / "structure" / "blocks.structured.jsonl"
|
||||
if not candidate.exists():
|
||||
candidate = fixture_root / "blocks.structured.jsonl"
|
||||
return [json.loads(line) for line in candidate.read_text(encoding="utf-8").splitlines() if line.strip()]
|
||||
|
||||
|
||||
def _str_id(val: object) -> str | None:
|
||||
"""Normalize a block ID to str, or None if empty/missing."""
|
||||
if val is None or val == "":
|
||||
return None
|
||||
return str(val)
|
||||
|
||||
|
||||
def _str_ids_sorted(vals: list) -> list[str]:
|
||||
"""Normalize block ID list to sorted strings, dropping None/empty entries."""
|
||||
return sorted(str(v) for v in (vals or []) if v is not None and v != "")
|
||||
|
||||
|
||||
def _normalize_table(table: dict) -> dict[str, object]:
|
||||
"""Normalize a table inventory entry so legacy and vnext results are comparable.
|
||||
|
||||
Benign drift handled:
|
||||
- block IDs stored as int vs str -> normalized to str
|
||||
- ``asset_block_id`` stored as ``""`` (legacy) vs ``None`` (vnext) -> both become None
|
||||
- ordering of list fields -> sorted
|
||||
"""
|
||||
asset_id = _str_id(table.get("asset_block_id"))
|
||||
return {
|
||||
"caption_block_id": _str_id(table.get("caption_block_id")),
|
||||
"page": table.get("page"),
|
||||
"formal_table_number": table.get("formal_table_number"),
|
||||
"asset_block_id": asset_id, # None if empty in either pipeline
|
||||
"match_status": table.get("match_status"),
|
||||
"note_block_ids": _str_ids_sorted(table.get("note_block_ids", [])),
|
||||
"bridge_block_ids": _str_ids_sorted(table.get("bridge_block_ids", [])),
|
||||
"consumed_block_ids": _str_ids_sorted(table.get("consumed_block_ids", [])),
|
||||
"render_rotation_deg": table.get("render_rotation_deg", 0),
|
||||
}
|
||||
|
||||
|
||||
def _table_sort_key(t: dict) -> tuple:
|
||||
return (t.get("page") or 0, t.get("formal_table_number") or 0)
|
||||
|
||||
|
||||
def compare_table_inventory_legacy_vs_vnext(fixture_root: Path) -> dict[str, object]:
|
||||
blocks = _load_blocks(fixture_root)
|
||||
legacy = build_table_inventory_legacy(copy.deepcopy(blocks))
|
||||
vnext = build_table_inventory_vnext(copy.deepcopy(blocks))
|
||||
legacy_norm = [_normalize_table(t) for t in legacy.get("tables", [])]
|
||||
vnext_norm = [_normalize_table(t) for t in vnext.get("tables", [])]
|
||||
# Sort deterministically so ordering drift does not cause false diffs.
|
||||
legacy_norm.sort(key=_table_sort_key)
|
||||
vnext_norm.sort(key=_table_sort_key)
|
||||
return {
|
||||
"legacy_raw": legacy,
|
||||
"vnext_raw": vnext,
|
||||
"legacy": legacy_norm,
|
||||
"vnext": vnext_norm,
|
||||
"diff": {
|
||||
"legacy_only": [item for item in legacy_norm if item not in vnext_norm],
|
||||
"vnext_only": [item for item in vnext_norm if item not in legacy_norm],
|
||||
},
|
||||
}
|
||||
292
tests/fixtures/ocr_vnext_real_papers/37LK5T97/blocks.structured.jsonl
vendored
Normal file
292
tests/fixtures/ocr_vnext_real_papers/37LK5T97/blocks.structured.jsonl
vendored
Normal file
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load diff
246
tests/test_ocr_pairing_framework.py
Normal file
246
tests/test_ocr_pairing_framework.py
Normal file
|
|
@ -0,0 +1,246 @@
|
|||
from __future__ import annotations
|
||||
|
||||
"""Baseline lock tests for the OCR pairing framework.
|
||||
|
||||
These tests lock the current vnext types/state module surface before extraction.
|
||||
They verify the framework types are importable and constructable.
|
||||
"""
|
||||
|
||||
|
||||
def test_vnext_types_module_is_currently_generic_surface():
|
||||
from paperforge.worker.ocr_figure_vnext_types import ClaimProposal, PassReport, ResourceRef
|
||||
|
||||
assert ResourceRef.__name__ == "ResourceRef"
|
||||
assert ClaimProposal.__name__ == "ClaimProposal"
|
||||
assert PassReport.__name__ == "PassReport"
|
||||
|
||||
|
||||
def test_vnext_state_module_is_currently_generic_surface():
|
||||
from paperforge.worker.ocr_figure_vnext_state import FigurePipelineState, OwnershipLedger
|
||||
|
||||
state = FigurePipelineState(corpus=None, candidate_index=None, ledger=OwnershipLedger())
|
||||
|
||||
assert state.matches == []
|
||||
assert state.reservations == []
|
||||
assert state.unresolved == []
|
||||
|
||||
|
||||
# ── Task 2: Generic pairing types extraction ──
|
||||
|
||||
|
||||
def test_pairing_types_module_exports_generic_types():
|
||||
from paperforge.worker.ocr_pairing_types import ClaimProposal, PassReport, ResourceRef
|
||||
|
||||
ref = ResourceRef(kind="legend", page=1, block_id="b1")
|
||||
proposal = ClaimProposal(
|
||||
pass_name="p",
|
||||
figure_no=1,
|
||||
claim_type="match",
|
||||
legends=[ref],
|
||||
assets=[],
|
||||
groups=[],
|
||||
confidence=0.9,
|
||||
evidence_rank=1,
|
||||
reason="test",
|
||||
)
|
||||
report = PassReport(pass_name="p")
|
||||
|
||||
assert proposal.pass_name == "p"
|
||||
assert report.pass_name == "p"
|
||||
assert ref.page == 1
|
||||
|
||||
|
||||
def test_figure_vnext_types_module_reexports_framework_types():
|
||||
from paperforge.worker.ocr_figure_vnext_types import ResourceRef as FigureModuleResourceRef
|
||||
from paperforge.worker.ocr_pairing_types import ResourceRef as FrameworkResourceRef
|
||||
|
||||
assert FigureModuleResourceRef is FrameworkResourceRef
|
||||
|
||||
|
||||
def test_resource_ref_rejects_page_agnostic_asset():
|
||||
import pytest
|
||||
|
||||
from paperforge.worker.ocr_pairing_types import ResourceRef
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
ResourceRef(kind="asset", page=None, block_id="a1")
|
||||
|
||||
|
||||
def test_resource_ref_normalizes_block_id_type():
|
||||
from paperforge.worker.ocr_pairing_types import ResourceRef
|
||||
|
||||
assert ResourceRef(kind="asset", page=1, block_id=1) == ResourceRef(kind="asset", page=1, block_id="1")
|
||||
|
||||
|
||||
def test_resource_ref_rejects_group_without_group_id():
|
||||
import pytest
|
||||
|
||||
from paperforge.worker.ocr_pairing_types import ResourceRef
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
ResourceRef(kind="group", page=1, block_id=None)
|
||||
|
||||
|
||||
# ── Task 3: Generic state and ledger extraction ──
|
||||
|
||||
|
||||
def test_pairing_state_module_exports_ownership_ledger():
|
||||
from paperforge.worker.ocr_pairing_state import OwnershipLedger
|
||||
from paperforge.worker.ocr_pairing_types import ResourceRef
|
||||
|
||||
ledger = OwnershipLedger()
|
||||
owner = ResourceRef(kind="legend", page=1, block_id="cap")
|
||||
asset = ResourceRef(kind="asset", page=1, block_id="asset")
|
||||
|
||||
assert ledger.try_claim_assets([asset], owner=owner, reason="test") is None
|
||||
assert ledger.owner_of_asset(page=1, block_id="asset") == owner
|
||||
|
||||
|
||||
def test_figure_vnext_state_module_reexports_framework_ledger():
|
||||
from paperforge.worker.ocr_figure_vnext_state import OwnershipLedger as FigureModuleLedger
|
||||
from paperforge.worker.ocr_pairing_state import OwnershipLedger as FrameworkLedger
|
||||
|
||||
assert FigureModuleLedger is FrameworkLedger
|
||||
|
||||
|
||||
# ── Task 4: Pairing framework runner ──
|
||||
|
||||
|
||||
def test_run_pairing_passes_executes_in_declared_order():
|
||||
from paperforge.worker.ocr_pairing_framework import run_pairing_passes
|
||||
from paperforge.worker.ocr_pairing_state import FigurePipelineState, OwnershipLedger
|
||||
from paperforge.worker.ocr_pairing_types import PassReport
|
||||
|
||||
seen = []
|
||||
|
||||
class FirstPass:
|
||||
name = "first"
|
||||
|
||||
def run(self, state):
|
||||
seen.append(self.name)
|
||||
return PassReport(pass_name=self.name)
|
||||
|
||||
class SecondPass:
|
||||
name = "second"
|
||||
|
||||
def run(self, state):
|
||||
seen.append(self.name)
|
||||
return PassReport(pass_name=self.name)
|
||||
|
||||
state = FigurePipelineState(corpus=None, candidate_index=None, ledger=OwnershipLedger())
|
||||
reports = run_pairing_passes(state, [FirstPass, SecondPass])
|
||||
|
||||
assert seen == ["first", "second"]
|
||||
assert [r.pass_name for r in reports] == ["first", "second"]
|
||||
|
||||
|
||||
# ── Task 5: Figure domain module ──
|
||||
|
||||
|
||||
def test_figure_domain_module_exports_corpus_and_candidate_index():
|
||||
from paperforge.worker.ocr_figure_domain import FigureCandidateIndex, FigureCorpus
|
||||
|
||||
assert FigureCorpus.__name__ == "FigureCorpus"
|
||||
assert FigureCandidateIndex.__name__ == "FigureCandidateIndex"
|
||||
|
||||
|
||||
def test_legacy_vnext_corpus_module_reexports_figure_domain_types():
|
||||
from paperforge.worker.ocr_figure_domain import FigureCorpus as DomainFigureCorpus
|
||||
from paperforge.worker.ocr_figure_vnext_corpus import FigureCorpus as LegacyModuleFigureCorpus
|
||||
|
||||
assert LegacyModuleFigureCorpus is DomainFigureCorpus
|
||||
|
||||
|
||||
def test_resource_ref_supports_text_kind_and_role() -> None:
|
||||
from paperforge.worker.ocr_pairing_types import ResourceRef
|
||||
|
||||
ref = ResourceRef(kind="text", page=5, block_id="note1", role="note")
|
||||
|
||||
assert ref.kind == "text"
|
||||
assert ref.role == "note"
|
||||
|
||||
|
||||
def test_claim_proposal_exposes_entity_no_alias_and_texts() -> None:
|
||||
from paperforge.worker.ocr_pairing_types import ClaimProposal, ResourceRef
|
||||
|
||||
proposal = ClaimProposal(
|
||||
pass_name="p",
|
||||
figure_no=3,
|
||||
claim_type="attach_text",
|
||||
legends=[ResourceRef(kind="legend", page=5, block_id="cap1")],
|
||||
assets=[],
|
||||
groups=[],
|
||||
texts=[ResourceRef(kind="text", page=5, block_id="note1", role="note")],
|
||||
confidence=0.8,
|
||||
evidence_rank=1,
|
||||
reason="test",
|
||||
)
|
||||
|
||||
assert proposal.entity_no == 3
|
||||
assert proposal.texts[0].block_id == "note1"
|
||||
|
||||
|
||||
def test_ownership_ledger_journals_text_attachments() -> None:
|
||||
from paperforge.worker.ocr_pairing_state import OwnershipLedger
|
||||
from paperforge.worker.ocr_pairing_types import ResourceRef
|
||||
|
||||
ledger = OwnershipLedger()
|
||||
owner = ResourceRef(kind="legend", page=5, block_id="cap1", figure_no=5)
|
||||
texts = [ResourceRef(kind="text", page=5, block_id="note1", role="note")]
|
||||
|
||||
ledger.journal_text_attachment(texts, owner=owner, reason="table-notes")
|
||||
|
||||
assert ledger.text_attachments_for(owner) == texts
|
||||
|
||||
|
||||
def test_pipeline_state_alias_preserves_figure_state_compatibility() -> None:
|
||||
from paperforge.worker.ocr_pairing_state import FigurePipelineState, OwnershipLedger, PipelineState
|
||||
|
||||
state = PipelineState(corpus=None, candidate_index=None, ledger=OwnershipLedger())
|
||||
|
||||
assert FigurePipelineState is PipelineState
|
||||
assert state.matches == []
|
||||
assert state.reservations == []
|
||||
|
||||
|
||||
def test_state_accept_match_defaults_to_no_enricher() -> None:
|
||||
"""The generic state must not enrich matches by default (domain-neutral)."""
|
||||
from paperforge.worker.ocr_pairing_state import OwnershipLedger, PipelineState
|
||||
|
||||
state = PipelineState(corpus=None, candidate_index=None, ledger=OwnershipLedger())
|
||||
assert state._match_pre_enricher is None
|
||||
|
||||
|
||||
def test_state_accept_match_invokes_enricher_when_set() -> None:
|
||||
"""When a _match_pre_enricher hook is wired, accept_match must call it."""
|
||||
from paperforge.worker.ocr_pairing_state import FigurePipelineState, OwnershipLedger
|
||||
from paperforge.worker.ocr_pairing_types import ClaimProposal, ResourceRef
|
||||
|
||||
state = FigurePipelineState(corpus=None, candidate_index=None, ledger=OwnershipLedger())
|
||||
called_with: list[tuple] = []
|
||||
|
||||
def fake_enricher(proposal, match_record, state_obj):
|
||||
called_with.append((proposal, match_record, state_obj))
|
||||
|
||||
state._match_pre_enricher = fake_enricher
|
||||
proposal = ClaimProposal(
|
||||
pass_name="test",
|
||||
figure_no=None,
|
||||
claim_type="match",
|
||||
legends=[ResourceRef(kind="legend", page=1, block_id="l1")],
|
||||
assets=[],
|
||||
groups=[],
|
||||
confidence=1.0,
|
||||
evidence_rank=0,
|
||||
reason="test",
|
||||
)
|
||||
match_record = {"test": True}
|
||||
state.accept_match(proposal, match_record)
|
||||
|
||||
assert len(called_with) == 1
|
||||
assert called_with[0][0] is proposal
|
||||
assert called_with[0][1] is match_record
|
||||
assert called_with[0][2] is state
|
||||
# The match_record should have been appended to state.matches *after* enrichment
|
||||
assert state.matches[0] is match_record
|
||||
assert state.matches[0]["test"] is True
|
||||
|
|
@ -188,7 +188,7 @@ def test_run_derived_rebuild_skips_span_backfill_when_valid(tmp_path: Path, monk
|
|||
encoding="utf-8",
|
||||
)
|
||||
(paper_root / "meta.json").write_text(
|
||||
'{"source_pdf":"sample.pdf","span_backfill_version":"2026-06-22.1","span_visual_container_version":"2026-06-26.6","span_pdf_fingerprint":"fp-1","span_backfill_coverage":1.0}',
|
||||
'{"source_pdf":"sample.pdf","span_backfill_version":"2026-07-01.1","span_visual_container_version":"2026-06-26.6","span_pdf_fingerprint":"fp-1","span_backfill_coverage":1.0}',
|
||||
encoding="utf-8",
|
||||
)
|
||||
pdf_path = tmp_path / "sample.pdf"
|
||||
|
|
@ -242,7 +242,7 @@ def test_run_derived_rebuild_records_unavailable_pdf_missing_without_rerun(tmp_p
|
|||
paper_root = tmp_path / "System" / "PaperForge" / "ocr" / key
|
||||
(paper_root / "canonical").mkdir(parents=True)
|
||||
(paper_root / "raw").mkdir(parents=True)
|
||||
(paper_root / "meta.json").write_text('{"span_backfill_version":"2026-06-22.1"}', encoding="utf-8")
|
||||
(paper_root / "meta.json").write_text('{"span_backfill_version":"2026-07-01.1"}', encoding="utf-8")
|
||||
(paper_root / "canonical" / "blocks.raw.jsonl").write_text('{"paper_id":"TESTKEY1","page":1,"block_id":"p1_b1","raw_label":"text","raw_order":0,"text":"A","bbox":[0,0,10,10],"page_width":600,"page_height":800,"span_metadata":[{"size":10}]}\n', encoding="utf-8")
|
||||
(paper_root / "raw" / "source_metadata.json").write_text('{"title":"Example Title"}', encoding="utf-8")
|
||||
|
||||
|
|
@ -292,8 +292,8 @@ def test_run_derived_rebuild_does_not_skip_when_current_fingerprint_is_unknown()
|
|||
from paperforge.worker.ocr_rebuild import _span_backfill_is_valid
|
||||
|
||||
meta = {
|
||||
"span_backfill_version": "2026-06-22.1",
|
||||
"span_visual_container_version": "2026-06-22.1",
|
||||
"span_backfill_version": "2026-07-01.1",
|
||||
"span_visual_container_version": "2026-06-26.6",
|
||||
"span_pdf_fingerprint": "unknown",
|
||||
"span_backfill_coverage": 1.0,
|
||||
}
|
||||
|
|
@ -306,7 +306,7 @@ def test_span_backfill_invalid_when_version_mismatch() -> None:
|
|||
|
||||
meta = {
|
||||
"span_backfill_version": "old",
|
||||
"span_visual_container_version": "2026-06-22.1",
|
||||
"span_visual_container_version": "2026-06-26.6",
|
||||
"span_pdf_fingerprint": "fp-1",
|
||||
}
|
||||
|
||||
|
|
@ -317,7 +317,7 @@ def test_span_backfill_invalid_when_visual_container_version_mismatch() -> None:
|
|||
from paperforge.worker.ocr_rebuild import _span_backfill_is_valid
|
||||
|
||||
meta = {
|
||||
"span_backfill_version": "2026-06-22.1",
|
||||
"span_backfill_version": "2026-07-01.1",
|
||||
"span_visual_container_version": "old",
|
||||
"span_pdf_fingerprint": "fp-1",
|
||||
}
|
||||
|
|
@ -329,8 +329,8 @@ def test_span_backfill_invalid_when_fingerprint_mismatch() -> None:
|
|||
from paperforge.worker.ocr_rebuild import _span_backfill_is_valid
|
||||
|
||||
meta = {
|
||||
"span_backfill_version": "2026-06-22.1",
|
||||
"span_visual_container_version": "2026-06-22.1",
|
||||
"span_backfill_version": "2026-07-01.1",
|
||||
"span_visual_container_version": "2026-06-26.6",
|
||||
"span_pdf_fingerprint": "fp-old",
|
||||
}
|
||||
|
||||
|
|
@ -341,8 +341,8 @@ def test_span_backfill_invalid_when_coverage_below_threshold() -> None:
|
|||
from paperforge.worker.ocr_rebuild import _span_backfill_is_valid
|
||||
|
||||
meta = {
|
||||
"span_backfill_version": "2026-06-22.1",
|
||||
"span_visual_container_version": "2026-06-22.1",
|
||||
"span_backfill_version": "2026-07-01.1",
|
||||
"span_visual_container_version": "2026-06-26.6",
|
||||
"span_pdf_fingerprint": "fp-1",
|
||||
}
|
||||
|
||||
|
|
@ -422,3 +422,123 @@ doi: 10.1000/example
|
|||
assert meta["authors"] == ["W. H. Marks"]
|
||||
assert meta.get("authors_incomplete") is True
|
||||
assert meta.get("authors_source") == "paper_note.first_author_fallback"
|
||||
|
||||
|
||||
# ── Plan A: rebuild compatibility proof (pairing framework) ──
|
||||
|
||||
|
||||
def test_run_derived_rebuild_for_keys_still_uses_public_build_figure_inventory(tmp_path: Path, monkeypatch) -> None:
|
||||
import json
|
||||
|
||||
from paperforge.worker.ocr_rebuild import run_derived_rebuild_for_keys
|
||||
|
||||
key = "TESTKEY1"
|
||||
paper_root = tmp_path / "System" / "PaperForge" / "ocr" / key
|
||||
(paper_root / "canonical").mkdir(parents=True)
|
||||
(paper_root / "raw").mkdir(parents=True)
|
||||
(paper_root / "structure").mkdir(parents=True)
|
||||
|
||||
(paper_root / "canonical" / "blocks.raw.jsonl").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"paper_id": key,
|
||||
"page": 1,
|
||||
"block_id": "p1_b1",
|
||||
"raw_label": "text",
|
||||
"raw_order": 0,
|
||||
"text": "Example text",
|
||||
"bbox": [10, 10, 100, 40],
|
||||
"page_width": 600,
|
||||
"page_height": 800,
|
||||
}
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(paper_root / "raw" / "source_metadata.json").write_text(
|
||||
json.dumps({"title": "Example Title"}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(paper_root / "meta.json").write_text(json.dumps({"source_pdf": ""}), encoding="utf-8")
|
||||
|
||||
called = {"count": 0}
|
||||
|
||||
def fake_build_figure_inventory(structured_blocks, page_width=1200, page_pdf_lines_by_page=None):
|
||||
called["count"] += 1
|
||||
return {
|
||||
"pipeline_mode": "vnext",
|
||||
"matched_figures": [],
|
||||
"ambiguous_figures": [],
|
||||
"unmatched_legends": [],
|
||||
"unmatched_assets": [],
|
||||
"unresolved_clusters": [],
|
||||
"held_figures": [],
|
||||
"rejected_legends": [],
|
||||
"page_ledger": {},
|
||||
"residual_ledger": {},
|
||||
"local_pairing_hypotheses": [],
|
||||
"pass_reports": [],
|
||||
"completeness": {"total_numbered_legends": 0, "accounted_for": 0, "details": []},
|
||||
}
|
||||
|
||||
monkeypatch.setattr("paperforge.worker.ocr_figures.build_figure_inventory", fake_build_figure_inventory)
|
||||
monkeypatch.setattr("paperforge.worker.ocr_blocks.write_structured_blocks_jsonl", lambda *args, **kwargs: None)
|
||||
monkeypatch.setattr(
|
||||
"paperforge.worker.ocr_metadata.extract_frontmatter_candidates_from_blocks",
|
||||
lambda structured: {"title": "Example Title", "authors_text": None, "doi_candidates": []},
|
||||
)
|
||||
|
||||
result = run_derived_rebuild_for_keys(tmp_path, [key])
|
||||
|
||||
assert result["rebuild_count"] == 1
|
||||
assert called["count"] > 0
|
||||
|
||||
|
||||
def test_run_derived_rebuild_for_keys_still_uses_public_build_table_inventory(tmp_path: Path, monkeypatch) -> None:
|
||||
import json
|
||||
|
||||
from paperforge.worker.ocr_rebuild import run_derived_rebuild_for_keys
|
||||
|
||||
key = "TESTKEY1"
|
||||
paper_root = tmp_path / "System" / "PaperForge" / "ocr" / key
|
||||
(paper_root / "canonical").mkdir(parents=True)
|
||||
(paper_root / "raw").mkdir(parents=True)
|
||||
(paper_root / "structure").mkdir(parents=True)
|
||||
|
||||
(paper_root / "canonical" / "blocks.raw.jsonl").write_text(
|
||||
json.dumps({"paper_id": key, "page": 1, "block_id": "p1_b1", "raw_label": "text", "raw_order": 0, "text": "text", "bbox": [10, 10, 100, 40], "page_width": 600, "page_height": 800}) + "\n", encoding="utf-8"
|
||||
)
|
||||
(paper_root / "raw" / "source_metadata.json").write_text(json.dumps({"title": "Example Title"}), encoding="utf-8")
|
||||
(paper_root / "meta.json").write_text(json.dumps({"source_pdf": ""}), encoding="utf-8")
|
||||
|
||||
called = {"count": 0}
|
||||
|
||||
def fake_build_table_inventory(structured_blocks):
|
||||
called["count"] += 1
|
||||
return {"tables": [], "held_tables": [], "unmatched_captions": [], "unmatched_assets": [], "official_table_count": 0}
|
||||
|
||||
monkeypatch.setattr("paperforge.worker.ocr_tables.build_table_inventory", fake_build_table_inventory)
|
||||
monkeypatch.setattr("paperforge.worker.ocr_blocks.write_structured_blocks_jsonl", lambda *a, **kw: None)
|
||||
monkeypatch.setattr("paperforge.worker.ocr_metadata.extract_frontmatter_candidates_from_blocks", lambda s: {"title": "Example Title", "authors_text": None, "doi_candidates": []})
|
||||
monkeypatch.setattr("paperforge.worker.ocr_figures.build_figure_inventory", lambda *a, **kw: {"matched_figures": [], "unmatched_assets": [], "unresolved_clusters": []})
|
||||
monkeypatch.setattr("paperforge.worker.ocr_figures.write_back_figure_roles", lambda *a, **kw: None)
|
||||
monkeypatch.setattr("paperforge.worker.ocr_figures.write_figure_inventory", lambda *a, **kw: None)
|
||||
monkeypatch.setattr("paperforge.worker.ocr_figure_reader.synthesize_reader_figures", lambda *a, **kw: {})
|
||||
monkeypatch.setattr("paperforge.worker.ocr_tables.write_back_table_roles", lambda *a, **kw: None)
|
||||
monkeypatch.setattr("paperforge.worker.ocr_tables.write_table_inventory", lambda *a, **kw: None)
|
||||
monkeypatch.setattr("paperforge.worker.ocr_objects.extract_and_write_objects", lambda *a, **kw: None)
|
||||
monkeypatch.setattr("paperforge.worker.ocr_render.render_fulltext_markdown", lambda *a, **kw: "")
|
||||
monkeypatch.setattr("paperforge.worker.ocr_render.write_render_outputs", lambda *a, **kw: None)
|
||||
monkeypatch.setattr("paperforge.worker.ocr_health.build_ocr_health", lambda *a, **kw: {})
|
||||
monkeypatch.setattr("paperforge.worker.ocr_health.build_ocr_raw_integrity_health", lambda *a, **kw: {})
|
||||
monkeypatch.setattr("paperforge.worker.ocr_health.write_ocr_health", lambda *a, **kw: None)
|
||||
monkeypatch.setattr("paperforge.worker.ocr_decisions.collect_decisions", lambda *a, **kw: [])
|
||||
monkeypatch.setattr("paperforge.worker.ocr_decisions.write_decision_log", lambda *a, **kw: None)
|
||||
monkeypatch.setattr("paperforge.worker.ocr_index.build_role_indexes", lambda *a, **kw: {})
|
||||
monkeypatch.setattr("paperforge.worker.ocr_index.write_role_index", lambda *a, **kw: None)
|
||||
monkeypatch.setattr("paperforge.worker.ocr.validate_ocr_meta", lambda *a, **kw: ("done", ""))
|
||||
|
||||
result = run_derived_rebuild_for_keys(tmp_path, [key])
|
||||
|
||||
assert result["rebuild_count"] == 1
|
||||
assert called["count"] > 0
|
||||
|
|
|
|||
450
tests/test_ocr_table_pairing_framework.py
Normal file
450
tests/test_ocr_table_pairing_framework.py
Normal file
|
|
@ -0,0 +1,450 @@
|
|||
from __future__ import annotations
|
||||
|
||||
|
||||
def test_table_corpus_collects_captions_assets_and_page_context() -> None:
|
||||
from paperforge.worker.ocr_table_domain import TableCorpus
|
||||
|
||||
blocks = [
|
||||
{
|
||||
"block_id": "cap1",
|
||||
"page": 5,
|
||||
"role": "table_caption",
|
||||
"text": "Table 1. Example",
|
||||
"bbox": [100, 100, 700, 140],
|
||||
},
|
||||
{
|
||||
"block_id": "asset1",
|
||||
"page": 5,
|
||||
"role": "table_html",
|
||||
"raw_label": "table",
|
||||
"text": "<table><tr><td>x</td></tr></table>",
|
||||
"bbox": [100, 160, 700, 500],
|
||||
},
|
||||
{
|
||||
"block_id": "note1",
|
||||
"page": 5,
|
||||
"role": "footnote",
|
||||
"raw_label": "vision_footnote",
|
||||
"text": "* p < 0.05",
|
||||
"bbox": [100, 520, 300, 550],
|
||||
},
|
||||
]
|
||||
|
||||
corpus = TableCorpus.from_blocks(blocks)
|
||||
|
||||
assert [b["block_id"] for b in corpus.raw_captions] == ["cap1"]
|
||||
assert [b["block_id"] for b in corpus.raw_assets] == ["asset1"]
|
||||
assert 5 in corpus.page_footnote_prior
|
||||
assert 5 in corpus.page_max_y
|
||||
|
||||
|
||||
def test_table_candidate_index_materializes_caption_records_and_assets_by_page() -> None:
|
||||
from paperforge.worker.ocr_table_domain import TableCandidateIndex, TableCorpus
|
||||
|
||||
blocks = [
|
||||
{
|
||||
"block_id": "cap2",
|
||||
"page": 6,
|
||||
"role": "table_caption_candidate",
|
||||
"text": "Table 2. (continued)",
|
||||
"bbox": [100, 100, 700, 130],
|
||||
},
|
||||
{
|
||||
"block_id": "cap1",
|
||||
"page": 5,
|
||||
"role": "table_caption",
|
||||
"text": "Table 1. Example",
|
||||
"bbox": [100, 100, 700, 140],
|
||||
},
|
||||
{
|
||||
"block_id": "asset1",
|
||||
"page": 5,
|
||||
"role": "table_html",
|
||||
"raw_label": "table",
|
||||
"text": "<table></table>",
|
||||
"bbox": [100, 160, 700, 500],
|
||||
},
|
||||
]
|
||||
|
||||
index = TableCandidateIndex.from_corpus(TableCorpus.from_blocks(blocks))
|
||||
|
||||
assert [r["caption_block_id"] for r in index.caption_records] == ["cap1", "cap2"]
|
||||
assert 5 in index.assets_by_page
|
||||
assert index.caption_records[1]["is_continuation"] is True
|
||||
|
||||
|
||||
def test_assemble_table_inventory_preserves_public_shape_for_empty_state() -> None:
|
||||
from paperforge.worker.ocr_pairing_state import OwnershipLedger, PipelineState
|
||||
from paperforge.worker.ocr_table_domain import TableCandidateIndex, TableCorpus, assemble_table_inventory
|
||||
|
||||
blocks = [
|
||||
{"block_id": "cap1", "page": 1, "role": "table_caption", "text": "Table 1. Example", "bbox": [0, 0, 10, 10]}
|
||||
]
|
||||
corpus = TableCorpus.from_blocks(blocks)
|
||||
index = TableCandidateIndex.from_corpus(corpus)
|
||||
state = PipelineState(corpus=corpus, candidate_index=index, ledger=OwnershipLedger())
|
||||
|
||||
inventory = assemble_table_inventory(state, index)
|
||||
|
||||
assert inventory == {
|
||||
"official_table_count": 0,
|
||||
"held_tables": [],
|
||||
"tables": [
|
||||
{
|
||||
"asset_bbox": [],
|
||||
"asset_block_id": None,
|
||||
"assistive_text": "",
|
||||
"bridge_block_ids": [],
|
||||
"candidate_assets": [],
|
||||
"caption_block_id": "cap1",
|
||||
"caption_text": "Table 1. Example",
|
||||
"consumed_block_ids": [],
|
||||
"continuation_of": None,
|
||||
"formal_table_number": 1,
|
||||
"has_asset": False,
|
||||
"is_continuation": False,
|
||||
"match_score": {"decision": "unmatched", "evidence": [], "matched_asset_id": "", "score": 0.0},
|
||||
"match_status": "pending",
|
||||
"note_band_bbox": [],
|
||||
"note_bboxes": [],
|
||||
"note_block_ids": [],
|
||||
"note_confidence": 0.0,
|
||||
"note_match_reason": "",
|
||||
"note_texts": [],
|
||||
"page": 1,
|
||||
"render_bbox": None,
|
||||
"render_rotation_deg": 0,
|
||||
"segments": [],
|
||||
"table_number": 1,
|
||||
"truth_source": "image",
|
||||
}
|
||||
],
|
||||
"unmatched_assets": [],
|
||||
"unmatched_captions": [blocks[0]],
|
||||
}
|
||||
|
||||
|
||||
# ── Task 4: vnext matching passes ──
|
||||
|
||||
|
||||
def test_build_table_inventory_vnext_matches_same_page_best_asset() -> None:
|
||||
from paperforge.worker.ocr_tables import build_table_inventory_vnext
|
||||
|
||||
structured_blocks = [
|
||||
{
|
||||
"block_id": "cap1",
|
||||
"page": 3,
|
||||
"role": "table_caption",
|
||||
"text": "Table 1. Example",
|
||||
"bbox": [100, 100, 700, 140],
|
||||
},
|
||||
{
|
||||
"block_id": "asset_good",
|
||||
"page": 3,
|
||||
"role": "table_html",
|
||||
"raw_label": "table",
|
||||
"text": "<table></table>",
|
||||
"bbox": [100, 160, 700, 500],
|
||||
},
|
||||
{
|
||||
"block_id": "asset_bad",
|
||||
"page": 3,
|
||||
"role": "media_asset",
|
||||
"raw_label": "image",
|
||||
"text": "",
|
||||
"bbox": [800, 160, 1100, 300],
|
||||
},
|
||||
]
|
||||
|
||||
inventory = build_table_inventory_vnext(structured_blocks)
|
||||
|
||||
assert inventory["tables"][0]["asset_block_id"] == "asset_good"
|
||||
assert inventory["tables"][0]["match_status"] in {"matched", "matched_low_confidence"}
|
||||
|
||||
|
||||
def test_build_table_inventory_vnext_holds_validation_first_weak_caption_without_same_page_asset() -> None:
|
||||
from paperforge.worker.ocr_tables import build_table_inventory_vnext
|
||||
|
||||
structured_blocks = [
|
||||
{
|
||||
"block_id": "cap1",
|
||||
"page": 4,
|
||||
"role": "body_text",
|
||||
"text": "Table 2",
|
||||
"bbox": [100, 100, 260, 130],
|
||||
"zone": "display_zone",
|
||||
"style_family": "table_caption_like",
|
||||
"marker_signature": {"type": "table_number"},
|
||||
}
|
||||
]
|
||||
|
||||
inventory = build_table_inventory_vnext(structured_blocks)
|
||||
|
||||
assert inventory["tables"] == []
|
||||
assert inventory["held_tables"][0]["hold_reason"] == "insufficient_caption_evidence"
|
||||
|
||||
|
||||
def test_build_table_inventory_vnext_materializes_split_caption_continuation() -> None:
|
||||
from paperforge.worker.ocr_tables import build_table_inventory_vnext
|
||||
|
||||
structured_blocks = [
|
||||
{"block_id": "cap1", "page": 5, "role": "table_caption", "text": "Table 3.", "bbox": [100, 100, 220, 130]},
|
||||
{"block_id": "cap2", "page": 5, "role": "body_text", "text": "Continuation text", "bbox": [100, 132, 700, 170]},
|
||||
{
|
||||
"block_id": "asset1",
|
||||
"page": 5,
|
||||
"role": "table_html",
|
||||
"raw_label": "table",
|
||||
"text": "<table></table>",
|
||||
"bbox": [100, 180, 700, 500],
|
||||
},
|
||||
]
|
||||
|
||||
inventory = build_table_inventory_vnext(structured_blocks)
|
||||
|
||||
assert "cap2" in inventory["tables"][0]["consumed_block_ids"]
|
||||
assert inventory["tables"][0]["caption_text"].startswith("Table 3.")
|
||||
|
||||
|
||||
def test_build_table_inventory_vnext_previous_page_continuation_gets_geometry_elevation() -> None:
|
||||
from paperforge.worker.ocr_tables import build_table_inventory_vnext
|
||||
|
||||
structured_blocks = [
|
||||
{
|
||||
"block_id": "asset_prev",
|
||||
"page": 7,
|
||||
"role": "table_html",
|
||||
"raw_label": "table",
|
||||
"text": "<table></table>",
|
||||
"bbox": [100, 1200, 900, 1480],
|
||||
},
|
||||
{
|
||||
"block_id": "cap1",
|
||||
"page": 8,
|
||||
"role": "table_caption",
|
||||
"text": "Table 4. (continued)",
|
||||
"bbox": [100, 110, 700, 150],
|
||||
},
|
||||
]
|
||||
|
||||
inventory = build_table_inventory_vnext(structured_blocks)
|
||||
|
||||
assert inventory["tables"][0]["asset_block_id"] == "asset_prev"
|
||||
assert "continuation_geometry_elevation" in inventory["tables"][0]["match_score"]["evidence"]
|
||||
|
||||
|
||||
# ── Task 5: Notes, accounting, diff tooling ──
|
||||
|
||||
|
||||
def test_build_table_inventory_vnext_collects_note_band_and_bridge_blocks() -> None:
|
||||
from paperforge.worker.ocr_tables import build_table_inventory_vnext
|
||||
|
||||
structured_blocks = [
|
||||
{
|
||||
"block_id": "cap1",
|
||||
"page": 5,
|
||||
"role": "table_caption",
|
||||
"text": "Table 1. Example",
|
||||
"bbox": [100, 100, 700, 140],
|
||||
},
|
||||
{
|
||||
"block_id": "asset1",
|
||||
"page": 5,
|
||||
"role": "table_html",
|
||||
"raw_label": "table",
|
||||
"text": "<table></table>",
|
||||
"bbox": [100, 160, 700, 520],
|
||||
},
|
||||
{"block_id": "note1", "page": 5, "role": "footnote", "text": "* p < 0.05", "bbox": [100, 530, 220, 555]},
|
||||
{
|
||||
"block_id": "bridge1",
|
||||
"page": 5,
|
||||
"bridge_eligible": True,
|
||||
"layout_region": "display_zone",
|
||||
"text": "",
|
||||
"bbox": [100, 150, 700, 155],
|
||||
},
|
||||
]
|
||||
|
||||
tables = build_table_inventory_vnext(structured_blocks)["tables"]
|
||||
|
||||
assert len(tables) == 1
|
||||
assert tables[0]["note_block_ids"] == ["note1"]
|
||||
assert tables[0]["bridge_block_ids"] == ["bridge1"]
|
||||
assert "note1" in tables[0]["consumed_block_ids"]
|
||||
|
||||
|
||||
def test_build_table_inventory_vnext_respects_page_footnote_prior() -> None:
|
||||
from paperforge.worker.ocr_tables import build_table_inventory_vnext
|
||||
|
||||
structured_blocks = [
|
||||
{
|
||||
"block_id": "cap1",
|
||||
"page": 5,
|
||||
"role": "table_caption",
|
||||
"text": "Table 1. Example",
|
||||
"bbox": [100, 100, 700, 140],
|
||||
},
|
||||
{
|
||||
"block_id": "asset1",
|
||||
"page": 5,
|
||||
"role": "table_html",
|
||||
"raw_label": "table",
|
||||
"text": "<table></table>",
|
||||
"bbox": [100, 160, 700, 1150],
|
||||
},
|
||||
{
|
||||
"block_id": "note1",
|
||||
"page": 5,
|
||||
"role": "footnote",
|
||||
"text": "* footer-area note",
|
||||
"bbox": [100, 1180, 300, 1210],
|
||||
},
|
||||
]
|
||||
|
||||
table = build_table_inventory_vnext(structured_blocks)["tables"][0]
|
||||
assert table["note_match_reason"] in {"note_band_geometry_match", "outside_vertical_range"}
|
||||
|
||||
|
||||
def test_build_table_inventory_vnext_sets_render_rotation_fields_for_rotated_table_asset() -> None:
|
||||
from paperforge.worker.ocr_tables import build_table_inventory_vnext
|
||||
|
||||
structured_blocks = [
|
||||
{
|
||||
"block_id": "cap1",
|
||||
"page": 8,
|
||||
"role": "table_caption",
|
||||
"text": "Table 2. Caption",
|
||||
"bbox": [100, 134, 967, 1442],
|
||||
},
|
||||
{
|
||||
"block_id": "asset1",
|
||||
"page": 8,
|
||||
"role": "table_html",
|
||||
"raw_label": "table",
|
||||
"text": "<table></table>",
|
||||
"bbox": [100, 100, 880, 1400],
|
||||
"span_metadata": [{"dir": [0.0, -1.0], "wmode": 0}],
|
||||
},
|
||||
]
|
||||
inventory = build_table_inventory_vnext(structured_blocks)
|
||||
tables = inventory["tables"]
|
||||
|
||||
assert len(tables) >= 1
|
||||
assert tables[0]["render_rotation_deg"] in {0, 270}
|
||||
|
||||
|
||||
def test_compare_table_inventory_legacy_vs_vnext_smoke_fixture() -> None:
|
||||
from pathlib import Path
|
||||
|
||||
from scripts.dev.compare_table_inventory_legacy_vs_vnext import compare_table_inventory_legacy_vs_vnext
|
||||
|
||||
result = compare_table_inventory_legacy_vs_vnext(Path("tests/fixtures/ocr_vnext_real_papers/2HEUD5P9"))
|
||||
|
||||
assert "legacy" in result
|
||||
assert "vnext" in result
|
||||
assert "diff" in result
|
||||
|
||||
|
||||
def test_compare_table_inventory_parity_on_all_real_paper_fixtures() -> None:
|
||||
"""Semantic parity after normalising benign drift (string/int, ordering, None vs '').
|
||||
|
||||
Runnable fixtures: 2HEUD5P9, 28JLIHLS, YGH7VEX6, DWQQK2YB, 24YKLTHQ, 37LK5T97.
|
||||
The first five achieve full parity across all fields including consumed_block_ids.
|
||||
|
||||
37LK5T97 gap: consumed_block_ids in legacy drops caption block_id=0 because
|
||||
the ``if bid`` filter treats integer 0 as falsy; vnext uses string ``'0'``
|
||||
which is truthy. This is a known legacy bug that vnext fixes — the core
|
||||
fields (caption_block_id, page, formal_table_number, asset_block_id,
|
||||
match_status, render_rotation_deg) are identical after normalisation.
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
from scripts.dev.compare_table_inventory_legacy_vs_vnext import compare_table_inventory_legacy_vs_vnext
|
||||
|
||||
fixtures = [
|
||||
("2HEUD5P9", False),
|
||||
("28JLIHLS", False),
|
||||
("YGH7VEX6", False),
|
||||
("DWQQK2YB", False),
|
||||
("24YKLTHQ", False),
|
||||
("37LK5T97", True), # True = known consumed_block_ids gap
|
||||
]
|
||||
|
||||
core_fields = {
|
||||
"caption_block_id",
|
||||
"page",
|
||||
"formal_table_number",
|
||||
"asset_block_id",
|
||||
"match_status",
|
||||
"render_rotation_deg",
|
||||
}
|
||||
|
||||
for pid, known_consumed_gap in fixtures:
|
||||
result = compare_table_inventory_legacy_vs_vnext(Path(f"tests/fixtures/ocr_vnext_real_papers/{pid}"))
|
||||
|
||||
assert len(result["legacy"]) == len(result["vnext"]), (
|
||||
f"{pid}: table count mismatch: {len(result['legacy'])} vs {len(result['vnext'])}"
|
||||
)
|
||||
|
||||
lo = result["diff"]["legacy_only"]
|
||||
vo = result["diff"]["vnext_only"]
|
||||
|
||||
# Compare core fields directly: for each fixture paper, every
|
||||
# legacy table's core fields must appear in the vnext list and
|
||||
# vice versa. This catches real semantic drift.
|
||||
legacy_core = {tuple(t.get(k) for k in core_fields) for t in result["legacy"]}
|
||||
vnext_core = {tuple(t.get(k) for k in core_fields) for t in result["vnext"]}
|
||||
|
||||
legacy_missing = legacy_core - vnext_core
|
||||
vnext_extra = vnext_core - legacy_core
|
||||
|
||||
assert not legacy_missing and not vnext_extra, (
|
||||
f"{pid}: core-field mismatch — "
|
||||
f"legacy has {legacy_missing} not in vnext, "
|
||||
f"vnext has {vnext_extra} not in legacy"
|
||||
)
|
||||
|
||||
# consumed_block_ids / note / bridge diffs: must be the known gap or absent.
|
||||
consumed_lo = {tuple(t["consumed_block_ids"]) for t in lo}
|
||||
consumed_vo = {tuple(t["consumed_block_ids"]) for t in vo}
|
||||
|
||||
if known_consumed_gap:
|
||||
assert consumed_lo or consumed_vo, f"{pid}: expected consumed_block_ids gap but found none"
|
||||
# note / bridge must be pair-wise equivalent across lo and vo
|
||||
# (same empty values for 37LK5T97 — no notes or bridges).
|
||||
lo_pairs = {(tuple(t.get("note_block_ids")), tuple(t.get("bridge_block_ids"))) for t in lo}
|
||||
vo_pairs = {(tuple(t.get("note_block_ids")), tuple(t.get("bridge_block_ids"))) for t in vo}
|
||||
assert lo_pairs == vo_pairs, (
|
||||
f"{pid}: note/bridge mismatch beyond consumed_block_ids gap: lo={lo_pairs} vo={vo_pairs}"
|
||||
)
|
||||
else:
|
||||
assert not consumed_lo and not consumed_vo, (
|
||||
f"{pid}: unexpected consumed_block_ids diffs: legacy={consumed_lo}, vnext={consumed_vo}"
|
||||
)
|
||||
|
||||
|
||||
def test_table_pipeline_does_not_set_figure_enricher() -> None:
|
||||
"""Table pipeline state must not wire the figure-only rotation enricher."""
|
||||
from paperforge.worker.ocr_table_domain import TableCandidateIndex, TableCorpus
|
||||
|
||||
# Build a minimal table inventory — the state created internally must not
|
||||
# have the figure rotation enricher set.
|
||||
structured_blocks = [
|
||||
{
|
||||
"block_id": "cap1",
|
||||
"page": 1,
|
||||
"role": "table_caption",
|
||||
"text": "Table 1. Results.",
|
||||
"bbox": [100, 50, 500, 80],
|
||||
},
|
||||
{"block_id": "tab1", "page": 1, "role": "table_asset", "bbox": [100, 100, 500, 400]},
|
||||
]
|
||||
from paperforge.worker.ocr_pairing_state import OwnershipLedger, PipelineState
|
||||
|
||||
corpus = TableCorpus.from_blocks(structured_blocks)
|
||||
candidate_index = TableCandidateIndex.from_corpus(corpus)
|
||||
state = PipelineState(corpus=corpus, candidate_index=candidate_index, ledger=OwnershipLedger())
|
||||
|
||||
# Table pipeline should never set the figure-only enricher
|
||||
assert state._match_pre_enricher is None
|
||||
|
|
@ -1307,22 +1307,16 @@ def test_extract_table_number_supports_roman_and_s_prefix(text, expected_number)
|
|||
assert _extract_table_number(text) == expected_number
|
||||
|
||||
|
||||
def test_build_table_inventory_delegates_to_legacy_during_plan_b(monkeypatch) -> None:
|
||||
def test_build_table_inventory_delegates_to_vnext_after_cutover(monkeypatch) -> None:
|
||||
from paperforge.worker import ocr_tables
|
||||
|
||||
called = {}
|
||||
|
||||
def fake_legacy(structured_blocks):
|
||||
def fake_vnext(structured_blocks):
|
||||
called["blocks"] = structured_blocks
|
||||
return {
|
||||
"tables": [],
|
||||
"held_tables": [],
|
||||
"unmatched_captions": [],
|
||||
"unmatched_assets": [],
|
||||
"official_table_count": 0,
|
||||
}
|
||||
return {"tables": [], "held_tables": [], "unmatched_captions": [], "unmatched_assets": [], "official_table_count": 0}
|
||||
|
||||
monkeypatch.setattr(ocr_tables, "build_table_inventory_legacy", fake_legacy, raising=False)
|
||||
monkeypatch.setattr(ocr_tables, "build_table_inventory_vnext", fake_vnext)
|
||||
|
||||
blocks = [{"block_id": "cap1", "page": 1, "role": "table_caption", "text": "Table 1. Example"}]
|
||||
result = ocr_tables.build_table_inventory(blocks)
|
||||
|
|
|
|||
Loading…
Reference in a new issue