fix: rebuild path writes meta.ocr_health_overall

_phase4_render_health now returns (markdown, health_overall),
_phase5_finalize writes it to meta before write_json.

Also adds ADR with architecture review decisions:
- #2 Meta divergence — fixed
- #3 Quality signals for consumers — deferred to role-override v1
- #4 Health report richness — keep all fields
- #5 PDF lines cache — rejected
- #6 Integration tests — deferred to role-override v1
This commit is contained in:
LLLin000 2026-07-05 22:02:40 +08:00
parent d50ea4631b
commit cd352cb007
2 changed files with 68 additions and 5 deletions

View file

@ -0,0 +1,63 @@
# ADR: Block Role Override Seam
**Date:** 2026-07-05
**Status:** Draft (next version)
## Context
OCR figure/table matching is never perfect. Rather than building a GUI for manual adjustment, we can insert a **data seam** between structured block building and render: allow an agent or user to override a block's `role` via a sidecar patch file, then let the existing render pipeline consume the corrected roles.
## Decision
Introduce `override_role_patch.json` (sidecar per paper):
```json
{
"version": 1,
"overrides": [
{"block_id": "p7_b3", "role": "figure_caption"},
{"block_id": "p9_b1", "role": "section_heading"}
]
}
```
Rebuild Phase 3→4 reads this file and applies overrides to structured blocks in memory before render. No changes to matching logic, figure inventory, or render.
### Scope (v1)
**Figure caption matching only.** Not text flow correction (ref zone contamination, heading misplacement). Rationale:
- Agent can 100% confirm a caption mismatch from fulltext + cropped image
- User can describe "page X 'Figure Y.' text should be a figure caption"
- Text flow correction requires position understanding, not just role change
### Block Identification
`block_id` (`p{N}_{label}`) is stable across rebuild. For user→agent handoff, natural language ("page 7, the 'Figure 3.' paragraph") is resolved to `block_id` by matching `page` + block `text` prefix.
### Consumer Tools and Quality Signals
Deferred. Quality indicators (`build_quality_indicators`, `evaluate_readiness`) remain unwired until the override seam is implemented — quality signals without a remediation path cause user frustration.
## Open Questions
- Override + rebuild cycle: does a rebuild re-derive roles and then apply override on top? Yes — override patches after structured block building, not instead of it.
- Backward compat: missing `override_role_patch.json` = no-op.
- Figure inventory consistency: if a block's role changes to `figure_caption`, does the figure inventory need updating? Current render doesn't read figure inventory for captions embedded in structured blocks — it reads block role directly. But this needs verification before v1.
## Related Decisions (Architecture Review 2026-07-05)
### #2 Meta Divergence — FIXED
`_rebuild_one_paper` now writes `meta["ocr_health_overall"]` from the rebuild health report, matching `postprocess_ocr_result`. Commit in feat/rebuild-speed.
### #3 Consumer Tools and Quality Signals
Deferred to role-override v1. Quality signals without a remediation path for the user cause frustration. Once role override gives users a way to fix mismatches, quality signals can gate downstream consumers (context, embed) meaningfully.
### #4 Health Report Richness
Keep all 60+ fields — they feed quality indicators when the module is wired in. No reduction.
### #5 PDF Lines Cache — REJECTED
Rebuild will continue to extract PDF lines from the source PDF each run. The cost is acceptable (~100ms per paper) and caching adds complexity for marginal gain. Role override will work with the current extraction.
### #6 Integration Tests
Deferred to role-override v1. Wire quality module first, then add 3 integration tests for the health → indicators → readiness flow.

View file

@ -503,7 +503,7 @@ def _rebuild_one_paper(vault: Path, key: str) -> dict:
write_decision_log(paper_root / "health" / "decision_log.jsonl", collect_decisions(structured))
return markdown
return markdown, health_report.get("overall", "unknown")
# ── Phase 5: indexes, version flags, write meta ──
def _phase5_finalize(
@ -511,6 +511,7 @@ def _rebuild_one_paper(vault: Path, key: str) -> dict:
structured: list[dict],
markdown: str,
span_meta_patch: dict,
health_overall: str = "unknown",
) -> None:
"""Rebuild indexes, apply version flags, write meta.json."""
from paperforge.worker.ocr_index import build_role_indexes, write_role_index
@ -544,6 +545,7 @@ def _rebuild_one_paper(vault: Path, key: str) -> dict:
_status, _err = validate_ocr_meta(paths_dict, meta)
meta["ocr_status"] = _status
meta["error"] = _err if _err else ""
meta["ocr_health_overall"] = health_overall
write_json(artifacts.meta_json, meta)
# ── Execute phases ──
@ -563,15 +565,13 @@ def _rebuild_one_paper(vault: Path, key: str) -> dict:
table_inventory = phase3_result["table_inventory"]
reader_payload = phase3_result["reader_payload"]
markdown = _phase4_render_health(
markdown, health_overall = _phase4_render_health(
structured, resolved, figure_inventory, table_inventory,
reader_payload, doc_structure, ocr_meta, source_pdf_path,
)
_phase5_finalize(resolved, structured, markdown, span_meta_patch)
_phase5_finalize(resolved, structured, markdown, span_meta_patch, health_overall=health_overall)
return {"key": key, "status": "ok"}
def _run_parallel_rebuild(vault: Path, keys: list[str], workers: int, checkpoint_dir: Path | None) -> list[dict]:
"""Run rebuild in parallel using a process pool."""
from concurrent.futures import ProcessPoolExecutor, as_completed