mirror of
https://github.com/lllin000/PaperForge.git
synced 2026-07-22 06:50:53 +00:00
fix: rebuild path writes meta.ocr_health_overall
_phase4_render_health now returns (markdown, health_overall), _phase5_finalize writes it to meta before write_json. Also adds ADR with architecture review decisions: - #2 Meta divergence — fixed - #3 Quality signals for consumers — deferred to role-override v1 - #4 Health report richness — keep all fields - #5 PDF lines cache — rejected - #6 Integration tests — deferred to role-override v1
This commit is contained in:
parent
d50ea4631b
commit
cd352cb007
2 changed files with 68 additions and 5 deletions
63
docs/adr/2026-07-05-role-override-seam.md
Normal file
63
docs/adr/2026-07-05-role-override-seam.md
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
# ADR: Block Role Override Seam
|
||||
|
||||
**Date:** 2026-07-05
|
||||
**Status:** Draft (next version)
|
||||
|
||||
## Context
|
||||
|
||||
OCR figure/table matching is never perfect. Rather than building a GUI for manual adjustment, we can insert a **data seam** between structured block building and render: allow an agent or user to override a block's `role` via a sidecar patch file, then let the existing render pipeline consume the corrected roles.
|
||||
|
||||
## Decision
|
||||
|
||||
Introduce `override_role_patch.json` (sidecar per paper):
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"overrides": [
|
||||
{"block_id": "p7_b3", "role": "figure_caption"},
|
||||
{"block_id": "p9_b1", "role": "section_heading"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Rebuild Phase 3→4 reads this file and applies overrides to structured blocks in memory before render. No changes to matching logic, figure inventory, or render.
|
||||
|
||||
### Scope (v1)
|
||||
|
||||
**Figure caption matching only.** Not text flow correction (ref zone contamination, heading misplacement). Rationale:
|
||||
- Agent can 100% confirm a caption mismatch from fulltext + cropped image
|
||||
- User can describe "page X 'Figure Y.' text should be a figure caption"
|
||||
- Text flow correction requires position understanding, not just role change
|
||||
|
||||
### Block Identification
|
||||
|
||||
`block_id` (`p{N}_{label}`) is stable across rebuild. For user→agent handoff, natural language ("page 7, the 'Figure 3.' paragraph") is resolved to `block_id` by matching `page` + block `text` prefix.
|
||||
|
||||
### Consumer Tools and Quality Signals
|
||||
|
||||
Deferred. Quality indicators (`build_quality_indicators`, `evaluate_readiness`) remain unwired until the override seam is implemented — quality signals without a remediation path cause user frustration.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Override + rebuild cycle: does a rebuild re-derive roles and then apply override on top? Yes — override patches after structured block building, not instead of it.
|
||||
- Backward compat: missing `override_role_patch.json` = no-op.
|
||||
- Figure inventory consistency: if a block's role changes to `figure_caption`, does the figure inventory need updating? Current render doesn't read figure inventory for captions embedded in structured blocks — it reads block role directly. But this needs verification before v1.
|
||||
|
||||
|
||||
## Related Decisions (Architecture Review 2026-07-05)
|
||||
|
||||
### #2 Meta Divergence — FIXED
|
||||
`_rebuild_one_paper` now writes `meta["ocr_health_overall"]` from the rebuild health report, matching `postprocess_ocr_result`. Commit in feat/rebuild-speed.
|
||||
|
||||
### #3 Consumer Tools and Quality Signals
|
||||
Deferred to role-override v1. Quality signals without a remediation path for the user cause frustration. Once role override gives users a way to fix mismatches, quality signals can gate downstream consumers (context, embed) meaningfully.
|
||||
|
||||
### #4 Health Report Richness
|
||||
Keep all 60+ fields — they feed quality indicators when the module is wired in. No reduction.
|
||||
|
||||
### #5 PDF Lines Cache — REJECTED
|
||||
Rebuild will continue to extract PDF lines from the source PDF each run. The cost is acceptable (~100ms per paper) and caching adds complexity for marginal gain. Role override will work with the current extraction.
|
||||
|
||||
### #6 Integration Tests
|
||||
Deferred to role-override v1. Wire quality module first, then add 3 integration tests for the health → indicators → readiness flow.
|
||||
|
|
@ -503,7 +503,7 @@ def _rebuild_one_paper(vault: Path, key: str) -> dict:
|
|||
|
||||
write_decision_log(paper_root / "health" / "decision_log.jsonl", collect_decisions(structured))
|
||||
|
||||
return markdown
|
||||
return markdown, health_report.get("overall", "unknown")
|
||||
|
||||
# ── Phase 5: indexes, version flags, write meta ──
|
||||
def _phase5_finalize(
|
||||
|
|
@ -511,6 +511,7 @@ def _rebuild_one_paper(vault: Path, key: str) -> dict:
|
|||
structured: list[dict],
|
||||
markdown: str,
|
||||
span_meta_patch: dict,
|
||||
health_overall: str = "unknown",
|
||||
) -> None:
|
||||
"""Rebuild indexes, apply version flags, write meta.json."""
|
||||
from paperforge.worker.ocr_index import build_role_indexes, write_role_index
|
||||
|
|
@ -544,6 +545,7 @@ def _rebuild_one_paper(vault: Path, key: str) -> dict:
|
|||
_status, _err = validate_ocr_meta(paths_dict, meta)
|
||||
meta["ocr_status"] = _status
|
||||
meta["error"] = _err if _err else ""
|
||||
meta["ocr_health_overall"] = health_overall
|
||||
write_json(artifacts.meta_json, meta)
|
||||
|
||||
# ── Execute phases ──
|
||||
|
|
@ -563,15 +565,13 @@ def _rebuild_one_paper(vault: Path, key: str) -> dict:
|
|||
table_inventory = phase3_result["table_inventory"]
|
||||
reader_payload = phase3_result["reader_payload"]
|
||||
|
||||
markdown = _phase4_render_health(
|
||||
markdown, health_overall = _phase4_render_health(
|
||||
structured, resolved, figure_inventory, table_inventory,
|
||||
reader_payload, doc_structure, ocr_meta, source_pdf_path,
|
||||
)
|
||||
|
||||
_phase5_finalize(resolved, structured, markdown, span_meta_patch)
|
||||
|
||||
_phase5_finalize(resolved, structured, markdown, span_meta_patch, health_overall=health_overall)
|
||||
return {"key": key, "status": "ok"}
|
||||
|
||||
def _run_parallel_rebuild(vault: Path, keys: list[str], workers: int, checkpoint_dir: Path | None) -> list[dict]:
|
||||
"""Run rebuild in parallel using a process pool."""
|
||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
||||
|
|
|
|||
Loading…
Reference in a new issue