From 7ffe3ddd8452dcdb6065bd6acd62fc087acea02c Mon Sep 17 00:00:00 2001 From: Research Assistant Date: Fri, 19 Jun 2026 23:26:40 +0800 Subject: [PATCH] docs: realign OCR truth surface roles --- PROJECT-MANAGEMENT.md | 57 +++- .../current/ocr-v2-generalization-boundary.md | 6 +- .../ocr-v2-remaining-issues-2026-06-18.md | 7 +- project/current/ocr_rebuild_audit.md | 280 ++++++++++++++++++ 4 files changed, 345 insertions(+), 5 deletions(-) create mode 100644 project/current/ocr_rebuild_audit.md diff --git a/PROJECT-MANAGEMENT.md b/PROJECT-MANAGEMENT.md index 2cd960f3..ada78c2f 100644 --- a/PROJECT-MANAGEMENT.md +++ b/PROJECT-MANAGEMENT.md @@ -1,6 +1,6 @@ # OCR-v2 Project Management Log -> **Branch:** `ocr-v2` | **Base:** `master` | **Last Updated:** 2026-06-19 (blind audit complete — all gates done) +> **Branch:** `ocr-v2` | **Base:** `master` | **Last Updated:** 2026-06-19 (full rebuild audit — 452 papers, 18 dimensions) > **Rule:** Every step is documented with: What was done, Why it was done, What comes next. --- @@ -19,6 +19,19 @@ A structural redesign of PaperForge's OCR pipeline. The original pipeline committed to semantic roles too early (raw OCR labels -> guess -> rescue -> render). **ocr-v2 replaces this with anchor-first parsing:** structural signatures -> stable anchors -> zone inference -> late role resolution -> figure/table validation -> render. +### 0.3 Truth-Surface Handoff + +OCR-v2 readiness-gate work is complete. +Post-readiness rebuild hardening is now the active queue. + +Execution authority: +- Active queue: `project/current/ocr-v2-active-queue.md` +- Evidence source: `project/current/ocr_rebuild_audit.md` +- Broader architecture note: `project/current/ocr-v2-generalization-boundary.md` +- Historical readiness residuals: `project/current/ocr-v2-remaining-issues-2026-06-18.md` + +`PROJECT-MANAGEMENT.md` records the handoff but does not override the active queue. + --- ## 1. Architecture Overview @@ -1285,3 +1298,45 @@ For each paper: rebuilt pipeline, generated annotated pages, performed visual bl **Design spec:** `docs/superpowers/specs/2026-06-19-ocr-maintenance-ux-design.md` **Implementation plan:** `docs/superpowers/plans/2026-06-19-ocr-maintenance-ux-implementation.md` + +### 11.1 OCR Rebuild Full Audit (2026-06-19) + +**Problem:** 452 rebuild papers showed 5% green / 61% yellow / 34% red health scores. Needed comprehensive analysis of all failure modes before attempting fixes. + +**Audit scope:** 18 dimensions across all 452 rebuild papers: +- Health color distribution and issue combinations +- Figure matching (76% match rate, 6 root causes) +- Table matching (71% match rate, truncated caption issue) +- Orphan explosion (6021 orphans, 93% papers affected) +- Heading counting bug (only counts `section_heading`, ignores subsections) +- Footnote rendering (100% inline, 0 as footnote refs) +- Reference ordering (19% misorder from column sort) +- Table notes (0/190 linked, `note_block_ids` dead field) +- Supplementary figure namespace collision +- `figure_asset_count` misnomer (actually matched figure count) +- `references_found` too weak (raw_label check) +- `page_assets` group risk (can consume entire page assets) +- Table caption blockquote rendering +- Layout audit 91% fail signal +- unknown_structural content loss (27/1217 blocks, 0.3%) +- Noise classification (correct header/footer/page number) +- Body-to-supp caption distinction (structural heuristic) +- Health binary threshold problem (7 binary gates, no ratio) + +**Root causes confirmed at code level:** +- `ocr_health.py:118`: heading count ignores subsection/sub_subsection +- `ocr_render.py:1226`: `_SKIPPED_BODY_ROLES` missing `"footnote"` +- `ocr_render.py:1359`: table_caption rendered as blockquote +- `ocr_tables.py:251`: `note_block_ids` written but never consumed +- `ocr_health.py:130`: `figure_asset_count` = `len(matched_figures)` +- `ocr_health.py:123-125`: `references_found` triggered by single raw_label +- `ocr_render.py:604`: column sort swaps adjacent refs via x_left tiebreaker +- `ocr_tables.py:16`: `_TRUNCATED_TABLE_ONLY_PATTERN` rejects `"Table N"` format + +**Recommended fix priority (two batches):** +- **Batch 1 (P0/P1)**: footnote removal + table note consumption, table caption no blockquote, heading count fix, Table N geometry matching, supplementary namespace split, page_assets gate +- **Batch 2 (P2)**: reference numbering + sort, figure/table asset arbitration, health ratio/weighted, completeness 7-bucket outcome, references_found zone check + +**Artifacts:** `project/current/ocr_rebuild_audit.md` + +**Test status:** Existing 131 tests pass with partial code changes applied. `page_assets` changes not yet merged pending gate review. diff --git a/project/current/ocr-v2-generalization-boundary.md b/project/current/ocr-v2-generalization-boundary.md index 888c140d..e32773d0 100644 --- a/project/current/ocr-v2-generalization-boundary.md +++ b/project/current/ocr-v2-generalization-boundary.md @@ -1,8 +1,8 @@ # OCR-v2 Generalization Boundary -> Status: active broader architecture note -> Last updated: 2026-06-18 -> Source discussion: current OCR-v2 review thread +> Status: broader architecture note +> Readiness status: complete +> Active execution queue: `project/current/ocr-v2-active-queue.md` ## Problem Statement diff --git a/project/current/ocr-v2-remaining-issues-2026-06-18.md b/project/current/ocr-v2-remaining-issues-2026-06-18.md index bd884355..e0bf5532 100644 --- a/project/current/ocr-v2-remaining-issues-2026-06-18.md +++ b/project/current/ocr-v2-remaining-issues-2026-06-18.md @@ -1,7 +1,12 @@ # OCR-v2 Remaining Issues & Root Cause Analysis > Date: 2026-06-19 -> Baseline: all 5 readiness gates complete; blind audit passed +> Status: historical readiness-cycle residuals +> Readiness gates: complete +> Active queue: `project/current/ocr-v2-active-queue.md` + +This file records the residual state at the end of the readiness-gate cycle. +It is not the active rebuild-hardening queue. ## Active Readiness Residuals diff --git a/project/current/ocr_rebuild_audit.md b/project/current/ocr_rebuild_audit.md new file mode 100644 index 00000000..293290c2 --- /dev/null +++ b/project/current/ocr_rebuild_audit.md @@ -0,0 +1,280 @@ +# OCR Rebuild 全量审计报告 + +> 审计范围: 452 篇 rebuild 论文 | 2026-06-19 +> 状态: post-readiness audit surface +> 角色: evidence source for rebuild hardening +> 不改变: 已完成的 OCR-v2 readiness-gate 结论 + +本审计在 readiness-gate 完成后评估了全量语料的 rebuild 输出质量。 +它定义的是 post-readiness hardening 工作；不重新打开 OCR-v2 骨架决策。 + +--- + +## 1. HEALTH 总览 + +| 颜色 | 数量 | 比例 | +| ------ | ---- | ---- | +| GREEN | 21 | 5% | +| YELLOW | 276 | 61% | +| RED | 155 | 34% | + +### 1.1 Issue 前向模拟 + +若修复所有已识别问题后的预期分布: + +``` + 修前 修后 +GREEN: 5% → 28% +YELLOW: 61% → 62% +RED: 34% → 11% +``` + +剩余 48 篇红色主要是真实的图表匹配失败，不是阈值可救的。 + +### 1.2 健康判定逻辑 + +`ocr_health.py:172-193` — 7 个二元门: + +```python +if caption_without_media > 0: issues += 1 # 360 篇 +if media_without_caption > 0: issues += 1 # 371 篇 +if empty_tables > 0: issues += 1 # 113 篇 +if not abstract_found: issues += 1 # 15 篇 +if not references_found: issues += 1 # 6 篇 +if section_heading_count < 2: issues += 1 # 42 篇 +if formal_legend_gaps > 0: issues += 1 # 69 篇 +``` + +`> 0` 阈值对大小论文同等惩罚，无 ratio 或加权。 + +--- + +## 2. 数据总揽 + +### 2.1 Figure 匹配 + +``` +总 Caption: 2488 | Matched Assets: 1893 | 匹配率: 76% +Caption w/o media: 1107 (avg 2.4/篇, 360 篇) +Media w/o caption: 2306 (avg 5.1/篇, 371 篇) +P50: 0.75 | P10: 0.33 | 0 匹配: 13 篇 +``` + +### 2.2 Table 匹配 + +``` +总 Caption: 727 | Matched Assets: 513 | 匹配率: 71% +Tables w/o asset: 214 (29%) +P50: 1.00 | P10: 0.00 | 0 匹配: 35 篇 +``` + +### 2.3 Block 角色分布 (50 篇采样) + +``` +reference_item: 28.9% noise: 19.9% +body_paragraph: 19.6% unknown_structural: 5.0% +subsection_heading: 3.9% figure_asset: 3.6% +media_asset: 3.4% figure_inner_text: 2.5% +figure_caption: 2.1% section_heading: 1.7% +footnote: 1.4% table_caption: 0.6% +abstract_body: 0.5% +``` + +### 2.4 Subject 名称问题 + +`ocr_health.py:130`: +```python +figure_asset_count = len(figure_inventory.get("matched_figures", [])) +``` +这是 **matched figure 数量**，不是 asset 数量。命名误导维护判断。 + +--- + +## 3. 已确认的 BUG（代码级验证） + +### 3.1 Heading 计数只数 `section_heading` — CRITICAL + +`ocr_health.py:118`: +```python +section_heading_count = sum(1 for b in structured_blocks if b.get("role") == "section_heading") +``` + +不含 `subsection_heading`、`sub_subsection_heading`。 + +实测: +``` +23T2B8ZX: health=1, actual=9 (1 sec + 8 sub) +27MSS3VH: health=0, actual=8 (0 sec + 7 sub + 1 subsub) +``` + +**修复方向**: 三类 heading 总数。 + +### 3.2 Footnote 渲染为 body text — CRITICAL + +`ocr_render.py:1226-1237` — `_SKIPPED_BODY_ROLES` 不含 `"footnote"`。 + +所有 footnote 掉入 `else` 分支当纯文本 body paragraph 渲染。验证 (23T2B8ZX p=1): 3 块 footnote (`raw_label=footnote`) 分类正确，但全文当正文输出。 + +**修复方向**: footnote 从正文跳过；但 table note 要提前被 table inventory 消费。 + +### 3.3 Table caption 被渲染成 blockquote + +`ocr_render.py:1359-1374` — `table_caption` 分支输出: + +```md +> **Table X ...** +![[render/tables/table_XXX.md]] +``` + +导致 caption 重复、table note 漂浮。**修复方向**: fulltext 主流只输出 table embed；caption + note + crop 放进 `render/tables/table_XXX.md`。 + +### 3.4 `Table N` 短 caption 被 held / unmatched + +`ocr_tables.py:16-18` — `_TRUNCATED_TABLE_ONLY_PATTERN` 匹配 `"Table 1"` → `_is_weak_explicit_table_caption` → True → held。 + +69TA9S8W 类型（12 个 table 全是 `"Table N"`）全部 unmatched。**修复方向**: 强几何证据 + `raw_label=table` 时允许匹配。 + +### 3.5 `note_block_ids` 是 dead field + +`ocr_tables.py:251` 写入 `note_block_ids`，但 `ocr_objects.py` 和 `ocr_render.py` 都不消费。就算识别到 table note 也不会进 table card，不会从正文移除。 + +--- + +## 4. 结构性问题（需设计级修复） + +### 4.1 Supplementary figure namespace 碰撞 + +`_extract_figure_number("Figure S1")` 返回 `1`，`"Figure 1"` 也返回 `1`。在同一个整数 namespace 竞争。 + +当前 dedup "保留第一个" 能防止 S figure 偷正文 figure，但 Supplementary Figure 可能被直接丢掉。正确修法是 figure key 改为 tuple: + +```python +("main", 1) / ("supplementary", 1) / ("extended_data", 1) +``` + +对应输出: `figure_001` / `figure_s001` / `figure_ed001` + +### 4.2 Reference 排序 + +`ocr_render.py:604-619` — `_sort_blocks_by_column` 对所有 tail block 按 `(col, y, x)` 排序。19% 论文有 ref 错序。 + +修复应分层: +1. 解析 `[1]` / `1.` 编号，按编号递增 +2. 检测 ref 区是否双栏 → 双栏用 `page → col → y`，单栏用 `page → y` +3. 保留 OCR 原始相邻关系作 tie-breaker + +### 4.3 Figure/Table 资产所有权冲突 + +`ocr_tables.py:90` 当前已加了宽高/aspect 过滤，但宽图仍可能同时进入 figure 和 table 候选。需要统一 figure/table asset 仲裁机制，避免重复 orphan。 + +### 4.4 `page_assets` group 风险 + +当前（已修改未合并）给同页 ≥3 asset 建 `page_assets` group，固定 score 0.55，按先到先得匹配。风险：可能把同一页多个独立 figure 的 assets 一次性占掉。 + +应加 gate: +- 同页只有一个 formal legend +- 或无其他 figure/table caption +- 或 reader 层仅为 ASSET_GROUP 参考，不作为 strict match + +### 4.5 Figure completeness gap 根因不只是 caption_score + +`caption_score < 0.4` → `unmatched_legends` 理论上被 accounted。真正产生 gap 的是: +- dedup 后被静默丢弃的重复编号 caption +- `rejected_legends` 没进 outcome bucket +- body mention / supplementary caption 被过滤后仍被 completeness 统计 + +每个 numbered caption 必须有明确 outcome: `matched / held / ambiguous / unmatched / rejected / deduped_duplicate / supplementary` + +### 4.6 `references_found` 信号太弱 + +`ocr_health.py:123-125`: +```python +references_found = any( + b.get("role") in ("reference_heading", "reference_item") + or b.get("raw_label") == "reference_content" # ← 一个 label 就过 + for b in structured_blocks +) +``` + +更合理的是: `reference_zone.status == ACCEPT` 或 `reference_item_count >= 阈值`。 + +--- + +## 5. Table Notes 识别 + +### 5.1 当前状态 + +- Table caption 分类准确率 100% +- `note_block_ids` written but dead (不被任何 reader 消费) +- 30 篇采样中 32 个 `footnote` block 在 table 页面上，0 个关联到 table +- `render_table_object_markdown()` 只写 caption + 图片，无 note + +### 5.2 修复方向 + +1. `build_table_inventory` 存入 `note_texts` +2. `render_table_object_markdown` 消费 note 并输出 +3. 关联逻辑应包含: `footnote` role + `vision_footnote` raw_label + 紧邻下方短 text + +--- + +## 6. 非图表内容审计 + +### 6.1 `unknown_structural` (1217 块) + +``` +530 有文本: + - 185: journal metadata (correctly excluded) + - 306: genuinely unclassifiable short fragments + - 27: real body paragraphs misclassified (content loss, 0.3%) +687 空: images/decorations (correctly ignored) +``` + +仅 27 块真实丢失，`unknown_structural` 基本工作正常。 + +### 6.2 其他 + +- Body paragraph 完整性: OK +- Heading 风格分布: 正常 +- Fulltext 大小 vs raw: 无显著内容丢失 +- 短 body paragraph (<5 词): 56 个，多为页眉/DOI 碎片 + +--- + +## 7. 已完成的代码修改（待验证合并） + +| 修改 | 位置 | 状态 | +| --------------------------------------------------- | --------------------------- | -------- | +| Figure 编号 regex 支持 "Figure S.X" | `ocr_figures.py:15` | 已改 | +| Table 编号 regex 支持 "Table S1" | `ocr_tables.py:11` | 已改 | +| Table media_asset 加 raw_label/aspect 过滤 | `ocr_tables.py:90-97` | 已改 | +| Figure cluster_bbox 用全量 assets 重算 | `ocr_figures.py:1136-1137` | 已改 | +| Figure dedup: 双方有 asset 时保留第一个 | `ocr_figures.py:862-863` | 已改 | +| Figure 行列约束放宽 (0.35→0.6, gap 8%→12%) | `ocr_figures.py:436-447` | 已改 | +| Figure `page_assets` group type (≥3 per page) | `ocr_figures.py:424-427` | **待审** | +| Figure `page_assets` 评分 (0.55 + family bonus) | `ocr_figures.py:474-488` | **待审** | + +--- + +## 8. 建议修复优先级 + +### 第一批: 污染正文 / 静默丢失 + +| 优先级 | 项目 | 影响 | +| ------ | ---------------------------- | ------------------------- | +| P0 | footnote 从正文移除 + table 消费 | 24 篇正文 noise 消失 | +| P0 | table caption 取消 blockquote | 46% 论文格式修正 | +| P0 | heading 计数含三类 | ~30 篇 red→yellow | +| P0 | Table N 强几何匹配 | 69TA9S8W 类型全匹配 | +| P1 | supplementary namespace 拆分 | S figure 碰撞根治 (46 篇) | +| P1 | page_assets 加 gate | 防整页吞图 | +| P1 | table note 进 object render | note 归位，不再散落正文 | + +### 第二批: 匹配优化 / 评分改善 + +| 优先级 | 项目 | 影响 | +| ------ | ------------------------------ | ------------------------- | +| P2 | reference 编号解析 + col/y 排序 | 19% 错序修复 | +| P2 | figure/table asset 所有权仲裁 | 孤儿减半 | +| P2 | health 二元→ratio/weighted | 评分解释性提升 | +| P2 | completeness gap 补齐 7 个桶 | gap 不再误报 | +| P2 | `references_found` 改为 zone 检测 | 不再被 raw_label 欺骗 |