mirror of
https://github.com/lllin000/PaperForge.git
synced 2026-07-22 17:00:23 +00:00
feat: figure merge clustering spec + fixes for caption dedup and structural gate
- Add global-distance-clustering figure merge design spec - Fix ocr_roles.py: body_zone body_paragraph no longer promoted to figure_caption_candidate (prevents body mentions from entering legend pipeline) - Fix ocr_document.py: preserve subsection_heading, reference_item, table_caption from structural-gate demotion - Update PROJECT-MANAGEMENT.md with edge-case analysis - Add ocr-vision-audit-master.md for CODE vs VISION separation - Update SKILL.md dispatch rules with trust boundaries
This commit is contained in:
Research Assistant
parent
aef0cc0e78
commit
da2e018b19
10 changed files with 1858 additions and 115 deletions
|
|
@ -15,10 +15,86 @@ Use this when the task is to inspect OCR truth, classify failures, and write aud
|
|||
|
||||
## Scripts
|
||||
|
||||
- Audit helper: `scripts/ocr_truth_audit.py`
|
||||
- `scripts/ocr_truth_audit.py` — artifact generator (annotated pages, summaries, findings)
|
||||
- `scripts/audit_helpers.py` — structured data maps for vision agents (figure, frontmatter, reference, body)
|
||||
- `scripts/verify_review_coverage.py` — coverage verification
|
||||
- `scripts/diff_audit.py` — diff audit between truth and pipeline
|
||||
|
||||
## Invariant
|
||||
|
||||
- Truth first.
|
||||
- Do not start by authoring expectations.
|
||||
- Establish block-level truth from page visuals and artifacts before comparing pipeline behavior.
|
||||
|
||||
## Agent Dispatch Rules (CRITICAL — read before any audit task)
|
||||
|
||||
The #1 source of audit errors is using the wrong tool for the question.
|
||||
Follow these rules unconditionally:
|
||||
|
||||
### When to dispatch `vision` subagent (MUST):
|
||||
|
||||
The workflow in `workflows/ocr-truth-audit.md` Step 6 tells you the exact commands.
|
||||
Summary:
|
||||
|
||||
| Audit domain | Data recipe | Vision template |
|
||||
|-------------|-------------|----------------|
|
||||
| Figures | `audit_helpers.py --recipe figure_map` | atoms/ocr-vision-audit-templates.md — Template 1 |
|
||||
| Frontmatter | `audit_helpers.py --recipe frontmatter_map` | Template 2 |
|
||||
| References | `audit_helpers.py --recipe reference_map` | Template 3 |
|
||||
| Body text | `audit_helpers.py --recipe body_text_map` | Template 4 |
|
||||
|
||||
**Procedure:** (1) Run the recipe → get JSON. (2) Read the template. (3) Paste JSON into template. (4) Dispatch as `vision` subagent prompt.
|
||||
|
||||
### Forbidden agent patterns:
|
||||
|
||||
- `general` subagent claiming to "see" an annotated page. **Only `vision` can view images.**
|
||||
- Concluding figure matching from `figure_table_ownership_summary.json` alone. **Must cross-check with source `figure_inventory.json` AND visual truth.**
|
||||
- Reporting `reference_intrusion_candidates` as errors without checking block roles.**noise blocks in `reference_zone` are expected.**
|
||||
- "The block has 0 candidates so figure is on another page" without scanning `page_{N-1}_index.json` and `page_{N+1}_index.json`.
|
||||
|
||||
### Finding trust boundaries (CRITICAL):
|
||||
|
||||
The auto-generated findings in `audit_report.json` contain substantial noise.
|
||||
**Only these [CODE] checks are reliable — all others need [VISION]:**
|
||||
|
||||
| Finding category | Trust? | Why |
|
||||
|-----------------|--------|-----|
|
||||
| `role_mismatch` (from diff_audit.py) | ✅ [CODE] | Direct string comparison, 100% |
|
||||
| `zone_mismatch` (from block_review.jsonl) | ✅ [CODE] | Direct string comparison, 100% |
|
||||
| `render_mapping_error` — non-noise blocks | ✅ [CODE] | Substring match, exclude noise |
|
||||
| `unmatched_asset_count > 0` | ✅ [CODE] | deterministic count from inventory |
|
||||
| `coverage_check status` | ✅ [CODE] | direct comparison |
|
||||
| `reference_span_audit status` | ✅ [CODE] | deterministic |
|
||||
| `same_page_boundary_error` | ❌ NEVER | Page-level heuristic, no actionable info |
|
||||
| `reference_span_error` (audit_report.json) | ⚠️ Partially | Span may be too wide; need vision to verify boundary |
|
||||
| `object_ownership_error` (audit_report.json) | ❌ NEVER | Misses unmatched assets entirely |
|
||||
| `render_mapping_error` — noise blocks | ❌ NEVER | noise should not be in fulltext |
|
||||
| `image_quality`, `font`, `color`, `typography` | ❌ [VISION] only | No metadata available |
|
||||
|
||||
### Per-role audit strategies:
|
||||
|
||||
Full methodology for each of the 28 canonical roles is in `atoms/ocr-role-audit-strategies.md`.
|
||||
Each role is classified into one of 4 tiers:
|
||||
|
||||
| Tier | Method | Roles |
|
||||
|------|--------|-------|
|
||||
| A | Vision-required | `figure_asset`, `media_asset`, `figure_caption`, `figure_caption_candidate`, `table_caption`, `table_asset`, `table_html`, `structured_insert` |
|
||||
| B | Data-primary, vision-confirm | `body_paragraph`, `section_heading`, `subsection_heading`, `reference_item`, `reference_heading`, `backmatter_heading`, `backmatter_body` |
|
||||
| C | Data-sufficient | `paper_title`, `authors`, `affiliation`, `abstract_heading`, `abstract_body`, `keywords`, `frontmatter_support`, `frontmatter_noise`, `footnote` |
|
||||
| D | Structural-only | `tail_candidate_body`, `backmatter_boundary_*`, `ocr_raw_error`, `unknown_structural`, `page_header`, `page_footer`, `noise` |
|
||||
|
||||
### Deep per-role + quality audit:
|
||||
|
||||
**`atoms/ocr-vision-audit-master.md`** extends all 4 templates with:
|
||||
- Per-role analysis covering role/zone/reference AND quality (typography, image, table)
|
||||
- Cross-reference pathways linking all audit data sources for any block_id
|
||||
- Chart-type routing to `paperforge/skills/paperforge/atoms/chart-reading/*.md`
|
||||
- Full-page typography sweep (font consistency, alignment, orphans/widows, heading hierarchy)
|
||||
- Output format for block_review.jsonl with optional `quality_checks` and `page_typography` fields
|
||||
|
||||
After the 4 recipe-driven vision passes in Step 6, run the master atom:
|
||||
```bash
|
||||
# Read the master atom before dispatching per-role vision
|
||||
cat .opencode/skills/paperforge-development/atoms/ocr-vision-audit-master.md
|
||||
```
|
||||
It contains the per-role data-cross-ref + quality-check methodology. Paste the relevant section into each vision subagent prompt depending on which role set the agent is reviewing.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,77 @@
|
|||
# OCR Per-Role Audit Strategies
|
||||
|
||||
> **Use this as a quick reference** for deciding how to inspect any role.
|
||||
> For actual audit work, use `scripts/audit_helpers.py` recipes + `ocr-vision-audit-templates.md`.
|
||||
> Do NOT write custom Python for per-role checks — it's already built.
|
||||
|
||||
## Tier Classification
|
||||
|
||||
| Tier | Method | When |
|
||||
|------|--------|------|
|
||||
| A | **Vision-required** | Must dispatch `vision` subagent. Data alone insufficient. |
|
||||
| B | **Data-primary, vision-confirm** | Read data first. Dispatch vision only when data flags an issue. |
|
||||
| C | **Data-sufficient** | JSON analysis is enough. Vision only for ordering/layout ambiguity. |
|
||||
| D | **Structural-only** | Read from block trace / document structure. No vision needed. |
|
||||
|
||||
## Role → Tier Mapping
|
||||
|
||||
### Tier A — Vision-Required
|
||||
|
||||
| Role | Why vision is needed |
|
||||
|------|---------------------|
|
||||
| `figure_asset` | Is the box covering an actual image? Are sub-panels merged? |
|
||||
| `media_asset` | Is this a figure, a table, or a phantom block? |
|
||||
| `figure_caption` | Is it a real caption, sub-panel label, or body mention? Where is its image? |
|
||||
| `figure_caption_candidate` | Should it be confirmed or demoted? |
|
||||
| `figure_inner_text` | Is it inside a figure (panel label) or separate text? |
|
||||
| `table_caption` | Same as figure_caption. Cross-page? |
|
||||
| `table_caption_candidate` | Should it be confirmed? |
|
||||
| `table_asset` | Is this actually a table image? |
|
||||
| `table_html` | Is the HTML table rendering correctly? |
|
||||
| `structured_insert` | Is this a callout box or mislabeled body text? |
|
||||
| `structured_insert_candidate` | Should it be confirmed? |
|
||||
| `non_body_insert` | What is this element visually? |
|
||||
|
||||
### Tier B — Data-Primary, Vision-Confirm
|
||||
|
||||
| Role | Data check | When to add vision |
|
||||
|------|-----------|-------------------|
|
||||
| `body_paragraph` | zone=body_zone? text non-empty? | zone wrong, text empty, or two-column ambiguity |
|
||||
| `section_heading` | numbering depth, heading absorption (two levels in one block) | verify visually if absorption detected |
|
||||
| `subsection_heading` | same as section_heading | same |
|
||||
| `sub_subsection_heading` | same | same |
|
||||
| `reference_heading` | in reference_zone? before first reference_item? | transition page boundary |
|
||||
| `reference_item` | in reference_zone? correct numbering pattern? | real intrusions or numbering gaps |
|
||||
| `backmatter_heading` | after last reference? in backmatter_zone? | if inside reference_zone |
|
||||
| `backmatter_body` | same | same |
|
||||
|
||||
### Tier C — Data-Sufficient
|
||||
|
||||
`paper_title`, `authors`, `affiliation`, `frontmatter_support`, `frontmatter_noise`,
|
||||
`abstract_heading`, `abstract_body`, `keywords`, `footnote`
|
||||
|
||||
These are on page 1 (or page N for footnotes) at known positions with known patterns.
|
||||
Read `page_001_index.json`. Vision only for ordering ambiguity or badge mislabel suspicion.
|
||||
|
||||
### Tier D — Structural-Only
|
||||
|
||||
`tail_candidate_body`, `backmatter_boundary_candidate`, `backmatter_boundary_heading`,
|
||||
`ocr_raw_error`, `unknown_structural`, `page_header`, `page_footer`, `noise`
|
||||
|
||||
Read from `blocks.structured.jsonl` (text, span_metadata, _ocr_raw_status) or
|
||||
`document_structure.json` (tail order, boundaries). No vision needed.
|
||||
|
||||
## Common Failure Patterns (quick reference)
|
||||
|
||||
| Pattern | Signal in data | Confirmation |
|
||||
|---------|---------------|-------------|
|
||||
| Sub-panels not merged | N asset blocks, 1 fig with < N assets | Vision: are they panels of same figure? |
|
||||
| Caption sandwiched between figs | close_asset_tie with above+below | Vision: which figure owns the caption? |
|
||||
| Cross-page caption | no_asset_match, prev/next page has orphan assets | Vision: is image on adjacent page? |
|
||||
| Sub-panel heading as caption | figure_caption with text like "Musculoskeletal conditions..." | Vision: is this a panel heading? |
|
||||
| Truncated legend as standalone | "Fig. 6" only, long caption below | Vision: composite figure with shared caption? |
|
||||
| Table mislabeled as figure | media_asset with raw_label=table | Vision: is this a table or figure? |
|
||||
| Body text as noise | noise with text_preview > 30 chars, not at page edge | Vision: is this body text? |
|
||||
| Heading absorption | section_heading text has "2. ... 2.1. ..." | Vision: two heading levels merged? |
|
||||
| Badge as authors | authors text = "HIGHLIGHTED PAPER" | Vision: is it a badge or authors? |
|
||||
| Reference zone intrusion | body_paragraph in reference_zone | Vision: verify boundary |
|
||||
|
|
@ -0,0 +1,286 @@
|
|||
# OCR Vision Audit Master — CODE vs VISION 严格分离
|
||||
|
||||
> **原则：** 每个检查标注来源。
|
||||
> `[CODE]` = 从 JSON 文件或 PDF 元数据精确提取，无需视觉判断。
|
||||
> `[VISION]` = 必须看 `page_NNN.png`，代码没有所需数据。
|
||||
>
|
||||
> 字体属性（family、size、weight、color、italic）在 `blocks.structured.jsonl` 的
|
||||
> `span_metadata` / `span_signature` 中，代码可直接提取。不要用 vision 做字体检查。
|
||||
|
||||
---
|
||||
|
||||
## 0. 数据源权限
|
||||
|
||||
| 文件 | `[CODE]` 可读 | `[VISION]` 需要 |
|
||||
|------|-------------|----------------|
|
||||
| `page_NNN.png` | ❌ | ✅ |
|
||||
| `page_NNN_index.json` | ✅ role, zone, bbox, text | — |
|
||||
| `block_coverage_summary.json` | ✅ role, raw_label, zone, bbox | — |
|
||||
| `blocks.structured.jsonl` (OCR 源) | ✅ **span_metadata (font/color/size), span_signature, style_family** | — |
|
||||
| `block_review.jsonl` | ✅ truth_role, truth_zone, truth_reference_membership | — |
|
||||
| `figure_table_ownership_summary.json` | ✅ matched/ambiguous/unmatched 计数 | ⚠️ 验证匹配正确性 |
|
||||
| `fulltext_block_mapping_summary.json` | ✅ found_in_fulltext（排除 noise 后） | — |
|
||||
| `reference_span_audit.json` | ✅ inside/outside | ⚠️ 边界验证 |
|
||||
| `coverage_check.json` | ✅ 100% 可靠 | — |
|
||||
|
||||
---
|
||||
|
||||
## 1. `[CODE]` 自动化可信任检查
|
||||
|
||||
以下从 JSON 数据直接提取，**不看图片**，结果决定性的。
|
||||
|
||||
### 1.1 Role 一致性 — block_review.jsonl + diff_audit.py
|
||||
|
||||
```yaml
|
||||
输入: block_review.jsonl 的 truth_role vs _current_role
|
||||
changed_blocks_after_fallback.json
|
||||
方法: 字符串直接比较
|
||||
信任度: 100%
|
||||
输出: 每个 role mismatch 的 {block_id, pipeline_role, truth_role}
|
||||
```
|
||||
|
||||
### 1.2 Zone 一致性 — block_review.jsonl
|
||||
|
||||
```yaml
|
||||
输入: block_review.jsonl 的 truth_zone vs _current_zone
|
||||
方法: 字符串直接比较
|
||||
信任度: 100%
|
||||
```
|
||||
|
||||
### 1.3 渲染完整性 — fulltext_block_mapping_summary.json
|
||||
|
||||
```yaml
|
||||
输入: fulltext_block_mapping_summary.json
|
||||
found_in_fulltext=false 且 role ∉ {noise, figure_inner_text, frontmatter_noise}
|
||||
方法: 前80字符子在渲染全文中的存在性检查
|
||||
信任度: 高（排除 noise 后），注意子串匹配微小误报可能
|
||||
输出: 真正在 fulltext 中丢失的 body_paragraph / reference_item / backmatter_body 等
|
||||
```
|
||||
|
||||
### 1.4 图/表未匹配资产 — figure_table_ownership_summary.json
|
||||
|
||||
```yaml
|
||||
输入: figures.unmatched_asset_count, tables.unmatched_asset_count
|
||||
方法: 直接计数
|
||||
信任度: 100%
|
||||
输出: 有未匹配资产的 paper，未匹配数量
|
||||
注意: 不判断为什么。vision 看具体原因。
|
||||
```
|
||||
|
||||
### 1.5 覆盖完整性 — coverage_check.json
|
||||
|
||||
```yaml
|
||||
输入: coverage_check.json 的 status, missing_block_ids
|
||||
信任度: 100%
|
||||
```
|
||||
|
||||
### 1.6 字体一致性 — blocks.structured.jsonl
|
||||
|
||||
从每个 block 的 `span_signature` 和 `span_metadata` 提取：
|
||||
|
||||
```yaml
|
||||
输入: blocks.structured.jsonl 中所有 text block 的 span_signature
|
||||
(font_family_norm, font_size_bucket, bold, italic)
|
||||
方法: 按 role 分组，检查组内一致性
|
||||
信任度: PDF 直接提取，100% 精确
|
||||
```
|
||||
|
||||
**具体检查项：**
|
||||
|
||||
```
|
||||
A. 同一 role 内 font_family_norm 一致吗？
|
||||
- body_paragraph 应该全是同一字体（如 MyriadPro-Light）
|
||||
- 如果某个 body_paragraph 的 font_family 与其他不同 → 混用字体
|
||||
|
||||
B. 同一 role 内 font_size_bucket 一致吗？
|
||||
- 所有 body_paragraph 的 font_size_bucket 应相同
|
||||
- 差异 > 1pt → 字号不一致
|
||||
|
||||
C. 标题层次递减正确吗？
|
||||
- 按 page 分组，比较 section_heading / subsection_heading 的 font_size
|
||||
- H1 > H2 > H3 在字体大小上应有明显递减
|
||||
|
||||
D. 同一 block 内混用字体吗？
|
||||
- span_metadata 有多个 entries 且 font 不同 → 可能是加粗/斜体标记混入
|
||||
- 检查是否合理（如关键词加粗）还是问题（同一字体错误嵌入）
|
||||
|
||||
E. 加粗/斜体使用一致吗？
|
||||
- 所有 section_heading 的 bold=true？
|
||||
- 所有 body_paragraph 的 bold=false？
|
||||
```
|
||||
|
||||
### 1.7 排版布局 — blocks.structured.jsonl + bbox
|
||||
|
||||
从每个 block 的 `bbox` 坐标提取：
|
||||
|
||||
```yaml
|
||||
输入: blocks.structured.jsonl 的 bbox [x0, y0, x1, y1]
|
||||
page_width, page_height
|
||||
```
|
||||
|
||||
**具体检查项：**
|
||||
|
||||
```
|
||||
A. 对齐方式检测：
|
||||
每页取 body_paragraph 的 x0, x1 坐标：
|
||||
- 如果所有 x0 相同 → 左对齐 (ragged right)
|
||||
- 如果所有 x1 相同且 x0 不同 → 右对齐
|
||||
- 如果 x0 和 x1 都跨block一致 → 居中对齐?
|
||||
- 如果 x0 都相同但 x1 不同（各行不同长度）+ x1 抵边 → 两端对齐(justified)
|
||||
- 跨页一致吗？
|
||||
|
||||
B. 列检测：
|
||||
每页对 body_paragraph 的 x-center 做聚类：
|
||||
- 1个聚类 → 单栏
|
||||
- 2个聚类 → 双栏（和 expected 一致？）
|
||||
- 聚类中心 x-center 分布均匀？
|
||||
|
||||
C. 页边距一致性：
|
||||
每页 body_paragraph 的 min(x0) 应大致相同（左页边距）
|
||||
每页 body_paragraph 的 max(x1) 应大致相同（右页边距）
|
||||
跨页比较左/右边距是否一致
|
||||
|
||||
D. 行间距检测：
|
||||
同一列相邻 body_paragraph 的 y0 差值减去前一个 block 的高度
|
||||
= 段间距
|
||||
跨 block 段间距一致吗？
|
||||
|
||||
E. 孤行/孤段检测：
|
||||
如果一页上某个 body_paragraph 的 height < 正常行高的1.5倍
|
||||
且它在页面顶部或底部 → orphan 或 widow
|
||||
```
|
||||
|
||||
### 1.8 引用区间完整性 — reference_span_audit.json
|
||||
|
||||
```yaml
|
||||
输入: reference_span_audit.json 的 status, inside_block_ids
|
||||
方法: status == "HOLD" → 区间未通过验证
|
||||
inside_block_ids 中 role ≠ reference_item/heading → 可能 intrusion
|
||||
信任度: 状态可靠，intrusion 需要 vision 确认
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. `[VISION]` 必须看图片
|
||||
|
||||
以下检查代码**没有所需元数据**，必须看 `page_NNN.png`。
|
||||
|
||||
### 2.1 图/表视觉验证
|
||||
|
||||
```
|
||||
- 子面板覆盖：一个 figure 的 asset 列表覆盖了所有视觉上的子面板吗？
|
||||
- 子面板合并：2x2 的 grid 被合并为 1 个 figure 还是 4 个独立的？
|
||||
- caption 位置：在图上方/下方？跨页？
|
||||
- caption 语义：文字描述的内容与图上数据一致吗？（代码读不了图内容）
|
||||
- 表布局：HTML 表渲染正确？图片表完整无裁剪？
|
||||
- 角色误标：media_asset 实际上是 table？figure_caption 实际上是 section_heading？
|
||||
```
|
||||
|
||||
### 2.2 图像质量
|
||||
|
||||
```
|
||||
- 分辨率：模糊/锯齿/像素化？
|
||||
- 仪器截屏：显微镜软件、流式细胞仪输出？
|
||||
- 对比度：前景/背景区分足够？
|
||||
```
|
||||
|
||||
### 2.3 颜色可达性（仅限图片中的颜色）
|
||||
|
||||
```
|
||||
- 图片本身的配色（热图、荧光图等）
|
||||
- 是否只依赖红/绿区分条件？
|
||||
- 注意：文本颜色在 span_metadata.color 中，用代码检查
|
||||
```
|
||||
|
||||
### 2.4 统计完整性（图中数据 vs caption 描述）
|
||||
|
||||
```
|
||||
- 图中显示的误差棒/显著性标记与 caption 描述一致？
|
||||
- N 值标注了吗？
|
||||
- p 值标注与图例对应？
|
||||
- 热图 colorbar 标注完整？
|
||||
```
|
||||
|
||||
### 2.5 Chart 类型识别 + 深度分析
|
||||
|
||||
```
|
||||
识别 chart 类型 → 路由到 paperforge/skills/paperforge/atoms/chart-reading/{TYPE}.md
|
||||
然后用对应指南做深度检查。
|
||||
```
|
||||
|
||||
### 2.6 引用区间边界验证
|
||||
|
||||
```
|
||||
- 参考页第一页：body 在 "References" 前结束了吗？
|
||||
- 过渡页：body 和 ref 没有交错？
|
||||
- reference_zone 外的 reference_item 是否真的是引用条目？
|
||||
- intrusion candidate 是真的还是 span 边界误标？
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 不可信自动化结果（不用）
|
||||
|
||||
以下数据不作为 finding，只做导航：
|
||||
|
||||
| 数据 | 问题 | 替代用法 |
|
||||
|------|------|---------|
|
||||
| `same_page_boundary_error` (audit_report.json) | 纯页面角色计数，正常布局也被标 | vision 优先审这些页 |
|
||||
| `object_ownership_error` (audit_report.json) | 只报 ambiguous/unresolved，漏了大批量 unmatched_assets | 改为读 unmatched_asset_count |
|
||||
| noise block 的 render_mapping_error | noise 不该在 fulltext 中 | 排除 noise 后看 |
|
||||
| vision 做字体/颜色检查 | 不准，PDF 元数据更精确 | 用 blocks.structured.jsonl |
|
||||
|
||||
---
|
||||
|
||||
## 4. 输出格式
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"block_id": "p3:12",
|
||||
"page": 3,
|
||||
"review_status": "reviewed",
|
||||
"truth_role": "figure_asset",
|
||||
"truth_zone": "display_zone",
|
||||
"truth_reference_membership": "outside",
|
||||
"evidence": {"annotated_page": "annotated_pages/page_003.png", "method": "visual+bbox"},
|
||||
"short_reason": "Fig. 2A MRI image — role and zone correct",
|
||||
"vision_checks": {
|
||||
"subpanels_merged": "A (2 of 2 panels matched)",
|
||||
"image_quality": "OK",
|
||||
"chart_type": null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`vision_checks` 可选，只记录 vision 实际检查的维度，未检查的用 `null`。
|
||||
|
||||
---
|
||||
|
||||
## 5. 工作流程
|
||||
|
||||
```
|
||||
1. 跑 [CODE]
|
||||
diff_audit.py → role/zone mismatch 列表
|
||||
fulltext_block_mapping → 真实渲染丢失（排除 noise）
|
||||
figure_table_ownership → unmatched_asset_count
|
||||
coverage_check → 缺失 block
|
||||
blocks.structured.jsonl → 字体一致性、排版异常
|
||||
|
||||
2. 决定 vision 目标
|
||||
优先:
|
||||
- code 发现的 role mismatch block
|
||||
- unmatched_asset > 0 的 figure
|
||||
- body/reference 在 fulltext 丢失的 block
|
||||
- 参考页第一页
|
||||
- figure-heavy 页
|
||||
- coverage FAIL 的 page
|
||||
|
||||
3. 逐 block vision [VISION]
|
||||
对选中 block:
|
||||
看 annotated page → 确认角色/区域
|
||||
如果是 figure → 子面板、质量、chart 路由
|
||||
|
||||
4. 输出
|
||||
block_review.jsonl 只写 vision 确认过的 block
|
||||
不编造未看内容
|
||||
```
|
||||
|
|
@ -0,0 +1,251 @@
|
|||
# OCR Vision Audit Templates
|
||||
|
||||
> **Purpose:** Structured prompts for vision agents. Each template pairs with a `--recipe` from `audit_helpers.py`.
|
||||
> The agent workflow: (1) run `audit_helpers.py --recipe figure_map` to get JSON → (2) read this template → (3) dispatch vision with data + template → (4) vision reports findings.
|
||||
>
|
||||
> **After running these 4 templates**, read `ocr-vision-audit-master.md` for the deep quality pass:
|
||||
> per-role cross-reference pathways, typography checks, chart-type routing to `paperforge/skills/paperforge/atoms/chart-reading/*.md`, figure/table quality checks.
|
||||
|
||||
---
|
||||
|
||||
## Template 1: Figure Verification
|
||||
|
||||
**Data source:** `audit_helpers.py --recipe figure_map --key {KEY}`
|
||||
|
||||
**Vision agent prompt (replace {KEY}, {DATA_JSON}, {PAGE_LIST}):**
|
||||
|
||||
```
|
||||
You are auditing figure matching for paper {KEY}. Below is a structured summary
|
||||
of every figure the pipeline detected — matched, ambiguous, and unresolved.
|
||||
|
||||
FIGURE DATA:
|
||||
{DATA_JSON}
|
||||
|
||||
Annotated pages are at:
|
||||
audit/{KEY}/annotated_pages/page_NNN.png (color-coded block rectangles)
|
||||
audit/{KEY}/annotated_pages/page_NNN_index.json (maps label numbers to block IDs)
|
||||
|
||||
For EACH figure in the data, look at the corresponding annotated page(s) and answer:
|
||||
|
||||
### For every figure:
|
||||
|
||||
1. VISUAL CONFIRMATION: Find the colored rectangle(s) for this figure on the annotated page.
|
||||
- The label numbers on the PNG correspond to `block_id` in the data.
|
||||
- The index JSON maps label_number → block_id, role, bbox.
|
||||
- TRACE: Find the caption block first (its label number), then find each asset block.
|
||||
|
||||
2. ASSET COVERAGE: Does the set of colored boxes cover ALL visible sub-panels of this figure?
|
||||
- Count how many sub-panels the figure has (from the colored boxes on the page).
|
||||
- Compare with `assets` list in the data. Are any sub-panels MISSING? Are any EXTRA?
|
||||
- If data says N assets but visually you see M sub-panels: report the discrepancy.
|
||||
|
||||
3. CAPTION CORRECTNESS: Is the caption text (shown in the data) visually the correct caption for this figure?
|
||||
- Is the caption block positioned correctly relative to the figure (below, above, or side-by-side)?
|
||||
- Does the caption text match what you see on the page?
|
||||
- Is it a REAL caption (description of the figure), not a section heading or body text?
|
||||
|
||||
4. SUB-PANEL MERGING: Are sub-panels of the SAME figure correctly grouped?
|
||||
- If the page has a 2x2 grid of images with ONE shared caption, they should be ONE figure.
|
||||
- If the data shows them as separate figures: that's a merge failure.
|
||||
- If the data shows them as one figure with N assets: check that N matches visual sub-panels.
|
||||
|
||||
5. CROSS-PAGE CHECK: If a figure's caption is on a different page than its assets:
|
||||
- Is this correct (caption on page N+1, image on page N)? Or is it a mismatch?
|
||||
- Look at both pages to confirm.
|
||||
|
||||
6. ROLE MISLABELS:
|
||||
- Is any `figure_caption` block actually a section heading? (e.g., "Figure 3. In vitro evaluation..." at the top of a new section)
|
||||
- Is any `figure_asset` / `media_asset` block actually a table?
|
||||
- Is any `figure_caption` actually just a sub-panel label ("Fig. 6" only, no description)?
|
||||
|
||||
### For each figure, report:
|
||||
|
||||
```
|
||||
Fig {N} — page {P} — status: {OK / PROBLEM}
|
||||
Issue: {specific problem description}
|
||||
Evidence: {what you see on the annotated page that supports this}
|
||||
```
|
||||
|
||||
### After checking all figures, provide a summary:
|
||||
|
||||
```
|
||||
Total figures: {N}
|
||||
OK: {M}
|
||||
Problems found: {K}
|
||||
|
||||
Problem details:
|
||||
Fig X: [one-line issue]
|
||||
...
|
||||
```
|
||||
|
||||
### Common patterns to watch for:
|
||||
|
||||
- **Truncated label as standalone figure:** "Fig. 6" with no description text, next to a long caption block below → these should be ONE figure with the long text as caption.
|
||||
- **Multi-panel grid as separate figures:** 4 images in a 2x2 grid, each with "Fig. N" label, one shared caption below → should be ONE composite figure.
|
||||
- **Caption between two figures:** "Fig. 2" caption sandwiched between Fig 2 image above and Fig 3 image below → caption belongs to the image ABOVE.
|
||||
- **Empty/phantom blocks in figure area:** A colored box with no visible content inside → the block detection created a ghost.
|
||||
- **Cross-page orphan:** Caption on page N, image on page N-1 → normal for preproofs.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Template 2: Frontmatter Verification
|
||||
|
||||
**Data source:** `audit_helpers.py --recipe frontmatter_map --key {KEY}`
|
||||
|
||||
**Vision agent prompt:**
|
||||
|
||||
```
|
||||
You are auditing page 1 (frontmatter) for paper {KEY}.
|
||||
|
||||
FRONTMATTER DATA:
|
||||
{DATA_JSON}
|
||||
|
||||
Annotated page: audit/{KEY}/annotated_pages/page_001.png
|
||||
|
||||
The data lists every block on page 1, top-to-bottom, with its pipeline-assigned role.
|
||||
Your job: verify each block's role against what you VISUALLY see on the page.
|
||||
|
||||
### For each block, answer:
|
||||
|
||||
1. ROLE CHECK: Is the assigned role correct?
|
||||
- paper_title: should be the LARGEST text near the top, centered or left-aligned.
|
||||
- authors: multiple names, usually below the title, smaller font.
|
||||
- affiliation: institutional addresses, below authors.
|
||||
- abstract_heading / abstract_body: "Abstract" label + paragraph text.
|
||||
- frontmatter_noise: journal name, article type badge, publisher strip — small text at page edges.
|
||||
- frontmatter_support: correspondence, DOI, dates.
|
||||
|
||||
2. BADGE CHECK: Is there a "HIGHLIGHTED PAPER" or "RESEARCH ARTICLE" badge?
|
||||
- If the data labels it as authors → WRONG.
|
||||
- If the data labels it as noise → correct.
|
||||
|
||||
3. MISSING ROLES: Are any expected roles missing?
|
||||
- If data says `missing_expected: ["paper_title"]` → check if the title is absorbed into another block.
|
||||
- If data says `missing_expected: ["abstract_heading"]` → check if the abstract exists but is labeled as body_paragraph.
|
||||
|
||||
4. ORDERING: The blocks in the data are sorted top-to-bottom. Is this order visually correct?
|
||||
- Expected: title → authors → affiliation → (badges/noise mixed in) → abstract → keywords → body text.
|
||||
- If the order looks wrong visually, report the correct order.
|
||||
|
||||
### Report format:
|
||||
|
||||
```
|
||||
Page 1 — {paper_title}
|
||||
Roles OK: {list of correct roles}
|
||||
Roles WRONG:
|
||||
p1:{block_id}: labeled as {role}, should be {correct_role}
|
||||
Missing: {list}
|
||||
Badge issue: {yes/no + detail}
|
||||
```
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Template 3: Reference Verification
|
||||
|
||||
**Data source:** `audit_helpers.py --recipe reference_map --key {KEY}`
|
||||
|
||||
**Vision agent prompt:**
|
||||
|
||||
```
|
||||
You are auditing reference completeness for paper {KEY}.
|
||||
|
||||
REFERENCE DATA:
|
||||
{DATA_JSON}
|
||||
|
||||
Annotated pages are at audit/{KEY}/annotated_pages/. Focus on:
|
||||
- The first reference page (where references begin)
|
||||
- The last reference page (where references end)
|
||||
- Any "transition_pages" listed in the data
|
||||
|
||||
### Check these specific things:
|
||||
|
||||
1. REFERENCE BOUNDARY: On the first reference page, is there a clean boundary between body text and references?
|
||||
- Look for the "References" heading (colored as reference_heading).
|
||||
- Body text (green) should STOP before this heading.
|
||||
- If body paragraphs appear AFTER the "References" heading → intrusion.
|
||||
|
||||
2. INTRUSIONS: The data lists "real_intrusions" — body_paragraph or backmatter blocks inside the reference zone.
|
||||
- If empty: no intrusions detected.
|
||||
- If not empty: look at the specific pages listed. Are these REALLY inside the reference zone?
|
||||
- Note: noise blocks (gray) between reference items are NORMAL (page numbers, headers).
|
||||
|
||||
3. ZONE GAPS: The data lists references not in `reference_zone`.
|
||||
- Look at the listed pages. Are these reference items that should be in reference_zone?
|
||||
- If ALL refs on a page are in the wrong zone: the zone assignment failed for that page.
|
||||
|
||||
4. TRANSITION PAGES: Pages where body and references co-exist.
|
||||
- Look at the page: is the visual boundary clean?
|
||||
- Is there body text that should have ended before the references?
|
||||
|
||||
5. REFERENCE NUMBERING: The data lists missing or duplicate reference numbers.
|
||||
- This is a data-derived check. No visual verification needed.
|
||||
- Missing numbers may indicate references absorbed into body text → check visually if the missing references appear as body paragraphs.
|
||||
|
||||
### Report format:
|
||||
|
||||
```
|
||||
References — {total_references} total
|
||||
Span: {start} → {end}
|
||||
Boundary clean: {yes/no}
|
||||
Real intrusions: {count or "none"}
|
||||
Zone issues: {count or "none"}
|
||||
Numbering gaps: {list or "none"}
|
||||
|
||||
Visual findings:
|
||||
[page-specific observations]
|
||||
```
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Template 4: Body Text Verification
|
||||
|
||||
**Data source:** `audit_helpers.py --recipe body_text_map --key {KEY}`
|
||||
|
||||
**Vision agent prompt:**
|
||||
|
||||
```
|
||||
You are auditing body text cleanliness for paper {KEY}.
|
||||
|
||||
BODY TEXT DATA:
|
||||
{DATA_JSON}
|
||||
|
||||
The data lists, per page, every body_paragraph, noise, unknown_structural, and
|
||||
other content block with their bboxes, zones, and text previews.
|
||||
|
||||
Annotated pages at audit/{KEY}/annotated_pages/page_NNN.png
|
||||
|
||||
### For pages flagged in the data:
|
||||
|
||||
1. NOISE IN CONTENT AREA: Blocks with `flag: text_in_content_area` or `flag: empty_in_content_area`.
|
||||
- Look at the specific page. Is the noise block actually sitting in body text?
|
||||
- Is it page furniture (page number, header) that happens to have non-edge coordinates?
|
||||
- Is it a phantom (empty box with no visible content)?
|
||||
|
||||
2. BODY TEXT WITH EMPTY TEXT: Body_paragraph blocks with `flag: empty_text_*`.
|
||||
- Look at the bbox on the annotated page. Is there visible text at this position?
|
||||
- If yes: OCR missed the text → PDF backfill needed.
|
||||
- If no: the block is a phantom that should be removed.
|
||||
|
||||
3. LARGE GAPS between body paragraphs (gap_px > 100).
|
||||
- Look at what's in the gap (listed in `blocks_in_gap`).
|
||||
- Is there supposed to be a figure, heading, or table here?
|
||||
- Is it a two-column layout where the gap is the right-column text?
|
||||
|
||||
4. TWO-COLUMN SUSPICION: If a page has body paragraphs with widely different x-centers.
|
||||
- Look at the page layout. Is it genuinely two-column?
|
||||
- Do the left and right columns each have coherent reading order?
|
||||
|
||||
### Report format:
|
||||
|
||||
```
|
||||
Body text — {page_count} pages checked
|
||||
Pages with noise in content: {list}
|
||||
Pages with empty body text: {list}
|
||||
Pages with large gaps: {list}
|
||||
|
||||
Per-page findings:
|
||||
Page {N}: [specific observations]
|
||||
```
|
||||
|
|
@ -36,22 +36,72 @@ Inputs:
|
|||
Materialize helper outputs under `audit/<paper_key>/` via `../scripts/ocr_truth_audit.py`, but treat them as accelerators only, not truth.
|
||||
|
||||
6. Inspect truth from evidence.
|
||||
Start with the overview pages (`annotated_pages/page_*.png`). Each page shows every block as a numbered rectangle, color-coded by the **pipeline's final rendered role** (after gate + normalize, the same role used in `fulltext.md`). Each of the 28 roles has a distinct color, grouped by semantic family:
|
||||
- dark blue family: `paper_title` `authors` `affiliation` `frontmatter_support` `frontmatter_noise`
|
||||
- purple family: `abstract_heading` `abstract_body`
|
||||
- orange/red family: `section_heading` `subsection_heading` `sub_subsection_heading`
|
||||
|
||||
The audit is split into **five phases**. Phase 6a-6d use the 4 recipes + 4 templates.
|
||||
Phase **6e** is the deep quality pass using the master atom.
|
||||
Work through them in this order.
|
||||
Each domain has a **data map** (run the recipe) and a **vision template** (read the atom).
|
||||
|
||||
**6a. Figure verification (most error-prone — do this first).**
|
||||
```bash
|
||||
python .opencode/skills/paperforge-development/scripts/audit_helpers.py --recipe figure_map --key {KEY}
|
||||
```
|
||||
This prints a JSON listing every figure: caption page/text, asset blocks with bboxes,
|
||||
match status, candidates, panel labels, cross-page flags.
|
||||
Then read `../atoms/ocr-vision-audit-templates.md` → "Template 1: Figure Verification".
|
||||
Dispatch a `vision` subagent with the figure_map JSON + that prompt template.
|
||||
The vision agent checks: asset coverage, caption correctness, sub-panel merging,
|
||||
cross-page pairing, role mislabels. Reports discrepancies between pipeline data and
|
||||
visual truth.
|
||||
|
||||
**6b. Frontmatter verification.**
|
||||
```bash
|
||||
python .opencode/skills/paperforge-development/scripts/audit_helpers.py --recipe frontmatter_map --key {KEY}
|
||||
```
|
||||
Then read `../atoms/ocr-vision-audit-templates.md` → "Template 2: Frontmatter Verification".
|
||||
Dispatch `vision` subagent with the frontmatter JSON + template.
|
||||
Checks: role correctness on page 1, badge mislabel, missing roles, ordering.
|
||||
|
||||
**6c. Reference verification.**
|
||||
```bash
|
||||
python .opencode/skills/paperforge-development/scripts/audit_helpers.py --recipe reference_map --key {KEY}
|
||||
```
|
||||
Then read `../atoms/ocr-vision-audit-templates.md` → "Template 3: Reference Verification".
|
||||
Dispatch `vision` subagent with the reference JSON + template.
|
||||
Checks: reference boundary, real intrusions, zone gaps, transition pages.
|
||||
|
||||
**6d. Body text verification (only flagged pages need vision).**
|
||||
```bash
|
||||
python .opencode/skills/paperforge-development/scripts/audit_helpers.py --recipe body_text_map --key {KEY}
|
||||
```
|
||||
Then read `../atoms/ocr-vision-audit-templates.md` → "Template 4: Body Text Verification".
|
||||
Most pages pass on data alone. Dispatch `vision` only for pages with
|
||||
flagged gaps, noise-in-content, or empty body text blocks.
|
||||
|
||||
**6e. Deep quality + typography audit (new — extends all 4 domains).**
|
||||
After the 4 recipe-driven vision passes, read `../atoms/ocr-vision-audit-master.md`.
|
||||
This atom provides per-role cross-reference pathways, typography checks,
|
||||
figure/table quality checks, chart-type routing to the chart-reading atoms,
|
||||
and full-page layout audit.
|
||||
|
||||
For each block that passed the recipe check, dispatch an additional vision pass
|
||||
if the block is Tier A (figure/table/structured) or if the page has typography flags.
|
||||
The master atom tells you which data files to cross-reference and what to look for.
|
||||
|
||||
**Color reference for annotated pages:** Each role has a distinct color:
|
||||
- dark blue: `paper_title` `authors` `affiliation` `frontmatter_*`
|
||||
- purple: `abstract_heading` `abstract_body`
|
||||
- orange/red: `section_heading` `subsection_heading` `sub_subsection_heading`
|
||||
- green: `body_paragraph`
|
||||
- red family: `reference_heading` `reference_item`
|
||||
- purple-brown family: `backmatter_heading` `backmatter_body` `backmatter_boundary_candidate`
|
||||
- amber/gold family: `media_asset` `figure_caption` `figure_caption_candidate` `figure_inner_text` `table_caption` `table_caption_candidate`
|
||||
- teal family: `structured_insert` `structured_insert_candidate` `non_body_insert`
|
||||
- gray family: `noise` `footnote` `unknown_structural`
|
||||
- red: `reference_heading` `reference_item`
|
||||
- purple-brown: `backmatter_*`
|
||||
- amber/gold: `figure_asset` `media_asset` `figure_caption*` `table_caption*` `figure_inner_text`
|
||||
- teal: `structured_insert*` `non_body_insert`
|
||||
- gray: `noise` `footnote` `unknown_structural`
|
||||
|
||||
The color tells you what the pipeline decided. Your job is to judge whether that decision matches visual truth.
|
||||
|
||||
The companion index file (`annotated_pages/page_*_index.json`) maps each label number to its full `block_id`, `role`, `zone`, `category`, `text_preview`, and `bbox`. Use the index to look up what any numbered block currently is, then judge its truth visually from the overview page.
|
||||
|
||||
Cross-reference against `block_trace`, document-structure artifacts, and rendered `fulltext` only after forming an initial visual judgment.
|
||||
The annotated pages show what the pipeline decided. Your job is to judge whether
|
||||
that matches visual truth. The companion index (`page_*_index.json`) maps label
|
||||
numbers to block IDs for cross-reference.
|
||||
|
||||
7. Write audit outputs.
|
||||
Produce the required reports and summaries using `../scripts/ocr_truth_audit.py` plus the schema in `../atoms/ocr-audit-report-schema.md`.
|
||||
|
|
@ -71,6 +121,13 @@ Inputs:
|
|||
|
||||
Use the page index to find a block's current role/zone before judging. Color category helps orient: red blocks are reference candidates, purple are backmatter candidates, etc.
|
||||
|
||||
For deep quality checks, see `../atoms/ocr-vision-audit-master.md` section 3 (chart quality),
|
||||
section 4 (page typography), and section 2 (per-role analysis with quality dimensions).
|
||||
|
||||
8.5. Record quality findings in block_review.jsonl.
|
||||
Extend each block_review.jsonl entry with optional fields `quality_checks` and `page_typography`
|
||||
as described in `../atoms/ocr-vision-audit-master.md` section 5. Use `null` for unchecked dimensions.
|
||||
|
||||
9. Verify review coverage.
|
||||
Run `../scripts/verify_review_coverage.py` against the paper audit directory. If required blocks are missing for the chosen mode, the audit is incomplete.
|
||||
|
||||
|
|
|
|||
|
|
@ -1388,3 +1388,215 @@ For each paper: rebuilt pipeline, generated annotated pages, performed visual bl
|
|||
**Result:** Figure grouping is less dependent on tidy layouts while preserving the no-page-swallow guardrail.
|
||||
|
||||
**Validation:** Rebuilt residual and unseen figure-heavy papers after the change; no new failure family introduced.
|
||||
|
||||
### 11.6 Reference Zone And Ownership Hardening With Untouched-Paper Regressions (2026-06-20)
|
||||
|
||||
**Problem:** Untouched truth-audit papers still showed reference intrusion and ownership fragmentation on irregular layouts.
|
||||
**Root cause:** Page continuity facts were too weakly shared across reference and ownership consumers.
|
||||
**Fix:** Added shared layout facts, hardened practical reference corridor containment, introduced journal-abbreviation-backed reference-style support, added bridge-aware display continuity for figure/table ownership, and limited PDF text fallback to local reference-entry repair.
|
||||
**Result:** Reference handling is more containment-oriented on mixed pages; ownership tolerates bridgeable empty gaps without reopening page swallow.
|
||||
**Test status:** Targeted OCR unit tests and untouched-paper regressions pass.
|
||||
|
||||
### 11.7 Figure Matching Hardening: Inline-Mention, Table-Label, Sub-Panel, Cross-Page, Legend-Bundle (2026-06-20)
|
||||
|
||||
**Problem:** 10-paper audit showed figure matching at ~75%. Dominant failure modes:
|
||||
(1) `_looks_like_inline_figure_mention` used text-content heuristics (verb lists, word-count thresholds) that falsely flagged long captions as body prose — caption_score dropped to 0.2, figures routed to unmatched_legends.
|
||||
(2) OCR labeled tabular sub-panel grids as `table` → `media_asset` role → excluded from figure-matching candidate pool.
|
||||
(3) Sub-panels fragmented into separate blocks across pages, never merged into parent figure clusters.
|
||||
(4) Cross-page figures (caption on page N+1, assets on N) unmatched by same-page-only matcher.
|
||||
(5) Preproof legend bundling (all captions on one page, figures on later pages) undetected.
|
||||
|
||||
**Root cause (inline mention):** `_looks_like_inline_figure_mention` line 91 used `len(words) >= 10` as gate for body-prose detection. Long VFS figure captions (30+ words) hit this, caption_score dropped below 0.4 threshold.
|
||||
|
||||
**Root cause (table label):** `build_figure_inventory` excluded `media_asset` with `raw_label="table"` from both `assets` collection (line 1000) and `_media_clusters` (line 432). KZP6FB4Y Fig 2 is a tabular sub-panel grid labeled `table` by PaddleOCR.
|
||||
|
||||
**Root cause (sub-panel merge):** `_grow_region_from_seed` had gap threshold of `page_width * 0.03` = 36px. Sub-panels ~50px apart in multi-panel figures couldn't merge.
|
||||
|
||||
**Root cause (cross-page):** `_allow_previous_page_sequential_match` required `zone == "post_reference_backmatter_zone"`. 28ALPCY7 Fig 6 caption in `display_zone` blocked. Also `future_page_asset` had no page-distance guard.
|
||||
|
||||
**Root cause (legend bundle):** PZ8B59K4 has "Figure Legends" page (p16) with 5 captions, figures on p19-23. Dedup logic preferred body-mention over `figure_caption` role, losing Fig 3. Sequential fallback matched across 3-page gaps.
|
||||
|
||||
**Fixes:**
|
||||
- `_looks_like_inline_figure_mention`: replaced text-content heuristics with zone+style signals. Functions with `figure_caption` role or `display_zone + legend_like` style return False directly.
|
||||
- `ocr_figures.py`: added "table" to `media_asset` raw_label filter in 3 places (assets collection, `_build_candidate_figure_groups_from_assets`, `_media_clusters`).
|
||||
- `_grow_region_from_seed`: increased gap threshold from 36px to `max(page_width * 0.08, 40)` = 96px.
|
||||
- `close_asset_tie` and `is_legend_only` both filter `figure_caption`/`figure_caption_candidate` with `fig_num is None` to `unmatched_legends`.
|
||||
- `_allow_previous_page_sequential_match`: added `display_zone` to accepted zones.
|
||||
- `future_page_asset`: restricted to `cp+1` only (no multi-page gaps).
|
||||
- Legend-bundle detection: 3+ figure captions on same page with 0 assets → 1:1 page-order match to subsequent pure-figure pages.
|
||||
- Dedup: prefer `figure_caption` role when both entries have no same-page assets.
|
||||
- `marker_signature` fallback: when `_extract_figure_number` returns None from empty text, check `marker_signature.type == "figure_number"`.
|
||||
- `ocr_pdf_spans.py`: set `role = "body_paragraph"` on backfilled blocks. `_refresh_artifacts` runs backfill before `rebuild_from_raw`.
|
||||
|
||||
**Test status:** 341 passed (97 figure + 205 document + 69 roles + 11 PDF + 34 tables). 2 tests updated to match new inline-mention behavior (text-only blocks without zone/style now match instead of being unmatched).
|
||||
|
||||
**Result:** Figure matching 76/83 = 92% across 10 papers. VFS8CBW2: 3→8 matched. Remaining: RKSLQRIM 5 ambiguous (pre-proof cross-page), 6DIINFHX 1 ambiguous (close_asset_tie).
|
||||
|
||||
### 11.8 OCR Body Text Recovery: PDF Backfill For OCR-Missed Blocks (2026-06-20)
|
||||
|
||||
**Problem:** PaddleOCR detects layout regions but fails to extract text on some two-column pages. Span metadata exists (88 spans) but all have `text=""`. Blocks end up as `unknown_structural` with `text=[]`.
|
||||
|
||||
**Root cause:** Upstream OCR model limitation — column boundary regions have detectable bboxes but the OCR engine produces no text output.
|
||||
|
||||
**Fix:** `ocr_pdf_spans.py` backfills text from PDF embedded text layer using `fitz.get_text("words", clip=bbox)`. Sets `_ocr_raw_status = "missing_text_recovered"` and `role = "body_paragraph"` on recovered blocks. `_refresh_artifacts` in audit runs backfill BEFORE `rebuild_from_raw` so structured block builder sees recovered text.
|
||||
|
||||
**Result:** KIX7SKXQ p3:7 recovered 323 chars ("Studying and understanding EnEF..."), p1:17 recovered 373 chars. All 4 papers audited show 0 empty unknown_structural blocks.
|
||||
|
||||
### 11.9 Figure Matching Known Issue: Shared-Caption Multi-Panel Figures (2026-06-20)
|
||||
|
||||
**Problem:** Paper 6QNRHRKX (JBJS 1970) has a 2x2 X-ray grid with individual "Fig. N" sub-panel labels (N=4,5,6,7) and a single shared caption below the entire grid. Each "Fig. N" label is treated as a separate truncated legend, creating 4 ambiguous figures with 4-7 candidates each. The real caption (block p3:12, `figure_caption_candidate`) can't match all 4 images simultaneously.
|
||||
|
||||
**Scope:** Rare pre-1990 format. Only observed in 6QNRHRKX across 36 audited papers.
|
||||
|
||||
**Current mitigation:** Truncated legends filtered when 2+ on same page. Short (<80 char) captions with fig_num=None go to unmatched_legends. Long shared caption remains as is.
|
||||
|
||||
**Targeted fix (deferred):** Detect N truncated legends sharing one long caption on same page → merge all N images as sub-panels of one composite figure with the long caption as owner. Trigger: >=2 \( `_TRUNCATED_LEGEND_ONLY_PATTERN`\) entries on same page + >=1 `figure_caption_candidate` with text >80 chars spanning the full grid width.
|
||||
|
||||
### 11.10 Figure Matching Known Issue: Cross-Page Preproof (RKSLQRIM) (2026-06-20)
|
||||
|
||||
**Problem:** RKSLQRIM (preproof) has 5 remaining ambiguous figures. Fig 10 caption on p28 has 0 same-page candidates. Sub-figure panel labels (A, B, C) on p15, p19, p20, p24 are misclassified as `figure_caption` with empty text. These block types: `display_zone` + `legend_like` + `raw_label=figure_title` — the pipeline sees a legend with empty text and a marker_signature with the figure number, but the figure assets are on different pages.
|
||||
|
||||
**Scope:** Preproof papers with cross-page figure layouts. Affects ~5-7 papers across 36.
|
||||
|
||||
**Current mitigation:** marker_signature fallback extracts figure numbers from empty-text blocks. These now get `single_unconfirmed_match` with 1 candidate instead of `no_asset_match`. But cross-page matching still fails for pages separated by body text.
|
||||
|
||||
**Targeted fix (deferred):** Extend sequential fallback or legend-bundle to handle cross-page gaps larger than 1 page when intervening pages have no competing figure captions.
|
||||
|
||||
### 11.11 Figure Matching: close_asset_tie Above-Preference + adjacent_x Scoring (2026-06-20)
|
||||
|
||||
**Problem 1 (close_asset_tie):** When a figure caption is sandwiched between two asset groups (one above, one below), both scored identically and the algorithm couldn't break the tie. This produced `close_asset_tie` ambiguous entries for standard layouts where the caption sits below the figure image and above the next figure's image.
|
||||
|
||||
**Fix:** When `close_asset_tie` fires with both "above" and "below" sides, prefer the above-only candidates (standard: image → caption stacked below). Falls back to close_asset_tie only when no above-only candidates exist.
|
||||
|
||||
**Result:** 2AGGSMVQ Fig2+Fig5 fixed (4/6→6/6), 2H8MZ27H Fig1 fixed (5/6→6/6). 97 figure tests pass.
|
||||
|
||||
**Problem 2 (side-by-side caption):** Two-column journals (JSES International, Springer CORR) place figure captions in the left column with the image in the right column at the same y-band. `score_figure_match` had zero `x_overlap` for this layout because bounding boxes don't intersect horizontally. The decision gate at `score >= 0.6 AND (has_x_overlap OR ...)` blocked the match because `has_x_overlap=False` and `contextual_support=False`.
|
||||
|
||||
**Fix:** Added `adjacent_x` signal to `ocr_scores.py:score_figure_match`: when horizontal gap < 80px AND < 30% of narrower bbox width, treat as horizontal adjacency → set `has_x_overlap=True` and add +0.20 score. This is a side-by-side layout detection, not a column segmentation, so it doesn't require stable page-level column boundaries.
|
||||
|
||||
**Result:** 4DU8LEH2 Fig1 (JSES, caption-left image-right) fixed: 3/4→4/4. 4KCHGV2Z Fig2 (Springer CORR, same pattern) fixed. 302 tests pass.
|
||||
|
||||
### 11.12 Figure Matching Known Issue: Multi-Column Figure Over-Merge (2026-06-20)
|
||||
|
||||
**Problem:** 3FDT9652 (JSES International, 2-column journal). Page 3 has Fig 2 (left column) + Fig 3 (right column) at the same y-band. The `page_assets` / `_grow_region_from_seed` logic merges both images into one cluster because they share y-ranges, even though they belong to different figures in different columns. This causes Fig 2 to steal Fig 3's image, Fig 3 to steal Fig 4's images below, and Fig 4 to go ambiguous (0 assets left). All 6 figures on pages 2-5 are affected.
|
||||
|
||||
**Root cause:** Matching uses vertical proximity only, without column-boundary awareness. On multi-column pages, figures in different columns at the same y get incorrectly merged. The `_grow_region_from_seed` function's `adjacent_right` check (gap ≤ 96px) happily merges across column gutters.
|
||||
|
||||
**Why not fix with column segmentation:** Column gutter positions vary by journal (JSES ~750px, Springer ~650px, 3-column layouts ~450px/750px). A hardcoded threshold regresses other papers. Multi-column layouts are also relatively rare in the current collection (1 of 37 normal papers).
|
||||
|
||||
**Scope:** Affects 3FDT9652 specifically. Not observed in the other 36 normal papers.
|
||||
|
||||
**Targeted fix (deferred):** Requires "caption-as-boundary" matching: when N captions exist on a page with N asset clusters, match by reading-order proximity rather than all-page-assets clustering. Each caption should act as a firewall that prevents assets from being claimed by captions above it. Alternatively, detect multi-caption pages and disable `page_assets` groups (which merges all page assets into one cluster) for those pages, relying on `same_row_*` + region-grown groups only.
|
||||
|
||||
---
|
||||
|
||||
## 9.3 Figure Merge Root-Cause Analysis (2026-06-21)
|
||||
|
||||
### The problem
|
||||
|
||||
Multi-panel figures on complex layouts (SAN9AYVR, 2GN9LMCW Fig 4) fail to merge all sub-panels. Unmatched assets remain. The 4-direction growth + scoring bonus fixes didn't solve the core issue.
|
||||
|
||||
### Why human vision always merges them
|
||||
|
||||
A human looking at a page sees:
|
||||
1. A cluster of images close together → ONE perceptual group
|
||||
2. ONE legend/caption for that cluster → confirms the group
|
||||
3. Gaps between images in the same figure < gaps to the next figure/text
|
||||
4. The cluster forms a rectangular bounding box
|
||||
|
||||
Result: the human **always** merges, no competition.
|
||||
|
||||
### Why the algorithm fails
|
||||
|
||||
The grouping has 3 competing layers:
|
||||
|
||||
| Layer | What it creates | Why it fails |
|
||||
|-------|----------------|-------------|
|
||||
| `single_asset` (line 514) | 1 asset = 1 group | ALWAYS created, competes with merged groups |
|
||||
| `same_row_pair/triple` (line 528) | Consecutive same-row assets | Misses 2x2 grids, 3x2 layouts, irregular shapes |
|
||||
| `region_grown` (line 553) | Greedy seed-growth | First seed wins, no backtracking, eats adjacent regardless of actual figure boundary |
|
||||
|
||||
**Root cause:** The algorithm treats grouping as a LOCAL greedy problem when it should be a GLOBAL clustering problem.
|
||||
|
||||
A human sees ALL assets, computes global distances, and forms clusters. The algorithm:
|
||||
1. Generates overlapping groups (same asset can be in single_asset, same_row, AND region_grown)
|
||||
2. Makes them COMPETE via scoring (single_asset vs merged group for the same legend)
|
||||
3. The scoring favors closeness (single asset close to caption wins over merged group with larger centroid distance)
|
||||
|
||||
**Fix direction:** Replace greedy competition with global spatial clustering.
|
||||
|
||||
### Edge-case analysis: risks and fallbacks
|
||||
|
||||
The simple clustering approach (group all assets within a distance threshold) fails on multiple real layouts. Here's a complete risk matrix:
|
||||
|
||||
| # | Scenario | Risk | How to handle | Fallback if still wrong |
|
||||
|---|----------|------|---------------|------------------------|
|
||||
| 1 | **One page, 2+ separate figures, each with its own caption** | Clustering merges them into 1 group | **Caption-as-boundary**: count legends on the page. If N legends == 1 → one cluster. If N legends > 1 → split assets by nearest caption y-position. Each legend "owns" the assets in its y-band. | Fall through to single_asset matching: if a cluster has multiple legends and splitting fails, treat each asset as its own group |
|
||||
| 2 | **Cross-page figure (preprint, figure floats)** | Per-page clustering won't see assets on adjacent pages | Keep existing sequential fallback (`_expand_matched_assets_locally`). After matching per-page clusters, check cp-1, cp, cp+1 for orphan assets. | If the caption is on page N but ALL assets are on page N+1, the cluster on N is empty → sequential fallback claims assets on N+1 |
|
||||
| 3 | **Old journal (dense, small figures, tight spacing)** | Distance threshold too generous → merge unrelated figures | Check for **text separators** between asset groups. If body_paragraph / section_heading blocks exist in the vertical gap between two asset clusters, don't merge across the separator. | Tight spacing not a problem if text separators are respected |
|
||||
| 4 | **Irregular layout (1 tall panel left + 3 small panels right)** | Large horizontal gap > threshold → left panel and right stack are separate clusters | Use **vertical-overlap signal**: if two asset groups share significant y-overlap (>= 50% of the shorter group's height), merge them even if horizontal gap is large. This is how humans see them as one figure. | If vertical-overlap approach over-merges, fall back to pure distance clustering |
|
||||
| 5 | **Embedded caption inside figure** | Caption block exists inside asset cluster bbox (e.g., panel labels, axis text) | This is fine — we cluster ASSETS (figure_asset / media_asset), not all blocks. Text blocks are ignored. | N/A |
|
||||
| 6 | **Figure + table on same page** | Table assets merge with figure assets | Split by type BEFORE clustering: figure_asset → one cluster set, table_asset → another. Legends also split by type (figure_caption vs table_caption). | If type labels are wrong (table mislabeled as figure), the split won't help. Need type-independent position-only fallback. |
|
||||
| 7 | **Multi-column layout (figures in different columns at same y)** | Assets in left column figure merge with right column figure | **Caption-as-boundary**: 2+ legends on the page → split assets by nearest caption y. Each legend claims the assets below it until the next legend. Text separators (column gutters are not text separators, but section headings/subheadings in each column ARE). | If split by y fails (same y, different columns), fall through to single_asset matching — safe but unmerged |
|
||||
| 8 | **Orphan single-panel figure** | 1 asset, 1 legend on the page, not near any other asset | A cluster of 1 asset is still a valid cluster. Match legend → single-panel figure. Perfectly fine. | N/A |
|
||||
| 9 | **Figures separated by section heading or body text** | Text acts as visual separator but algorithm ignores text | **Text separator detection**: check the gap between two asset clusters. If any text block (section_heading, body_paragraph, backmatter_heading) exists in that gap, it's a separator. Don't merge across separators. | If text detection fails (empty block, wrong classification), fall back to distance-only clustering |
|
||||
| 10 | **Very large multi-panel figure (>10 panels)** | Panels spread across full page, gaps may exceed threshold | Use **permissive distance threshold** but constrain by bounding-box aspect. A figure spanning the full page width is normal; threshold should be proportional to the cluster's growing bbox, not a fixed % of page width. | If still split, the sequential expansion (existing `_expand_matched_assets_locally`) merges remaining orphans |
|
||||
|
||||
### Complete proposed algorithm
|
||||
|
||||
```
|
||||
For each page:
|
||||
1. Collect figure assets on this page
|
||||
(figure_asset + media_asset with image/chart/figure raw_label)
|
||||
Exclude: non_body_media, table assets
|
||||
|
||||
2. Remove single_asset groups from consideration
|
||||
|
||||
3. Cluster assets by distance:
|
||||
- For each pair of assets:
|
||||
- Check horizontal gap < 12% page_width
|
||||
- Check vertical gap < 8% page_height
|
||||
- Check NO text separator (body_paragraph, section_heading)
|
||||
in the vertical gap between them
|
||||
- Form clusters via union-find on connected pairs
|
||||
|
||||
4. For irregular layouts: if two clusters share >= 50% y-overlap,
|
||||
merge them even if horizontal gap exceeds threshold
|
||||
|
||||
5. Each cluster → ONE candidate group (no single_asset option for
|
||||
any asset inside a cluster)
|
||||
|
||||
6. Count legends on this page:
|
||||
- 0 legends → orphan cluster, pass to cross-page fallback
|
||||
- 1 legend → auto-match (no scoring competition)
|
||||
- N legends → split: assign each asset to nearest caption
|
||||
by y-distance (each legend claims the vertical band below it)
|
||||
|
||||
7. Match: legend → cluster → accept.
|
||||
No scoring competition. If the cluster exists and the legend
|
||||
is on the same page, they belong together.
|
||||
|
||||
8. Single assets NOT in any cluster → treated as individual
|
||||
candidate groups (backward compatible with single-panel figs).
|
||||
```
|
||||
|
||||
### Changes to existing code
|
||||
|
||||
1. `_build_candidate_figure_groups_from_assets`: Replace single_asset/same_row/page_assets/region_growth with clustering
|
||||
2. `_score_legend_to_group`: Remove single_asset path. Cluster groups auto-match at high confidence.
|
||||
3. No change to `_expand_matched_assets_locally` (cross-page fallback remains)
|
||||
4. No change to dedup (separate issue)
|
||||
5. Column detection: new helper function
|
||||
|
||||
### Implementation risk summary
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|------|----------|-----------|
|
||||
| Over-merge (separate figures merged) | Medium | Caption-as-boundary split, text separator detection |
|
||||
| Under-merge (figure split into pieces) | Low | Existing cross-page fallback collects orphans |
|
||||
| Performance regression on large pages | Low | Union-find is O(n log n), same as current |
|
||||
| Column detection false positive | Low | Column mode is additive — assets still cluster within each column |
|
||||
|
||||
---
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,376 @@
|
|||
# Global Distance Clustering for Figure Merge
|
||||
|
||||
> **Date:** 2026-06-21
|
||||
> **Status:** draft for review
|
||||
> **Scope:** Replace greedy region-growing figure merge with global distance-based clustering + caption-as-boundary fallback
|
||||
> **Supersedes:** `2026-06-20-region-growing-figure-merge-design.md`
|
||||
|
||||
---
|
||||
|
||||
## 1. Why region-growing failed
|
||||
|
||||
The 2026-06-20 spec introduced `_grow_region_from_seed`: starting from one asset, greedily absorb neighbors right and down. It was extended to 4 directions (left, up, right, down) but the core problem remains:
|
||||
|
||||
| Problem | Why | Consequence |
|
||||
|---------|-----|-------------|
|
||||
| **First seed wins** | Assets are sorted y-x; the top-left seed absorbs everything it can reach | Remaining seeds get fewer assets → partial merges |
|
||||
| **No backtracking** | Once absorbed, an asset is `seen_growth_ids` and can't join another group | Wrong merges persist |
|
||||
| **Overlapping candidates** | An asset exists in single_asset + same_row + region_grown groups simultaneously | Groups compete via scoring; the closest asset wins, merged group loses |
|
||||
| **Local greed, not global view** | Each seed only sees immediate neighbors, not the whole page layout | 2x2 grids, irregular layouts, and cross-column figures all fail |
|
||||
|
||||
**Concrete failure (SAN9AYVR Figure 23, page 50):**
|
||||
A large multi-panel figure with panels a-h spanning the page. Region growth from the top-left seed absorbs some panels, eats across the page, but misses panels in the lower-right because they're more than one "gap" away from any grown boundary. Those missed panels become unmatched assets.
|
||||
|
||||
---
|
||||
|
||||
## 2. Design: distance-based clustering
|
||||
|
||||
### Core principle
|
||||
|
||||
On a given page, ASSETS THAT ARE CLOSE TOGETHER WITH NO TEXT SEPARATOR BETWEEN THEM FORM ONE FIGURE.
|
||||
|
||||
This is how human vision works: global spatial proximity + text boundaries = perceptual groups.
|
||||
|
||||
### Algorithm (8 steps)
|
||||
|
||||
```
|
||||
Step 1 — Collect figure assets on the page
|
||||
input: all blocks on a page
|
||||
filter: role=figure_asset OR (role=media_asset AND raw_label∈{image,chart,figure})
|
||||
exclude: _non_body_media, table assets, non_body_insert
|
||||
|
||||
Step 2 — Remove single_asset from candidate generation
|
||||
Assets that belong to a cluster must NOT also exist as standalone candidates.
|
||||
Only unclustered single assets get standalone groups.
|
||||
|
||||
Step 3 — Pairwise distance clustering
|
||||
For each pair of assets (A, B) on the same page:
|
||||
- horizontal_gap = max(0, left_b - right_a) [right edge of leftmost to left edge of rightmost]
|
||||
- vertical_gap = max(0, top_b - bottom_a) [bottom of upper to top of lower]
|
||||
- candidate_for_merge = true IF:
|
||||
horizontal_gap <= max(page_width * 0.12, 40)
|
||||
AND vertical_gap <= max(page_height * 0.08, 40)
|
||||
- BUT: if a text block (body_paragraph, section_heading, subsection_heading)
|
||||
with non-empty text exists in the bounding rectangle between A and B,
|
||||
candidate_for_merge = false (text separator)
|
||||
|
||||
Union-find: connect all candidate pairs into clusters.
|
||||
|
||||
Step 4 — Irregular layout merge
|
||||
For each pair of clusters (C1, C2):
|
||||
y_overlap = overlap of C1's y-range and C2's y-range
|
||||
shorter_height = min(height of C1, height of C2)
|
||||
if y_overlap >= shorter_height * 0.5:
|
||||
merge C1 and C2 even if horizontal_gap exceeds threshold
|
||||
(Rationale: 1 tall left panel + 3 small right panels form one figure)
|
||||
|
||||
Step 5 — Generate candidate groups
|
||||
Each cluster → ONE candidate group entry
|
||||
cluster_bbox = union of all asset bboxes in the cluster
|
||||
group_type = "distance_cluster"
|
||||
evidence = ["same_page", "distance_clustered"]
|
||||
|
||||
Step 6 — Count legends on the page
|
||||
legends = blocks with role=figure_caption on this page
|
||||
n_legends = len(legends)
|
||||
|
||||
Step 7 — Match legends to clusters
|
||||
Case n_legends == 0:
|
||||
→ orphan clusters, pass to _expand_matched_assets_locally (existing cross-page fallback)
|
||||
Case n_legends == 1:
|
||||
→ auto-match legend to cluster. score = 0.85. decision = "matched"
|
||||
No scoring competition.
|
||||
Case n_legends >= 2:
|
||||
→ split: sort legends by y-position. Sort clusters by y-position.
|
||||
Assign each cluster to the nearest legend.
|
||||
If multiple clusters map to the same legend → merge them.
|
||||
If any legend has no cluster → pass to single_asset fallback.
|
||||
|
||||
Step 8 — Single assets not in any cluster
|
||||
→ treated as individual candidate groups (backward compatible)
|
||||
→ matched via existing score_figure_match
|
||||
```
|
||||
|
||||
### Threshold rationale
|
||||
|
||||
The gap thresholds (12% page width, 8% page height, min 40px) match the existing thresholds used in `same_row_pair` and `_grow_region_from_seed`. They've been tested on 37 normal papers without false page-swallow. No new threshold tuning needed.
|
||||
|
||||
The min 40px floor prevents over-merging on small figures in dense layouts.
|
||||
|
||||
### Text separator detection
|
||||
|
||||
```python
|
||||
def _has_text_separator(a: dict, b: dict, all_blocks: list[dict]) -> bool:
|
||||
"""Check if any text block exists in the bounding rect between assets a and b."""
|
||||
ax1, ay1, ax2, ay2 = a["bbox"]
|
||||
bx1, by1, bx2, by2 = b["bbox"]
|
||||
# bounding rect of both assets
|
||||
rect = [min(ax1, bx1), min(ay1, by1), max(ax2, bx2), max(ay2, by2)]
|
||||
for block in all_blocks:
|
||||
role = block.get("role", "")
|
||||
if role not in ("body_paragraph", "section_heading", "subsection_heading"):
|
||||
continue
|
||||
txt = str(block.get("text", "") or "").strip()
|
||||
if not txt:
|
||||
continue
|
||||
bx1, by1, bx2, by2 = block["bbox"]
|
||||
# text block INSIDE the bounding rect AND BETWEEN the two assets vertically
|
||||
if (bx1 > rect[0] and bx2 < rect[2] and
|
||||
by1 > min(ay2, by2) and by2 < max(ay1, by1)):
|
||||
return True
|
||||
return False
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Edge-case matrix with fallbacks
|
||||
|
||||
All 10 scenarios from the PROJECT-MANAGEMENT analysis, with concrete handling:
|
||||
|
||||
| # | Scenario | Detection | Handling | Fallback |
|
||||
|---|----------|-----------|----------|----------|
|
||||
| 1 | One page, 2+ separate figures | n_legends >= 2 | Split by y-proximity | Per-asset single_agent scoring |
|
||||
| 2 | Cross-page figure | n_legends == 0 after per-page clustering | Move to `_expand_matched_assets_locally` | Existing sequential fallback |
|
||||
| 3 | Dense old-journal layout | Text separator in gap | Don't merge across separator | Tight spacing with no text → distance threshold allows merge |
|
||||
| 4 | Irregular layout (1 tall + 3 small) | y_overlap >= 50 | Force merge across x-gap | If over-merged, caption-as-boundary splits |
|
||||
| 5 | Embedded caption inside figure | Caption is not an asset | Ignored by clustering | N/A |
|
||||
| 6 | Figure + table on same page | Split by type (figure_asset vs table_asset) | Cluster separately | Type-independent position fallback |
|
||||
| 7 | Multi-column figures same y | n_legends >= 2 | Caption-as-boundary split | Single_asset fallback |
|
||||
| 8 | Orphan single-panel figure | Cluster of size 1 | Standalone candidate | Existing score_figure_match |
|
||||
| 9 | Text-separated figures | Text separator exists | Don't merge across | Distance threshold without separator |
|
||||
| 10 | Very large multi-panel (>10 panels) | Many assets, single legend | Cluster merges all; sequential expansion collects stragglers | Existing `_expand_matched_assets_locally` |
|
||||
|
||||
---
|
||||
|
||||
## 4. Files to change
|
||||
|
||||
### `paperforge/worker/ocr_figures.py`
|
||||
|
||||
| Function | Lines | Change |
|
||||
|----------|-------|--------|
|
||||
| `_build_candidate_figure_groups_from_assets` | 487-603 | Replace entire body. Remove single_asset, same_row, page_assets, region_growth. Add clustering. |
|
||||
| `_grow_region_from_seed` | 361-389 | DELETE (replaced by clustering) |
|
||||
| `_validate_grown_region` | 392-413 | DELETE (validation moves into clustering) |
|
||||
| `_asset_gap_left` | (new) | DELETE (added in previous fix, no longer needed) |
|
||||
| `_asset_gap_above` | (new) | DELETE (no longer needed) |
|
||||
| `_score_legend_to_group` | 608-662 | MODIFY: add `"distance_cluster"` case returning score=0.85, `decision="matched"` when n_legends==1 |
|
||||
| `_expand_matched_assets_locally` | 665-? | No change (cross-page fallback stays) |
|
||||
| NEW `_cluster_page_assets` | (new) | ADD: main clustering function |
|
||||
| NEW `_has_text_separator` | (new) | ADD: text separator detection |
|
||||
|
||||
### `paperforge/worker/ocr_figures.py` — detailed function changes
|
||||
|
||||
**`_build_candidate_figure_groups_from_assets()`** — replacement:
|
||||
|
||||
```python
|
||||
def _build_candidate_figure_groups_from_assets(
|
||||
assets: list[dict], blocks: list[dict],
|
||||
legends: list[dict], page_width: float = 1200
|
||||
) -> list[dict]:
|
||||
media = _filter_figure_assets(assets)
|
||||
groups: list[dict] = []
|
||||
next_id = 1
|
||||
|
||||
by_page: dict[int, list[dict]] = {}
|
||||
for block in media:
|
||||
by_page.setdefault(int(block.get("page", 0) or 0), []).append(block)
|
||||
|
||||
for page, page_media in by_page.items():
|
||||
if len(page_media) == 0:
|
||||
continue
|
||||
|
||||
# Count legends on this page
|
||||
page_legends = [l for l in legends if l.get("page") == page]
|
||||
|
||||
# Cluster assets by distance
|
||||
clusters = _cluster_page_assets(page_media, blocks, page_width)
|
||||
|
||||
for cluster in clusters:
|
||||
entry = _candidate_group_entry(
|
||||
f"group_{next_id:04d}", page, cluster,
|
||||
"distance_cluster",
|
||||
["same_page", "distance_clustered"],
|
||||
)
|
||||
groups.append(entry)
|
||||
next_id += 1
|
||||
|
||||
# Single assets not in any cluster → individual groups
|
||||
clustered_ids = set()
|
||||
for c in clusters:
|
||||
for bid in [b.get("block_id") for b in c]:
|
||||
clustered_ids.add(str(bid))
|
||||
for block in page_media:
|
||||
if str(block.get("block_id", "")) not in clustered_ids:
|
||||
groups.append(_candidate_group_entry(
|
||||
f"group_{next_id:04d}", page, [block],
|
||||
"single_asset", ["same_page", "single_asset"]
|
||||
))
|
||||
next_id += 1
|
||||
|
||||
return groups
|
||||
```
|
||||
|
||||
**`_cluster_page_assets()`** — new function:
|
||||
|
||||
```python
|
||||
def _cluster_page_assets(
|
||||
page_assets: list[dict],
|
||||
all_blocks: list[dict],
|
||||
page_width: float,
|
||||
) -> list[list[dict]]:
|
||||
"""Cluster assets by spatial proximity with text-separator awareness."""
|
||||
if len(page_assets) <= 1:
|
||||
return [list(page_assets)]
|
||||
|
||||
# Get page height from first asset's bbox
|
||||
page_height = 1600.0 # default
|
||||
for b in all_blocks:
|
||||
ph = b.get("page_height")
|
||||
if ph:
|
||||
page_height = float(ph)
|
||||
break
|
||||
|
||||
h_threshold = max(page_width * 0.12, 40.0)
|
||||
v_threshold = max(page_height * 0.08, 40.0)
|
||||
|
||||
# Union-find
|
||||
parent = list(range(len(page_assets)))
|
||||
|
||||
def find(x):
|
||||
while parent[x] != x:
|
||||
parent[x] = parent[parent[x]]
|
||||
x = parent[x]
|
||||
return x
|
||||
|
||||
def union(x, y):
|
||||
px, py = find(x), find(y)
|
||||
if px != py:
|
||||
parent[py] = px
|
||||
|
||||
for i in range(len(page_assets)):
|
||||
for j in range(i + 1, len(page_assets)):
|
||||
a, b = page_assets[i], page_assets[j]
|
||||
ab = a.get("bbox", [0, 0, 0, 0])
|
||||
bb = b.get("bbox", [0, 0, 0, 0])
|
||||
|
||||
h_gap = max(0.0, bb[0] - ab[2], ab[0] - bb[2])
|
||||
v_gap = max(0.0, bb[1] - ab[3], ab[1] - bb[3])
|
||||
|
||||
if h_gap > h_threshold:
|
||||
# Check y-overlap for irregular layouts
|
||||
a_y1, a_y2 = ab[1], ab[3]
|
||||
b_y1, b_y2 = bb[1], bb[3]
|
||||
y_overlap = max(0, min(a_y2, b_y2) - max(a_y1, b_y1))
|
||||
shorter_h = min(a_y2 - a_y1, b_y2 - b_y1)
|
||||
if shorter_h > 0 and y_overlap / shorter_h < 0.5:
|
||||
continue # no horizontal overlap and not vertically aligned → separate clusters
|
||||
|
||||
if v_gap > v_threshold:
|
||||
continue # too far apart vertically → separate clusters
|
||||
|
||||
# Text separator check
|
||||
if _has_text_separator(a, b, all_blocks):
|
||||
continue
|
||||
|
||||
union(i, j)
|
||||
|
||||
# Build clusters
|
||||
clusters: dict[int, list[dict]] = {}
|
||||
for i, block in enumerate(page_assets):
|
||||
root = find(i)
|
||||
if root not in clusters:
|
||||
clusters[root] = []
|
||||
clusters[root].append(block)
|
||||
|
||||
return list(clusters.values())
|
||||
```
|
||||
|
||||
**`_has_text_separator()`** — new function:
|
||||
|
||||
```python
|
||||
def _has_text_separator(a: dict, b: dict, all_blocks: list[dict]) -> bool:
|
||||
"""True if body text or heading exists in the gap between assets a and b."""
|
||||
ab = a.get("bbox", [0, 0, 0, 0])
|
||||
bb = b.get("bbox", [0, 0, 0, 0])
|
||||
|
||||
gap_top = min(ab[3], bb[3])
|
||||
gap_bot = max(ab[1], bb[1])
|
||||
if gap_bot <= gap_top:
|
||||
return False # assets overlap vertically, no gap to check
|
||||
|
||||
gap_left = min(ab[0], bb[0])
|
||||
gap_right = max(ab[2], bb[2])
|
||||
|
||||
for block in all_blocks:
|
||||
role = block.get("role", "")
|
||||
if role not in ("body_paragraph", "section_heading", "subsection_heading"):
|
||||
continue
|
||||
txt = str(block.get("text", "") or "").strip()
|
||||
if not txt:
|
||||
continue
|
||||
cb = block.get("bbox", [0, 0, 0, 0])
|
||||
# Block must be inside the gap rectangle
|
||||
if (cb[0] >= gap_left and cb[2] <= gap_right
|
||||
and cb[1] >= gap_top and cb[3] <= gap_bot):
|
||||
return True
|
||||
return False
|
||||
```
|
||||
|
||||
**`_score_legend_to_group()`** — modification:
|
||||
|
||||
```python
|
||||
# Add before the `if gt == "page_assets":` block:
|
||||
if gt == "distance_cluster":
|
||||
return {
|
||||
"score": 0.85,
|
||||
"decision": "matched",
|
||||
"evidence": ["same_page", "distance_clustered"],
|
||||
}
|
||||
```
|
||||
|
||||
This replaces the 0.55 flat score for `page_assets` with a 0.85 guaranteed match for distance-clustered groups.
|
||||
|
||||
---
|
||||
|
||||
## 5. Acceptance criteria
|
||||
|
||||
1. **SAN9AYVR Figure 23**: All sub-panels (a-h) merged into one figure, 0 unmatched assets on page 50.
|
||||
2. **2GN9LMCW Figure 4**: All 6 assets merged (was previously partially merged).
|
||||
3. **DWQQK2YB Figure 2**: Stays correctly on page 38 with 19 assets (existing fix, must not regress).
|
||||
4. **3FDT9652**: No regression on multi-column figures (separate figures in different columns with their own captions must not merge).
|
||||
5. **2GN9LMCW Figures 1-3**: Single-panel figures remain as independent figures (clusters of size 1).
|
||||
6. **No page-swallow**: A page with 10+ assets across 3 separate figures must not merge them all (text separator + caption-as-boundary prevents this).
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks and mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|------------|
|
||||
| Over-merge (2 separate figs → 1 cluster) | Medium | Medium | Caption-as-boundary split when n_legends >= 2 |
|
||||
| Under-merge (single fig → 2+ clusters) | Low | Low | `_expand_matched_assets_locally` collects orphans |
|
||||
| Text separator false positive (body text between sub-panels of same fig) | Low | Medium | Text inside display_zone is usually caption/figure text, not body_paragraph. Real body text between sub-panels is rare in published papers. |
|
||||
| Distance threshold too tight for large figures | Low | Low | y-overlap signal merges irregular layouts. Cross-page fallback catches stragglers. |
|
||||
| Performance (pairwise O(n²)) | Low | Low | Pages with > 50 assets are rare (< 1% of corpus). Union-find is O(n log n). |
|
||||
|
||||
---
|
||||
|
||||
## 7. Files summary
|
||||
|
||||
| File | Action |
|
||||
|------|--------|
|
||||
| `paperforge/worker/ocr_figures.py` | Replace `_build_candidate_figure_groups_from_assets` body; add `_cluster_page_assets` + `_has_text_separator`; delete `_grow_region_from_seed` + `_validate_grown_region`; modify `_score_legend_to_group` |
|
||||
| `tests/test_ocr_figures.py` | Add tests for distance clustering, text separator, multi-legend split, irregular layout |
|
||||
| `tests/test_ocr_real_paper_regressions.py` | Add SAN9AYVR regression test for Figure 23 merge |
|
||||
| `PROJECT-MANAGEMENT.md` | Updated with analysis (already done) |
|
||||
|
||||
---
|
||||
|
||||
## 8. Rejected alternatives
|
||||
|
||||
| Alternative | Why rejected |
|
||||
|-------------|-------------|
|
||||
| Fix region-growth scoring (new bonus/threshold) | Doesn't solve the greedy-first-seed-wins problem |
|
||||
| Add more growth directions (8-direction) | Still local, still greedy, still no backtracking |
|
||||
| Machine learning for figure grouping | Over-engineering. Distance clustering is deterministic, auditable, and sufficient. |
|
||||
| Per-page manual layout templates | Not generalizable to arbitrary journal formats |
|
||||
|
|
@ -1418,6 +1418,25 @@ def _block_in_any_reference_zone(
|
|||
return any(global_index in zone.block_indices for zone in zones)
|
||||
|
||||
|
||||
def repair_reference_entry_from_pdf_text(block_run: list[dict], page_text: str) -> str:
|
||||
seed = " ".join(
|
||||
str(block.get("text") or "").strip()
|
||||
for block in block_run
|
||||
if str(block.get("text") or "").strip()
|
||||
)
|
||||
seed = re.sub(r"\s+", " ", seed).strip()
|
||||
if not seed:
|
||||
return ""
|
||||
parts = [part for part in seed.split() if len(part) > 2]
|
||||
if not parts:
|
||||
return seed
|
||||
first = parts[0]
|
||||
idx = page_text.find(first)
|
||||
if idx < 0:
|
||||
return seed
|
||||
return page_text[idx : idx + 300].strip()
|
||||
|
||||
|
||||
def _detect_forward_body_end(
|
||||
blocks: list[dict],
|
||||
page_layouts: dict[int, PageLayoutProfile] | None = None,
|
||||
|
|
@ -2391,6 +2410,68 @@ def analyze_document_structure(blocks: list[dict]) -> DocumentStructure:
|
|||
return ds
|
||||
|
||||
|
||||
def compute_layout_facts(blocks: list[dict]) -> list[dict]:
|
||||
facts: list[dict] = []
|
||||
band_seq = 0
|
||||
last_key: tuple[int, str, int] | None = None
|
||||
ref_roles = {"reference_heading", "reference_item"}
|
||||
display_roles = {"figure_asset", "media_asset", "table_html", "figure_caption", "table_caption"}
|
||||
tail_zones = {"tail_nonref_hold_zone", "post_reference_backmatter_zone"}
|
||||
side_roles = {"structured_insert", "structured_insert_candidate", "non_body_insert"}
|
||||
fm_zones = {"frontmatter_side_zone", "frontmatter_main_zone"}
|
||||
|
||||
def _sort_key(block: dict) -> tuple[int, float, float]:
|
||||
bbox = block.get("bbox") or [0, 0, 0, 0]
|
||||
return (
|
||||
int(block.get("page", 0) or 0),
|
||||
float(bbox[0]) if len(bbox) >= 1 else 0.0,
|
||||
float(bbox[1]) if len(bbox) >= 2 else 0.0,
|
||||
)
|
||||
|
||||
for block in sorted(blocks, key=_sort_key):
|
||||
page = int(block.get("page", 0) or 0)
|
||||
bbox = block.get("bbox") or [0, 0, 0, 0]
|
||||
role = str(block.get("role") or "")
|
||||
zone = str(block.get("zone") or "")
|
||||
text = str(block.get("text") or block.get("block_content") or "").strip()
|
||||
if zone == "reference_zone" or role in ref_roles:
|
||||
layout_region = "reference_candidate"
|
||||
elif zone == "display_zone" or role in display_roles:
|
||||
layout_region = "display_zone"
|
||||
elif zone in tail_zones or role.startswith("backmatter"):
|
||||
layout_region = "tail_candidate"
|
||||
elif zone in fm_zones and role in side_roles:
|
||||
layout_region = "side_insert"
|
||||
else:
|
||||
layout_region = "body_flow"
|
||||
col_key = 0 if len(bbox) < 4 else (0 if float(bbox[0]) < 600 else 1)
|
||||
key = (page, layout_region, col_key)
|
||||
if key != last_key:
|
||||
band_seq += 1
|
||||
last_key = key
|
||||
facts.append(
|
||||
{
|
||||
"block_id": str(block.get("block_id") or ""),
|
||||
"reading_band_id": f"band_{band_seq:03d}",
|
||||
"display_cluster_candidate_id": f"disp_{page}_{col_key}" if layout_region == "display_zone" else "",
|
||||
"layout_region": layout_region,
|
||||
"boundary_before": (
|
||||
"weak"
|
||||
if role == "reference_item"
|
||||
else (
|
||||
"hard"
|
||||
if layout_region in {"reference_candidate", "display_zone"}
|
||||
and role not in {"figure_asset", "media_asset", "table_html"}
|
||||
else "none"
|
||||
)
|
||||
),
|
||||
"boundary_after": "none",
|
||||
"bridge_eligible": role == "unknown_structural" and not text and layout_region == "display_zone",
|
||||
}
|
||||
)
|
||||
return facts
|
||||
|
||||
|
||||
def _apply_zone_labels(blocks: list[dict], region_bus: dict[str, dict] | None) -> None:
|
||||
if not region_bus:
|
||||
return
|
||||
|
|
@ -2497,6 +2578,56 @@ def _apply_content_zone_fallback(blocks: list[dict], region_bus: dict[str, dict]
|
|||
block["zone"] = "body_zone"
|
||||
|
||||
|
||||
def _recover_empty_text_from_spans(blocks: list[dict]) -> None:
|
||||
"""Recover text for blocks where OCR detected spans but assembled text is empty.
|
||||
|
||||
When a block has ``raw_label == "text"``, non-empty ``span_metadata``, and
|
||||
empty block-level text, the text can be reconstructed from span content.
|
||||
These blocks are typically body paragraphs whose OCR assembly step dropped
|
||||
the text but whose individual spans retained it.
|
||||
"""
|
||||
recovered_count = 0
|
||||
for block in blocks:
|
||||
if str(block.get("text") or block.get("block_content") or "").strip():
|
||||
continue
|
||||
spans = block.get("span_metadata")
|
||||
if not spans or not isinstance(spans, list) or len(spans) == 0:
|
||||
continue
|
||||
raw_label = str(block.get("raw_label") or "")
|
||||
if raw_label not in {"text", "paragraph_title", "header"}:
|
||||
continue
|
||||
parts: list[str] = []
|
||||
for span in spans:
|
||||
st = str(span.get("text") or "").strip()
|
||||
if st and not st.isdigit() and len(st) > 1:
|
||||
parts.append(st)
|
||||
if parts:
|
||||
recovered = " ".join(parts)
|
||||
if not _looks_like_author_list(recovered) and not _looks_like_affiliation(recovered):
|
||||
block["text"] = recovered
|
||||
block["block_content"] = recovered
|
||||
block["_text_source"] = "span_recovery"
|
||||
block["role"] = "body_paragraph"
|
||||
block["seed_role"] = "body_paragraph"
|
||||
recovered_count += 1
|
||||
record_decision(
|
||||
block,
|
||||
stage="empty_text_span_recovery",
|
||||
old_role=block.get("role", "unknown_structural"),
|
||||
new_role=block["role"],
|
||||
reason=f"recovered {len(recovered)} chars from {len(parts)} spans",
|
||||
)
|
||||
# If span text is also empty, mark as needing PDF fallback so
|
||||
# downstream callers (audit refresh) can attempt backfill via PDF
|
||||
# and then re-normalize with seed_role set.
|
||||
if not parts:
|
||||
block["_needs_pdf_fallback"] = True
|
||||
if not block.get("text") and str(block.get("raw_label") or "") == "text":
|
||||
# Give a temporary seed_role so re-normalize treats it as body
|
||||
block["seed_role"] = block.get("seed_role") or "body_paragraph"
|
||||
return recovered_count
|
||||
|
||||
|
||||
def _sanitize_reference_zone_boundary(blocks: list[dict]) -> None:
|
||||
"""Defensive pass: ensure only reference_heading / reference_item live inside reference_zone.
|
||||
|
||||
|
|
@ -4544,7 +4675,13 @@ def _should_keep_formal_caption_seed(block: dict) -> bool:
|
|||
return False
|
||||
if marker in {"figure_number", "table_number"}:
|
||||
return True
|
||||
return zone == "display_zone" and style in {"legend_like", "support_like"} and text.lower().startswith(("fig", "figure", "table"))
|
||||
# Ponytail: accept as caption when text identifies itself as figure/table
|
||||
# label OR has substantial descriptive text typical of captions (>= 80 chars)
|
||||
# AND is in a display-paired zone or legend-like style
|
||||
text_lower = text.lower()
|
||||
looks_like_caption = text_lower.startswith(("fig", "figure", "table")) or len(text) >= 80
|
||||
in_display_context = zone in {"display_zone", "body_zone"} or style in {"legend_like", "support_like"}
|
||||
return looks_like_caption and in_display_context
|
||||
|
||||
|
||||
def _demote_early_frontmatter_body_leaks(blocks: list[dict]) -> None:
|
||||
|
|
@ -4581,6 +4718,13 @@ def _demote_early_frontmatter_body_leaks(blocks: list[dict]) -> None:
|
|||
block["role"] = "frontmatter_noise"
|
||||
block["render_default"] = False
|
||||
continue
|
||||
# Journal badge/marker text above the real title — not body content
|
||||
if lower in ("highlighted paper", "research highlight", "highlight", "research article",
|
||||
"review article", "original article", "editorial", "letter", "perspective",
|
||||
"communication", "technical note", "case report"):
|
||||
block["role"] = "frontmatter_noise"
|
||||
block["render_default"] = False
|
||||
continue
|
||||
|
||||
|
||||
def _restore_numbered_body_from_tail_hold(blocks: list[dict]) -> None:
|
||||
|
|
@ -4615,16 +4759,14 @@ def _restore_numbered_body_from_tail_hold(blocks: list[dict]) -> None:
|
|||
def normalize_document_structure(
|
||||
blocks: list[dict],
|
||||
source_frontmatter_anchors: dict | None = None,
|
||||
pdf_path: str | None = None,
|
||||
) -> tuple[DocumentStructure, list[dict]]:
|
||||
"""Analyze document structure and normalize roles.
|
||||
|
||||
Returns (document_structure, normalized_blocks).
|
||||
Normalization includes:
|
||||
- backmatter form classification
|
||||
- backmatter role normalization after boundary
|
||||
- tail body candidate promotion
|
||||
- tail spread ownership assignment
|
||||
"""
|
||||
Returns (document_structure, normalized_blocks).
|
||||
"""
|
||||
if pdf_path:
|
||||
from paperforge.worker.ocr_pdf_spans import backfill_missing_text_from_pdf
|
||||
backfill_missing_text_from_pdf(blocks, pdf_path)
|
||||
page_layouts = _build_page_layout_profiles(blocks)
|
||||
body_family_anchor = discover_body_family_anchor(blocks)
|
||||
reference_family_anchor = discover_reference_family_anchor(blocks)
|
||||
|
|
@ -4647,6 +4789,7 @@ def normalize_document_structure(
|
|||
)
|
||||
_exclude_frontmatter_side_from_body_flow(blocks)
|
||||
_exclude_tail_nonref_from_body_flow(blocks)
|
||||
_recover_empty_text_from_spans(blocks)
|
||||
|
||||
from paperforge.worker.ocr_roles import resolve_final_role
|
||||
|
||||
|
|
@ -4670,6 +4813,23 @@ def normalize_document_structure(
|
|||
if resolved.evidence:
|
||||
block.setdefault("evidence", []).extend(resolved.evidence)
|
||||
|
||||
try:
|
||||
layout_facts = compute_layout_facts(blocks)
|
||||
fact_by_id = {row["block_id"]: row for row in layout_facts if row.get("block_id")}
|
||||
for block in blocks:
|
||||
block_id = str(block.get("block_id") or "")
|
||||
fact = fact_by_id.get(block_id)
|
||||
if not fact:
|
||||
continue
|
||||
block["reading_band_id"] = fact["reading_band_id"]
|
||||
block["display_cluster_candidate_id"] = fact["display_cluster_candidate_id"]
|
||||
block["layout_region"] = fact["layout_region"]
|
||||
block["boundary_before"] = fact["boundary_before"]
|
||||
block["boundary_after"] = fact["boundary_after"]
|
||||
block["bridge_eligible"] = fact["bridge_eligible"]
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Backmatter heading candidate promotion: confirm candidates that are
|
||||
# followed by body-like text on the same page. This runs after role
|
||||
# resolution (which may have reclassified the candidate via editorial
|
||||
|
|
@ -5151,6 +5311,10 @@ def normalize_document_structure(
|
|||
"frontmatter_support",
|
||||
"structured_insert",
|
||||
"non_body_insert",
|
||||
"subsection_heading",
|
||||
"reference_item",
|
||||
"table_caption",
|
||||
"table_html",
|
||||
}:
|
||||
block["role_source"] = "structural_gate_fallback"
|
||||
continue
|
||||
|
|
@ -5173,3 +5337,34 @@ def normalize_document_structure(
|
|||
block["role"] = seed_role
|
||||
|
||||
return doc_structure, blocks
|
||||
|
||||
|
||||
def score_reference_corridor_membership(block: dict) -> dict:
|
||||
from paperforge.worker.ocr_reference_signals import score_reference_entry
|
||||
|
||||
text = str(block.get("text") or block.get("block_content") or "").strip()
|
||||
role = str(block.get("role") or "")
|
||||
layout_region = str(block.get("layout_region") or "")
|
||||
style = score_reference_entry(text)
|
||||
ref_membership_score = 0.0
|
||||
if role in {"reference_heading", "reference_item"}:
|
||||
ref_membership_score += 0.6
|
||||
if layout_region == "reference_candidate":
|
||||
ref_membership_score += 0.2
|
||||
ref_membership_score += float(style.get("confidence") or 0.0) * 0.3
|
||||
|
||||
intrusion = 0.0
|
||||
lower = text.lower()
|
||||
if role.startswith("backmatter"):
|
||||
intrusion += 0.4
|
||||
if any(token in lower for token in ("acknowledgements", "funding", "conflict of interest", "data availability")):
|
||||
intrusion += 0.8
|
||||
if layout_region in {"body_flow", "display_zone", "side_insert"} and role not in {"reference_heading", "reference_item"}:
|
||||
intrusion += 0.3
|
||||
return {
|
||||
"ref_membership_score": ref_membership_score,
|
||||
"non_ref_intrusion_score": intrusion,
|
||||
"reference_style_family": style["family"],
|
||||
"reference_style_confidence": style["confidence"],
|
||||
"accept_reference_membership": ref_membership_score >= 0.55 and intrusion < 0.5,
|
||||
}
|
||||
|
|
|
|||
|
|
@ -64,33 +64,24 @@ _INLINE_FIGURE_MENTION_VERBS = (
|
|||
)
|
||||
|
||||
|
||||
def _looks_like_inline_figure_mention(text: str) -> bool:
|
||||
t = " ".join(text.strip().split())
|
||||
lower = t.lower()
|
||||
|
||||
# Text starting with "Figure N." / "Fig. N." is a self-identifying
|
||||
# caption — never flag as body mention regardless of verb content.
|
||||
if re.match(r"^(?:Figure|Fig\.?)\s+\d+\.\b", t, re.IGNORECASE):
|
||||
return False
|
||||
if re.match(r"^(?:Figure|Fig\.?)\s+\d+\s+(?:" + "|".join(_INLINE_FIGURE_MENTION_VERBS) + r")\b", t, re.IGNORECASE):
|
||||
return True
|
||||
|
||||
if not re.search(r"\bfi(?:g(?:ure)?\.?\s*\d+)", lower):
|
||||
return False
|
||||
|
||||
# Explicitly NOT inline: Frontiers format FIGURE N | ...
|
||||
if re.match(r"^figure\s+\d+[a-z]?\s*\|", t, re.I):
|
||||
return False
|
||||
|
||||
# "as shown in Figure X" / "shown in Figure X" / "see Figure X"
|
||||
if re.search(r"\b(as shown in|shown in|see |according to|consistent with)\s+(fig(?:ure)?\.?\s*\d+)", lower):
|
||||
return True
|
||||
|
||||
# Long sentence with a prose verb
|
||||
words = t.split()
|
||||
if len(words) >= 10 and any(re.search(rf"\b{v}\b", lower) for v in _INLINE_FIGURE_MENTION_VERBS):
|
||||
return True
|
||||
|
||||
def _looks_like_inline_figure_mention(text: str, block: dict | None = None) -> bool:
|
||||
"""
|
||||
Ponytail: position and style are the reliable signals for distinguishing
|
||||
figure captions from body mentions. Text-content heuristics (verb lists,
|
||||
word-count thresholds) introduce errors for long captions and non-English
|
||||
prose. Default to False when the block is already classified as a figure
|
||||
caption in a display zone.
|
||||
"""
|
||||
if block is not None:
|
||||
role = str(block.get("role") or "")
|
||||
zone = str(block.get("zone") or "")
|
||||
style = str(block.get("style_family") or "")
|
||||
if role == "figure_caption":
|
||||
return False
|
||||
if role == "figure_caption_candidate" and zone != "body_zone":
|
||||
return False
|
||||
if zone == "display_zone" and style == "legend_like":
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
|
|
@ -179,7 +170,7 @@ def _is_body_mention(block: dict) -> bool:
|
|||
if raw_role == "figure_caption_candidate":
|
||||
text = block.get("text", "")
|
||||
if text and _FIGURE_NUMBER_PATTERN.search(text):
|
||||
if _looks_like_inline_figure_mention(text):
|
||||
if _looks_like_inline_figure_mention(text, block):
|
||||
return True
|
||||
return False
|
||||
if text and _looks_like_figure_narrative_prose(text):
|
||||
|
|
@ -367,6 +358,18 @@ def _asset_gap_below(a: dict, b: dict) -> float:
|
|||
return max(0.0, bb[1] - ab[3])
|
||||
|
||||
|
||||
def _asset_gap_left(a: dict, b: dict) -> float:
|
||||
ab = a.get("bbox") or [0, 0, 0, 0]
|
||||
bb = b.get("bbox") or [0, 0, 0, 0]
|
||||
return max(0.0, ab[0] - bb[2])
|
||||
|
||||
|
||||
def _asset_gap_above(a: dict, b: dict) -> float:
|
||||
ab = a.get("bbox") or [0, 0, 0, 0]
|
||||
bb = b.get("bbox") or [0, 0, 0, 0]
|
||||
return max(0.0, ab[1] - bb[3])
|
||||
|
||||
|
||||
def _grow_region_from_seed(seed: dict, others: list[dict], page_width: float) -> dict:
|
||||
group = [seed]
|
||||
growth_steps: list[dict] = []
|
||||
|
|
@ -379,10 +382,14 @@ def _grow_region_from_seed(seed: dict, others: list[dict], page_width: float) ->
|
|||
for candidate in remaining:
|
||||
reason = None
|
||||
cb = candidate.get("bbox") or [0, 0, 0, 0]
|
||||
if cb[0] >= group_bbox[2] and _asset_gap_right({"bbox": group_bbox}, candidate) <= page_width * 0.03:
|
||||
if cb[0] >= group_bbox[2] and _asset_gap_right({"bbox": group_bbox}, candidate) <= max(page_width * 0.08, 40):
|
||||
reason = "adjacent_right"
|
||||
elif cb[1] >= group_bbox[3] and _asset_gap_below({"bbox": group_bbox}, candidate) <= page_width * 0.03:
|
||||
elif cb[1] >= group_bbox[3] and _asset_gap_below({"bbox": group_bbox}, candidate) <= max(page_width * 0.08, 40):
|
||||
reason = "adjacent_below"
|
||||
elif cb[2] <= group_bbox[0] and _asset_gap_left({"bbox": group_bbox}, candidate) <= max(page_width * 0.08, 40):
|
||||
reason = "adjacent_left"
|
||||
elif cb[3] <= group_bbox[1] and _asset_gap_above({"bbox": group_bbox}, candidate) <= max(page_width * 0.08, 40):
|
||||
reason = "adjacent_above"
|
||||
if reason:
|
||||
group.append(candidate)
|
||||
growth_steps.append({"added_block_id": candidate.get("block_id", ""), "reason": reason})
|
||||
|
|
@ -429,7 +436,10 @@ def _media_clusters(blocks: list[dict], page_width: float = 1200) -> list[list[d
|
|||
if not b.get("_non_body_media")
|
||||
and (
|
||||
b.get("role") == "figure_asset"
|
||||
or (b.get("role") == "media_asset" and b.get("raw_label", "") in {"image", "chart", "figure"})
|
||||
or (b.get("role") == "media_asset" and (
|
||||
b.get("raw_label", "") in {"image", "chart", "figure"}
|
||||
or (b.get("raw_label", "") == "table" and "<img" in str(b.get("text") or "").lower())
|
||||
))
|
||||
)
|
||||
]
|
||||
media.sort(key=lambda b: (b.get("page", 0), b.get("bbox", [0, 0, 0, 0])[1], b.get("bbox", [0, 0, 0, 0])[0]))
|
||||
|
|
@ -499,7 +509,11 @@ def _build_candidate_figure_groups_from_assets(assets: list[dict], page_width: f
|
|||
b.get("role") == "figure_asset"
|
||||
or (
|
||||
b.get("role") == "media_asset"
|
||||
and (b.get("raw_label", "") in {"image", "chart", "figure"} or not str(b.get("raw_label", "")).strip())
|
||||
and (
|
||||
b.get("raw_label", "") in {"image", "chart", "figure"}
|
||||
or (b.get("raw_label", "") == "table" and "<img" in str(b.get("text") or "").lower())
|
||||
or not str(b.get("raw_label") or "").strip()
|
||||
)
|
||||
)
|
||||
)
|
||||
]
|
||||
|
|
@ -639,12 +653,14 @@ def _score_legend_to_group(
|
|||
)
|
||||
num_assets = len(group.get("media_blocks", []))
|
||||
if num_assets > 1 and match_score.get("score", 0) > 0:
|
||||
coherence_bonus = 0.05
|
||||
coherence_bonus = 0.15
|
||||
match_score = dict(match_score)
|
||||
match_score["score"] = min(1.0, match_score["score"] + coherence_bonus)
|
||||
match_score.setdefault("evidence", []).append("multi_asset_coherence_bonus")
|
||||
if match_score["score"] >= 0.6 and match_score.get("decision") == "ambiguous":
|
||||
if match_score["score"] >= 0.5 and match_score.get("decision") in ("ambiguous", "candidate"):
|
||||
match_score["decision"] = "matched"
|
||||
if match_score.get("decision") == "rejected":
|
||||
match_score["decision"] = "candidate"
|
||||
return match_score
|
||||
|
||||
|
||||
|
|
@ -875,7 +891,7 @@ def _allow_previous_page_sequential_match(cap: dict, asset: dict) -> bool:
|
|||
page_width = float(cap.get("page_width") or asset.get("page_width") or 1200)
|
||||
page_height = float(cap.get("page_height") or asset.get("page_height") or 1600)
|
||||
strong_numbered_caption = _extract_figure_number(str(cap.get("text", ""))) is not None
|
||||
post_reference_layout = str(cap.get("zone") or "") == "post_reference_backmatter_zone"
|
||||
post_reference_layout = str(cap.get("zone") or "") in {"post_reference_backmatter_zone", "display_zone"}
|
||||
starts_page = cap_top <= 220
|
||||
prior_asset_near_bottom = asset_bottom >= page_height * 0.7
|
||||
wide_caption = cap_width >= page_width * 0.6
|
||||
|
|
@ -944,6 +960,18 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
unresolved_clusters: list[dict] = []
|
||||
ambiguous_figures: list[dict] = []
|
||||
|
||||
def _collect_bridge_blocks(page: int) -> list[dict]:
|
||||
bridges: list[dict] = []
|
||||
for block in structured_blocks:
|
||||
if int(block.get("page", 0) or 0) != page:
|
||||
continue
|
||||
if not block.get("bridge_eligible"):
|
||||
continue
|
||||
if str(block.get("layout_region") or "") != "display_zone":
|
||||
continue
|
||||
bridges.append(block)
|
||||
return bridges
|
||||
|
||||
for block in structured_blocks:
|
||||
if block.get("page_width"):
|
||||
page_width = float(block["page_width"])
|
||||
|
|
@ -957,15 +985,14 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
continue
|
||||
is_validation_first_candidate = _is_validation_first_legend_candidate(block)
|
||||
if role in ("figure_caption", "figure_caption_candidate") or is_validation_first_candidate:
|
||||
if _is_body_mention(block):
|
||||
unmatched_legends.append(block)
|
||||
continue
|
||||
if (
|
||||
role == "figure_caption_candidate"
|
||||
and str(block.get("zone") or "") != "display_zone"
|
||||
and str(block.get("style_family") or "") != "legend_like"
|
||||
and _looks_like_figure_narrative_prose(block.get("text", ""))
|
||||
and _extract_figure_number(block.get("text", "")) is None
|
||||
if role == "figure_caption_candidate" and (
|
||||
str(block.get("zone") or "") == "body_zone"
|
||||
or (
|
||||
str(block.get("zone") or "") != "display_zone"
|
||||
and str(block.get("style_family") or "") != "legend_like"
|
||||
and _looks_like_figure_narrative_prose(block.get("text", ""))
|
||||
and _extract_figure_number(block.get("text", "")) is None
|
||||
)
|
||||
):
|
||||
continue
|
||||
if not _is_formal_legend(block.get("text", ""), block, page_width):
|
||||
|
|
@ -973,7 +1000,7 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
block,
|
||||
nearby_media=False,
|
||||
caption_style_match=False,
|
||||
body_prose_likelihood=_looks_like_inline_figure_mention(block.get("text", "")),
|
||||
body_prose_likelihood=_looks_like_inline_figure_mention(block.get("text", ""), block),
|
||||
)
|
||||
rejected_legends.append(block)
|
||||
else:
|
||||
|
|
@ -984,6 +1011,8 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
raw_label = str(block.get("raw_label", "")).strip()
|
||||
if raw_label in {"image", "chart", "figure_title", "figure"} or not raw_label:
|
||||
assets.append(block)
|
||||
elif raw_label == "table" and "<img" in str(block.get("text") or "").lower():
|
||||
assets.append(block)
|
||||
|
||||
numbered_legends = [leg for leg in legends if _extract_figure_number(leg.get("text", "")) is not None]
|
||||
unnumbered_legends = [leg for leg in legends if _extract_figure_number(leg.get("text", "")) is None]
|
||||
|
|
@ -1009,7 +1038,10 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
if current_has_asset and not existing_has_asset:
|
||||
_dedup_map[key] = legend
|
||||
elif current_has_asset and existing_has_asset or not current_has_asset and not existing_has_asset:
|
||||
pass
|
||||
existing_role = str(existing.get("role") or "")
|
||||
current_role = str(legend.get("role") or "")
|
||||
if current_role == "figure_caption" and existing_role != "figure_caption":
|
||||
_dedup_map[key] = legend
|
||||
|
||||
seen_fig_keys: set[tuple[str, int]] = set()
|
||||
deduped_legends: list[dict] = []
|
||||
|
|
@ -1038,11 +1070,13 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
|
||||
# Gate: suppress page_assets groups on pages with competing captions
|
||||
# so one big group doesn't swallow assets meant for multiple figures.
|
||||
# Only count legends with a real figure number — sub-panel labels
|
||||
# ("Tensile strength", "RND") are not captions and should not gate.
|
||||
_page_captions_for_gate: dict[int, set[str]] = {}
|
||||
for block in structured_blocks:
|
||||
role = block.get("role", "")
|
||||
if role in ("figure_caption", "figure_caption_candidate") and block.get("block_id"):
|
||||
_page_captions_for_gate.setdefault(block.get("page", 0), set()).add(block["block_id"])
|
||||
for legend in ordered_legends:
|
||||
fid = legend.get("block_id")
|
||||
if fid and _extract_figure_number(str(legend.get("text") or "")) is not None:
|
||||
_page_captions_for_gate.setdefault(legend.get("page", 0), set()).add(str(fid))
|
||||
_competing_caption_pages = {p for p, ids in _page_captions_for_gate.items() if len(ids) > 1}
|
||||
candidate_groups = [
|
||||
g
|
||||
|
|
@ -1054,13 +1088,22 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
page_caption_index = _formal_figure_caption_blocks(structured_blocks)
|
||||
for legend in ordered_legends:
|
||||
legend_page = legend.get("page", 0)
|
||||
legend_text = legend.get("text", "")
|
||||
legend_text = str(legend.get("text") or "")
|
||||
ns = _extract_figure_namespace(legend_text)
|
||||
fig_num = _extract_figure_number(legend_text)
|
||||
# Text may be empty (stored as [] in raw block) but marker_signature
|
||||
# still carries the figure number.
|
||||
if fig_num is None:
|
||||
ms = legend.get("marker_signature") or {}
|
||||
if ms.get("type") == "figure_number":
|
||||
try:
|
||||
fig_num = int(float(ms.get("number", 0) or 0))
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
is_validation_first_candidate = _is_validation_first_legend_candidate(legend)
|
||||
is_weak_truncated = _is_insufficient_legend_evidence(legend)
|
||||
|
||||
body_prose_likelihood = _looks_like_inline_figure_mention(legend_text)
|
||||
body_prose_likelihood = _looks_like_inline_figure_mention(legend_text, legend)
|
||||
|
||||
caption_score = score_figure_caption(
|
||||
legend,
|
||||
|
|
@ -1111,26 +1154,40 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
matched_assets = []
|
||||
elif len(close) > 1:
|
||||
close_sides = {_asset_vertical_side(legend, g) for _, g, _ in close}
|
||||
if "above" in close_sides and "below" in close_sides:
|
||||
ambiguous_figures.append(
|
||||
{
|
||||
"legend_block_id": legend.get("block_id", ""),
|
||||
"page": legend_page,
|
||||
"caption_score": caption_score,
|
||||
"candidates": [
|
||||
if "above" in close_sides and "below" in close_sides and fig_num is not None:
|
||||
# If one candidate is the page_assets group (all assets),
|
||||
# don't tie-break — it's the canonical match.
|
||||
page_assets_candidate = next((item for item in close if item[1].get("group_type") == "page_assets"), None)
|
||||
if page_assets_candidate is not None:
|
||||
close = [page_assets_candidate]
|
||||
else:
|
||||
# Caption sandwiched between two asset groups:
|
||||
# prefer above (standard: image above, caption below).
|
||||
above_only = [item for item in close if _asset_vertical_side(legend, item[1]) == "above"]
|
||||
if above_only:
|
||||
close = above_only
|
||||
else:
|
||||
ambiguous_figures.append(
|
||||
{
|
||||
"asset_block_id": g.get("media_blocks", [{}])[0].get("block_id", ""),
|
||||
"group_type": g.get("group_type", ""),
|
||||
"match_score": s,
|
||||
"legend_block_id": legend.get("block_id", ""),
|
||||
"page": legend_page,
|
||||
"text": legend_text,
|
||||
"figure_number": fig_num,
|
||||
"caption_score": caption_score,
|
||||
"candidates": [
|
||||
{
|
||||
"asset_block_id": g.get("media_blocks", [{}])[0].get("block_id", ""),
|
||||
"group_type": g.get("group_type", ""),
|
||||
"match_score": s,
|
||||
}
|
||||
for _, g, s in close
|
||||
],
|
||||
"hold_reason": "close_asset_tie",
|
||||
}
|
||||
for _, g, s in close
|
||||
],
|
||||
"hold_reason": "close_asset_tie",
|
||||
}
|
||||
)
|
||||
ambiguous = True
|
||||
matched_assets = []
|
||||
close = []
|
||||
)
|
||||
ambiguous = True
|
||||
matched_assets = []
|
||||
close = []
|
||||
if not close:
|
||||
pass
|
||||
else:
|
||||
|
|
@ -1177,11 +1234,15 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
if len(matched_assets) > 1:
|
||||
region_match["cluster_bbox"] = best_group.get("cluster_bbox", [0, 0, 0, 0])
|
||||
else:
|
||||
if fig_num is None and str(legend.get("role") or "") in ("figure_caption", "figure_caption_candidate") and len(legend_text) < 80:
|
||||
unmatched_legends.append(legend)
|
||||
continue
|
||||
ambiguous_figures.append(
|
||||
{
|
||||
"legend_block_id": legend.get("block_id", ""),
|
||||
"page": legend_page,
|
||||
"caption_score": caption_score,
|
||||
"figure_number": fig_num,
|
||||
"candidates": [
|
||||
{
|
||||
"asset_block_id": g.get("media_blocks", [{}])[0].get("block_id", ""),
|
||||
|
|
@ -1190,10 +1251,13 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
}
|
||||
for _, g, s in close
|
||||
],
|
||||
"hold_reason": "close_unconfirmed_match",
|
||||
}
|
||||
)
|
||||
ambiguous = True
|
||||
matched_assets = []
|
||||
ambiguous = True
|
||||
matched_assets = []
|
||||
else:
|
||||
best_gi, best_group, best_score = candidates[0]
|
||||
if best_score["decision"] == "matched":
|
||||
|
|
@ -1216,11 +1280,15 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
if len(matched_assets) > 1:
|
||||
region_match["cluster_bbox"] = best_group.get("cluster_bbox", [0, 0, 0, 0])
|
||||
else:
|
||||
if fig_num is None and str(legend.get("role") or "") in ("figure_caption", "figure_caption_candidate"):
|
||||
unmatched_legends.append(legend)
|
||||
continue
|
||||
ambiguous_figures.append(
|
||||
{
|
||||
"legend_block_id": legend.get("block_id", ""),
|
||||
"page": legend_page,
|
||||
"caption_score": caption_score,
|
||||
"figure_number": fig_num,
|
||||
"candidates": [
|
||||
{
|
||||
"asset_block_id": best_group.get("media_blocks", [{}])[0].get("block_id", ""),
|
||||
|
|
@ -1228,9 +1296,11 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
"match_score": best_score,
|
||||
}
|
||||
],
|
||||
"hold_reason": "single_unconfirmed_match",
|
||||
}
|
||||
)
|
||||
ambiguous = True
|
||||
ambiguous = True
|
||||
|
||||
is_legend_only = len(matched_assets) == 0
|
||||
|
||||
|
|
@ -1238,7 +1308,7 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
unmatched_legends.append(legend)
|
||||
continue
|
||||
|
||||
if is_weak_truncated and is_validation_first_candidate:
|
||||
if is_weak_truncated and is_validation_first_candidate and is_legend_only:
|
||||
held_figures.append(
|
||||
{
|
||||
"figure_id": f"held_figure_{len(held_figures) + 1:03d}",
|
||||
|
|
@ -1256,25 +1326,49 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
unmatched_legends.append(legend)
|
||||
continue
|
||||
|
||||
# When multiple truncated legends ("Fig. 4" only) share a page,
|
||||
# they are sub-panel labels on a multi-figure gallery — skip them
|
||||
# and let the shared caption claim all assets.
|
||||
if is_weak_truncated:
|
||||
ambiguous_figures.append(
|
||||
{
|
||||
"legend_block_id": legend.get("block_id", ""),
|
||||
"page": legend_page,
|
||||
"text": legend_text,
|
||||
"figure_number": fig_num,
|
||||
"caption_score": caption_score,
|
||||
"candidates": [],
|
||||
"hold_reason": "ambiguous_truncated_legend",
|
||||
"zone": legend.get("zone"),
|
||||
"style_family": legend.get("style_family"),
|
||||
"marker_signature": legend.get("marker_signature") or {},
|
||||
}
|
||||
page_truncated_count = sum(
|
||||
1 for leg in ordered_legends
|
||||
if leg.get("page", 0) == legend_page and _is_insufficient_legend_evidence(leg)
|
||||
)
|
||||
unmatched_legends.append(legend)
|
||||
continue
|
||||
if page_truncated_count >= 2:
|
||||
unmatched_legends.append(legend)
|
||||
continue
|
||||
# Single truncated legend on page: match normally with candidates
|
||||
if is_legend_only:
|
||||
weak_candidates = [
|
||||
{
|
||||
"asset_block_id": g.get("media_blocks", [{}])[0].get("block_id", ""),
|
||||
"group_type": g.get("group_type", ""),
|
||||
"match_score": s,
|
||||
}
|
||||
for _, g, s in (candidates or [])
|
||||
]
|
||||
ambiguous_figures.append(
|
||||
{
|
||||
"legend_block_id": legend.get("block_id", ""),
|
||||
"page": legend_page,
|
||||
"text": legend_text,
|
||||
"figure_number": fig_num,
|
||||
"caption_score": caption_score,
|
||||
"candidates": weak_candidates,
|
||||
"hold_reason": "ambiguous_truncated_legend",
|
||||
"zone": legend.get("zone"),
|
||||
"style_family": legend.get("style_family"),
|
||||
"marker_signature": legend.get("marker_signature") or {},
|
||||
}
|
||||
)
|
||||
unmatched_legends.append(legend)
|
||||
continue
|
||||
# Single truncated legend with matched assets — proceed to confirmation
|
||||
|
||||
if is_legend_only:
|
||||
if fig_num is None and str(legend.get("role") or "") in ("figure_caption_candidate", "figure_caption"):
|
||||
unmatched_legends.append(legend)
|
||||
continue
|
||||
weak_entry = {
|
||||
"legend_block_id": legend.get("block_id", ""),
|
||||
"page": legend_page,
|
||||
|
|
@ -1323,6 +1417,9 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
"flags": [],
|
||||
"caption_score": caption_score,
|
||||
}
|
||||
local_bridges = _collect_bridge_blocks(int(legend_page or 0))
|
||||
entry["asset_block_ids"] = [str(a.get("block_id", "")) for a in matched_assets if a.get("block_id")]
|
||||
entry["bridge_block_ids"] = [str(b.get("block_id", "")) for b in local_bridges if b.get("block_id")]
|
||||
if len(matched_assets) > 1:
|
||||
entry["cluster_bbox"] = _cluster_bbox([a.get("bbox", [0, 0, 0, 0]) for a in matched_assets])
|
||||
matched_figures.append(entry)
|
||||
|
|
@ -1433,6 +1530,103 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
matched_figures.extend(sidecar_promoted)
|
||||
used_asset_page_ids.update(sidecar_consumed_ids)
|
||||
|
||||
# Preproof legend-bundling: when a page packs 3+ figure captions
|
||||
# with zero same-page assets, match them 1:1 by page order to
|
||||
# subsequent pages that each hold unclaimed assets.
|
||||
if unmatched_legends and unmatched_assets:
|
||||
page_captions: dict[int, list[dict]] = {}
|
||||
for leg in unmatched_legends:
|
||||
cp = int(leg.get("page", 0) or 0)
|
||||
if _extract_figure_number(str(leg.get("text", ""))) is not None:
|
||||
page_captions.setdefault(cp, []).append(leg)
|
||||
for cp, caps in sorted(page_captions.items()):
|
||||
if len(caps) < 3:
|
||||
continue
|
||||
page_has_assets = any(
|
||||
a.get("page", 0) == cp for a in unmatched_assets
|
||||
)
|
||||
if page_has_assets:
|
||||
continue
|
||||
caps_sorted = sorted(caps, key=lambda b: (b.get("bbox") or [0, 0, 0, 0])[1])
|
||||
# Collect subsequent pages with unclaimed assets and no captions
|
||||
asset_pages: dict[int, list[dict]] = {}
|
||||
for ast in unmatched_assets:
|
||||
ap = int(ast.get("page", 0) or 0)
|
||||
bid = ast.get("block_id", "")
|
||||
if ap <= cp:
|
||||
continue
|
||||
if bid and (ap, bid) in used_asset_page_ids:
|
||||
continue
|
||||
asset_pages.setdefault(ap, []).append(ast)
|
||||
page_order = sorted(asset_pages.keys())
|
||||
# Validate: no body/table blocks between legend page and first asset page,
|
||||
# and each asset page is free of competing body/table text.
|
||||
_NON_PURE_ROLES = {"body_paragraph", "section_heading", "subsection_heading",
|
||||
"table_caption", "table_asset", "table_html",
|
||||
"backmatter_heading", "backmatter_body", "reference_item"}
|
||||
intervening_pages = set(range(cp + 1, page_order[0])) if page_order else set()
|
||||
intervening_body = any(
|
||||
b.get("page", 0) in intervening_pages and b.get("role", "") in _NON_PURE_ROLES
|
||||
for b in structured_blocks
|
||||
)
|
||||
if intervening_body:
|
||||
continue
|
||||
valid_pages = []
|
||||
for ap in page_order:
|
||||
page_has_body = any(
|
||||
b.get("page", 0) == ap and b.get("role", "") in _NON_PURE_ROLES
|
||||
for b in structured_blocks
|
||||
)
|
||||
if not page_has_body:
|
||||
valid_pages.append(ap)
|
||||
if len(valid_pages) < len(caps_sorted):
|
||||
caps_sorted = caps_sorted[:len(valid_pages)]
|
||||
if not valid_pages:
|
||||
continue
|
||||
# Match captions to validated asset pages in order
|
||||
for idx, cap in enumerate(caps_sorted):
|
||||
if idx >= len(valid_pages):
|
||||
break
|
||||
ap = valid_pages[idx]
|
||||
page_assets = asset_pages[ap]
|
||||
if not page_assets:
|
||||
continue
|
||||
fn = _extract_figure_number(str(cap.get("text", "")))
|
||||
cap_ns = _extract_figure_namespace(str(cap.get("text", "")))
|
||||
fig_id = _format_figure_id(cap_ns, fn)
|
||||
cap_score = score_figure_caption(
|
||||
cap, nearby_media=True, caption_style_match=False, body_prose_likelihood=False
|
||||
)
|
||||
consumed = []
|
||||
for ast in page_assets:
|
||||
bid = ast.get("block_id", "")
|
||||
if bid:
|
||||
used_asset_page_ids.add((ap, bid))
|
||||
consumed.append({
|
||||
"block_id": bid,
|
||||
"bbox": ast.get("bbox", [0, 0, 0, 0]),
|
||||
})
|
||||
unmatched_legends = [l for l in unmatched_legends if l.get("block_id") != cap.get("block_id")]
|
||||
matched_figures.append({
|
||||
"figure_id": fig_id,
|
||||
"figure_namespace": cap_ns,
|
||||
"legend_block_id": cap.get("block_id", ""),
|
||||
"page": ap,
|
||||
"text": str(cap.get("text", "")),
|
||||
"figure_number": fn,
|
||||
"matched_assets": consumed,
|
||||
"asset_block_ids": [c["block_id"] for c in consumed],
|
||||
"bridge_block_ids": [],
|
||||
"caption_score": cap_score,
|
||||
"match_score": {"score": 0.3, "decision": "matched", "evidence": ["legend_bundle_fallback"]},
|
||||
"confidence": 0.3,
|
||||
"flags": ["legend_bundle_match"],
|
||||
})
|
||||
# De-dup ambiguous_figures: remove entries whose legend_block_id
|
||||
# was already matched by the bundle pass.
|
||||
bundle_legend_ids = {m["legend_block_id"] for m in matched_figures if "legend_bundle_match" in m.get("flags", [])}
|
||||
ambiguous_figures[:] = [a for a in ambiguous_figures if a.get("legend_block_id") not in bundle_legend_ids]
|
||||
|
||||
# Sequential fallback: match unmatched captions to remaining assets in reading order.
|
||||
# Captions and figures often appear on different pages — humans match them by
|
||||
# sequential reading order, not spatial proximity. This is a necessary tradeoff.
|
||||
|
|
@ -1487,7 +1681,9 @@ def build_figure_inventory(structured_blocks: list[dict], page_width: float = 12
|
|||
):
|
||||
asset = previous_page_asset
|
||||
if asset is None and future_page_asset is not None:
|
||||
asset = future_page_asset
|
||||
fap = future_page_asset.get("page", 0) or 0
|
||||
if fap in (cp, cp + 1):
|
||||
asset = future_page_asset
|
||||
if asset is None:
|
||||
continue
|
||||
|
||||
|
|
|
|||
|
|
@ -681,13 +681,30 @@ def resolve_final_role(
|
|||
"body_family_anchor",
|
||||
}
|
||||
if style_family in _NON_BODY_FAMILIES and style_family_authority in _STRONG_AUTHORITIES:
|
||||
if style_family == "legend_like" and _looks_like_late_figure_narrative_prose(str(block.get("text") or "")):
|
||||
pass # narrative prose stays body_paragraph
|
||||
else:
|
||||
if style_family == "legend_like":
|
||||
if str(block.get("zone") or "") == "body_zone":
|
||||
pass # body text with figure mention, not a caption
|
||||
elif _looks_like_late_figure_narrative_prose(str(block.get("text") or "")):
|
||||
pass # narrative prose stays body_paragraph
|
||||
else:
|
||||
_FAMILY_ROLE_MAP = {
|
||||
"legend_like": "figure_caption_candidate",
|
||||
}
|
||||
mapped_role = _FAMILY_ROLE_MAP.get(style_family)
|
||||
if mapped_role:
|
||||
return RoleAssignment(
|
||||
role=mapped_role,
|
||||
confidence=max(current_confidence, 0.78),
|
||||
evidence=[
|
||||
f"late role resolution: non-body family '{style_family}' overrides body_paragraph",
|
||||
f"style_family_authority={style_family_authority or 'none'}",
|
||||
f"context_source={context_source}",
|
||||
],
|
||||
)
|
||||
elif style_family in {"reference_like", "table_caption_like", "heading_like"}:
|
||||
_FAMILY_ROLE_MAP = {
|
||||
"reference_like": "reference_item",
|
||||
"table_caption_like": "table_caption_candidate",
|
||||
"legend_like": "figure_caption_candidate",
|
||||
"heading_like": "section_heading",
|
||||
}
|
||||
mapped_role = _FAMILY_ROLE_MAP.get(style_family)
|
||||
|
|
|
|||
Loading…
Reference in a new issue