---
phase: 25-surface-convergence-doctor-repair
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - paperforge/worker/status.py
  - paperforge/worker/asset_index.py (minor: add index summarization helper)
  - tests/test_status.py
autonomous: true
requirements:
  - SURF-01
  - SURF-02
  - MIG-01
  - MIG-03

must_haves:
  truths:
    - "paperforge status --json reads from canonical index, showing lifecycle, health, and maturity aggregates"
    - "paperforge status --json includes paper_count, lifecycle_level_counts, health_aggregate (4 dimensions), maturity_distribution"
    - "paperforge status falls back to filesystem scan when canonical index is unavailable"
    - "paperforge doctor shows a new 'Index Health' section with PDF Health, OCR Health, Note Health, and Asset Health counts"
    - "paperforge doctor detects stale configs, old index formats, old Base/templates, and partial OCR assets"
  artifacts:
    - path: paperforge/worker/status.py
      provides: "Refactored run_status() reading from canonical index; run_doctor() with Index Health section"
    - path: paperforge/worker/asset_index.py
      provides: "New helper to summarize index entries (lifecycle counts, health aggregates, maturity distribution)"
    - path: tests/test_status.py
      provides: "Tests for index-backed status output and doctor Index Health section"
  key_links:
    - from: paperforge/worker/status.py:run_status
      to: paperforge/worker/asset_index.get_index_path / read_index
      via: "read_index(vault) to load canonical index, then aggregate lifecycle/health/maturity fields"
      pattern: "from paperforge\\.worker\\.asset_index import.*read_index"
    - from: paperforge/worker/status.py:run_doctor
      to: paperforge/worker/asset_index module
      via: "read canonical index, aggregate health dimension values (pdf_health, ocr_health, note_health, asset_health)"
      pattern: "health.*index|Index Health"
---

<objective>
Make `paperforge status` and `paperforge doctor` consume the canonical asset index as their data source, replacing legacy filesystem scanning. status --json outputs lifecycle/health/maturity aggregates. doctor gains a new "Index Health" section with per-dimension health counts.

Purpose: These two surfaces must consume the same canonical semantics as the plugin and Base views, forming the foundation of surface convergence (SURF-01, SURF-02). Doctor must also detect brownfield migration issues (MIG-01, MIG-03).

Output:
- Refactored `paperforge/worker/status.py` — run_status reads canonical index; run_doctor adds Index Health section
- `paperforge/worker/asset_index.py` — new `summarize_index()` helper for lifecycle/health/maturity aggregates
- `tests/test_status.py` — tests covering index-backed status output and doctor health section
</objective>

<execution_context>
@/C/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md
@/C/Users/Lin/.opencode/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/STATE.md
@.planning/ROADMAP.md
@.planning/phases/25-surface-convergence-doctor-repair/25-CONTEXT.md
@.planning/phases/24-derived-lifecycle-health-maturity/24-02-SUMMARY.md
@paperforge/worker/status.py
@paperforge/worker/asset_index.py
@paperforge/worker/asset_state.py

<interfaces>
From paperforge/worker/asset_index.py:
```python
def get_index_path(vault: Path) -> Path           # Resolve canonical index file path
def read_index(vault: Path) -> dict | list | None # Read existing index (envelope or legacy)
def build_index(vault: Path, verbose: bool) -> int # Full rebuild
def refresh_index_entry(vault: Path, key: str) -> bool  # Single-entry incremental
def build_envelope(items: list[dict]) -> dict     # Wrap items in versioned envelope
```

From paperforge/worker/asset_state.py:
```python
def compute_lifecycle(entry: dict) -> str          # indexed/pdf_ready/fulltext_ready/deep_read_done/ai_context_ready
def compute_health(entry: dict) -> dict[str, str]  # {pdf_health, ocr_health, note_health, asset_health}
def compute_maturity(entry: dict) -> dict          # {level, level_name, checks, blocking}
def compute_next_step(entry: dict) -> str          # sync/ocr/repair//pf-deep/rebuild index/ready
```

Canonical index entry carries these derived fields (from Phase 24):
```python
entry["lifecycle"]   # str: indexed|pdf_ready|fulltext_ready|deep_read_done|ai_context_ready
entry["health"]      # dict: {pdf_health, ocr_health, note_health, asset_health}
entry["maturity"]    # dict: {level, level_name, checks, blocking}
entry["next_step"]   # str: sync|ocr|repair|/pf-deep|rebuild index|ready
```

Current run_status() JSON output format (paperforge/worker/status.py:528-548):
```python
{
    "version": "...",
    "vault": "...",
    "system_dir": "...",
    "resources_dir": "...",
    "exports": int,
    "domains": int,
    "total_papers": record_count,
    "formal_notes": note_count,
    "bases": base_count,
    "ocr": {"total": N, "pending": N, "processing": N, "done": N, "failed": N},
    "path_errors": int,
    "env_configured": bool,
}
```
</interfaces>
</context>

<tasks>

<task type="auto" tdd="false">
<name>Task 1: Add summarize_index() helper to asset_index.py</name>
<files>
  paperforge/worker/asset_index.py
  tests/test_asset_index.py
</files>

<read_first>
  paperforge/worker/asset_index.py (current state — add new function at end)
  paperforge/worker/asset_state.py (derived field shapes — compute_lifecycle returns str, compute_health returns dict with 4 keys, compute_maturity returns dict with level)
</read_first>

<action>
Add a new function `summarize_index(vault: Path) -> dict | None` to `paperforge/worker/asset_index.py`. This helper reads the canonical index via `read_index(vault)`, validates it's an envelope dict, and returns summary aggregates consumed by `run_status()` and `run_doctor()`. Return `None` if index is missing/legacy/corrupt.

The returned dict must have this exact shape:
```python
{
    "paper_count": int,                      # len(items)
    "lifecycle_level_counts": {              # count of papers at each lifecycle state
        "indexed": int,                      # compute_lifecycle returns "indexed"
        "pdf_ready": int,
        "fulltext_ready": int,
        "deep_read_done": int,
        "ai_context_ready": int
    },
    "health_aggregate": {                    # per-dimension count of healthy/unhealthy
        "pdf_health": {"healthy": int, "unhealthy": int},
        "ocr_health": {"healthy": int, "unhealthy": int},
        "note_health": {"healthy": int, "unhealthy": int},
        "asset_health": {"healthy": int, "unhealthy": int}
    },
    "maturity_distribution": {               # count of papers at each maturity level (1-6)
        "1": int,
        "2": int,
        "3": int,
        "4": int,
        "5": int,
        "6": int
    }
}
```

Logic:
1. Call `read_index(vault)`. If `None` or `is_legacy_format(data) is True`, return `None`.
2. Get `items` from the envelope dict. If no items, return empty-count dict (all zeros).
3. For each item, read `lifecycle`, `health`, `maturity` fields (use `.get()` with safe defaults: lifecycle="pdf_ready", health={"pdf_health":"healthy","ocr_health":"healthy","note_health":"healthy","asset_health":"healthy"}, maturity={"level":1}).
4. Count: lifecycle states, health dimension "healthy"/non-"healthy" counts, maturity levels.
5. Return the aggregate dict.

Per D-04 (filesystem fallback): the caller (status.py) handles fallback when `summarize_index` returns `None`. The helper itself does NOT implement fallback.

IMPORTANT: Place this function AFTER `refresh_index_entry()` in the file. Use `# ---------------------------------------------------------------------------` comment separator. Use existing lazy import pattern if needed.

Write tests in `tests/test_asset_index.py`:
- test_summarize_index_returns_aggregates: Feed a mock envelope with 3 items (one per lifecycle state), verify counts match expected lifecycle_level_counts.
- test_summarize_index_missing_index_returns_none: When read_index returns None, verify return is None.
- test_summarize_index_legacy_format_returns_none: When read_index returns a bare list, verify return is None.
- test_summarize_index_empty_items_all_zeros: When items list is empty, verify all counts are 0.
</action>

<acceptance_criteria>
- `paperforge/worker/asset_index.py` contains `def summarize_index(vault: Path) -> dict | None:`
- Function returns dict with keys: `paper_count`, `lifecycle_level_counts`, `health_aggregate`, `maturity_distribution`
- Health aggregate has all 4 dimensions with `healthy`/`unhealthy` counts
- Returns None when index is missing or legacy format
- 4 new tests pass in `tests/test_asset_index.py`
</acceptance_criteria>

<verify>
  <automated>pytest tests/test_asset_index.py -x -q --tb=short -k "summarize"</automated>
</verify>

<done>
summarize_index() implemented and tested. Returns correct aggregates for mock data. Returns None for missing/legacy index.
</done>
</task>

<task type="auto" tdd="false">
<name>Task 2: Refactor run_status() to read from canonical index</name>
<files>
  paperforge/worker/status.py
  tests/test_status.py
</files>

<read_first>
  paperforge/worker/status.py:443-567 (current run_status implementation)
  paperforge/worker/asset_index.py (summarize_index from Task 1)
</read_first>

<action>
Refactor `run_status()` in `paperforge/worker/status.py` per decisions D-01, D-02, D-03, D-04:

**D-01:** `run_status()` reads from canonical index instead of filesystem scanning.
**D-02:** JSON output includes: `paper_count`, `lifecycle_level_counts`, `health_aggregate` (4 dimensions with count per status), `maturity_distribution`.
**D-03:** `run_status()` delegates to `asset_index.summarize_index()` + `asset_state` functions for derived data.
**D-04:** Filesystem scan is retained as fallback when canonical index is unavailable.

**Implementation steps:**

1. Add imports at top of status.py:
   ```python
   from paperforge.worker.asset_index import get_index_path, read_index, summarize_index
   ```
   (Use existing lazy import inside `run_status()` to preserve module-level import safety.)

2. Inside `run_status()`, after the existing `paths = pipeline_paths(vault)` line, add:
   ```python
   summary = summarize_index(vault)
   ```
   If `summary is not None`, use index data for derived fields. If `None`, fall back to the existing filesystem-scan code path.

3. The index-backed code path computes:
   - `total_papers` = summary["paper_count"]
   - `lifecycle_level_counts` = summary["lifecycle_level_counts"]
   - `health_aggregate` = summary["health_aggregate"]
   - `maturity_distribution` = summary["maturity_distribution"]

4. Update the JSON output dict (the `data = {...}` block) to include the four new fields:
   ```python
   data = {
       # ... existing fields preserved for backward compatibility ...
       "version": __import__("paperforge").__version__,
       "vault": str(vault),
       "system_dir": cfg["system_dir"],
       "resources_dir": cfg["resources_dir"],
       "exports": len(export_files),
       "domains": len(config.get("domains", [])),
       "total_papers": record_count,
       "formal_notes": note_count,
       "bases": base_count,
       "ocr": { ... },          # preserved
       "path_errors": path_error_count,   # preserved
       "env_configured": ...,    # preserved
       # NEW FIELDS (when index-backed):
       "lifecycle_level_counts": lifecycle_level_counts,   # dict or None
       "health_aggregate": health_aggregate,                # dict or None
       "maturity_distribution": maturity_distribution,      # dict or None
   }
   ```
   When falling back to filesystem scan, set the three new fields to `None`.
   The `formal_notes` and `book_count` (if present) should still be populated from filesystem scan always (they're informational, not derived from the index).

5. Preserve the text (non-JSON) output path. Add a new section when index data is available:
   ```
   index: {total_papers} papers
   lifecycle: indexed={N} pdf_ready={N} fulltext_ready={N} deep_read={N} ai_ready={N}
   health: pdf={healthy}/{unhealthy} ocr={healthy}/{unhealthy} note={healthy}/{unhealthy} asset={healthy}/{unhealthy}
   ```
   Only show if `summary is not None`.

Write tests in `tests/test_status.py`:
- `test_status_json_includes_lifecycle_health_maturity`: Create a mock vault with a valid canonical index containing 2 entries (one ai_context_ready, one indexed). Run `run_status(vault, json_output=True)`. Verify output JSON contains `lifecycle_level_counts`, `health_aggregate`, `maturity_distribution` with correct values.
- `test_status_json_fallback_when_index_missing`: Create a vault with no index file. Run `run_status(vault, json_output=True)`. Verify output JSON has lifecycle/health/maturity fields set to `None` and `total_papers` is derived from filesystem.
- `test_status_text_output_includes_index_section`: Verify text output contains "lifecycle:" line when index is present.

Verification: the existing `test_doctor_on_empty_vault` and other existing doctor tests must still pass (doctor is NOT changed in this task).
</action>

<acceptance_criteria>
- `run_status()` calls `summarize_index(vault)` and uses index data when available
- JSON output includes `lifecycle_level_counts`, `health_aggregate`, `maturity_distribution` (dict or None)
- Text output shows lifecycle counts line when index available
- Filesystem fallback path preserved — all 3 new fields set to None
- Existing doctor tests continue to pass
</acceptance_criteria>

<verify>
  <automated>pytest tests/test_status.py -x -q --tb=short && python -c "from paperforge.worker.status import run_status; print('imports OK')"</automated>
</verify>

<done>
status --json outputs lifecycle/health/maturity aggregates from canonical index. Falls back to filesystem when index missing. Text output shows lifecycle counts.
</done>
</task>

<task type="auto" tdd="false">
<name>Task 3: Add Index Health section to run_doctor()</name>
<files>
  paperforge/worker/status.py
  tests/test_status.py
</files>

<read_first>
  paperforge/worker/status.py:197-421 (current run_doctor implementation)
  paperforge/worker/asset_index.py (summarize_index from Task 1)
</read_first>

<action>
Add a new "Index Health" section to `run_doctor()` per decisions D-08, D-09, D-10.

**D-08:** New "Index Health" section in `paperforge doctor` output, aggregating from canonical index health fields.
**D-09:** Shows: PDF Health (OK/warn/fail), OCR Health (OK/warn/fail), Note Health, Asset Health counts.
**D-10:** Existing checks unchanged — this is additive.

**Implementation:**

1. Inside `run_doctor()` after the "Agent 脚本" check block and before the "print" section (around line 396), add:
   ```python
   # --- Index Health section (Phase 25: derived from canonical index) ---
   try:
       summary = summarize_index(vault)
   except Exception:
       summary = None
   
   if summary and summary["paper_count"] > 0:
       health = summary.get("health_aggregate", {})
       
       def _health_status(dim: dict) -> str:
           """Return 'pass' if all entries in this dimension are healthy, 'warn' if mixed, 'fail' if all unhealthy."""
           healthy = dim.get("healthy", 0)
           unhealthy = dim.get("unhealthy", 0)
           total = healthy + unhealthy
           if total == 0:
               return ("pass", "0 papers")
           if unhealthy == 0:
               return ("pass", f"{healthy}/{total} healthy")
           if healthy == 0:
               return ("fail", f"0/{total} healthy — run doctor/repair")
           return ("warn", f"{healthy}/{total} healthy, {unhealthy} issues")
       
       for dim_name, dim_key in [("PDF Health", "pdf_health"), ("OCR Health", "ocr_health"), 
                                  ("Note Health", "note_health"), ("Asset Health", "asset_health")]:
           dim = health.get(dim_key, {})
           if not isinstance(dim, dict):
               dim = {"healthy": 0, "unhealthy": 0}
           status, msg = _health_status(dim)
           add_check("Index Health", status, f"{dim_name}: {msg}")
   else:
       add_check("Index Health", "info", "No canonical index — run `paperforge sync` to generate")
   ```

2. Add brownfield migration detection (MIG-01, MIG-03) inside the Index Health section:
   - After checking health, read the index directly and check `schema_version`:
     ```python
     index_data = read_index(vault)
     if isinstance(index_data, dict):
         sv = index_data.get("schema_version", "0")
         if sv and int(sv) < 2:
             add_check("Index Health", "warn", f"Legacy index schema v{sv} — consider rebuild to v2", "Run `paperforge sync --rebuild-index`")
     ```
   - Check if old Base templates exist (pre-lifecycle format):
     ```python
     bases_dir = paths.get("bases")
     if bases_dir and bases_dir.exists():
         legacy_bases = 0
         for base_path in bases_dir.glob("*.base"):
             content = base_path.read_text(encoding="utf-8") if base_path.exists() else ""
             if "has_pdf" in content and "lifecycle" not in content:
                 legacy_bases += 1
         if legacy_bases > 0:
             add_check("Index Health", "warn", 
                       f"{legacy_bases} Base file(s) use legacy columns (has_pdf, do_ocr) instead of lifecycle",
                       "Run `paperforge sync` to regenerate Base views")
     ```
   - Check partial OCR assets:
     ```python
     ocr_dir = paths.get("ocr")
     if ocr_dir and ocr_dir.exists():
         partial_ocr = 0
         for meta_path in ocr_dir.glob("*/meta.json"):
             try:
                 meta = json.loads(meta_path.read_text(encoding="utf-8"))
                 status = str(meta.get("ocr_status", "")).strip().lower()
                 if status == "done_incomplete":
                     partial_ocr += 1
             except Exception:
                 pass
         if partial_ocr > 0:
             add_check("Index Health", "warn", f"{partial_ocr} partial OCR asset(s) found", "Re-run `paperforge ocr` on affected items")
     ```

Write tests in `tests/test_status.py`:
- `test_doctor_includes_index_health_section`: Create a vault with a valid canonical index. Run `run_doctor(vault)`. Verify output contains "Index Health" section.
- `test_doctor_index_health_with_mixed_health`: Create an index with entries at various health levels. Verify counts for each dimension.
- `test_doctor_index_health_no_index`: Create a vault with no index. Verify Index Health shows "No canonical index" info message.
- `test_doctor_detects_legacy_bases`: Create a mock Base file with `has_pdf` and no `lifecycle`. Verify doctor warns about legacy Base format.
</action>

<acceptance_criteria>
- `run_doctor()` output includes "Index Health" section with PDF/OCR/Note/Asset Health counts
- Missing index shows info message instead of crash
- Doctor detects legacy schema versions, legacy Base columns, and partial OCR assets
- All existing doctor checks unchanged
</acceptance_criteria>

<verify>
  <automated>pytest tests/test_status.py -x -q --tb=short -k "doctor" && python -c "from paperforge.worker.status import run_doctor; print('doctor import OK')"</automated>
</verify>

<done>
run_doctor() outputs Index Health section with per-dimension health counts and brownfield migration detection. All existing checks preserved.
</done>
</task>

</tasks>

<verification>
1. All 3 new summarization tests pass: `pytest tests/test_asset_index.py -k "summarize"`
2. All status tests pass: `pytest tests/test_status.py`
3. All existing doctor tests pass (no regressions)
4. `python -c "from paperforge.worker.status import run_status, run_doctor; from paperforge.worker.asset_index import summarize_index"` — all three functions importable
5. `python -c "from paperforge.worker.asset_index import summarize_index; help(summarize_index)"` shows docstring
</verification>

<success_criteria>
- SURF-01: status --json and doctor both read lifecycle/health/maturity from canonical index, establishing surface convergence
- SURF-02: Doctor Index Health section provides concrete findings about paper health from source-derived evidence
- MIG-01: Doctor detects stale formats, legacy Base columns, and partial OCR assets during brownfield migration
- MIG-03: Doctor Index Health explicitly identifies migration issues (legacy schema, old Base templates, partial OCR)
</success_criteria>

<output>
After completion, create `.planning/phases/25-surface-convergence-doctor-repair/25-01-SUMMARY.md`
</output>
