# Phase 8 Implementation Plan — Deep Helper Deployment And Sandbox Regression Gate

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development

**Goal:** Turn the manual sandbox audit into a deterministic release gate. Fix `ld_deep.py` importability, create OCR-complete fixtures, extend smoke tests to cover regressions, verify docs.

**Architecture:** Extend existing test infrastructure (`test_smoke.py`, `test_ld_deep_config.py`) rather than creating standalone scripts.

**Tech Stack:** Python 3.11+, pytest, existing PaperForge Lite codebase.

---

## File Map

| File | Role |
|------|------|
| `setup_wizard.py` | Add `pip install -e .` step (DEEP-04) |
| `pipeline/worker/scripts/literature_pipeline.py` | Extend doctor with `ld_deep.py` importability check (DEEP-04) |
| `skills/literature-qa/scripts/ld_deep.py` | Add rollback-on-failure to `prepare_deep_reading()` (D-13..D-15) |
| `tests/sandbox/ocr-complete/` | New: static OCR fixture files (DEEP-06) |
| `tests/test_smoke.py` | Extend with Phase 8 smoke tests (REG-01, REG-02) |
| `tests/test_prepare_rollback.py` | New: test rollback behavior in prepare (D-13..D-15) |
| `README.md`, `docs/INSTALLATION.md`, `AGENTS.md`, `command/*.md` | Verify commands are extractable and executable (REG-03) |

---

## Tasks

### Task 1: Fix ld_deep.py Importability (DEEP-04)

**Files:**
- Modify: `setup_wizard.py` (add pip install step)
- Modify: `pipeline/worker/scripts/literature_pipeline.py` (extend doctor)

**1a. Add pip install step to setup wizard:**
- After writing `.env` (line ~1016), add a step that runs `pip install -e <repo_root>`
- `repo_root` is the directory containing `setup_wizard.py` itself (already known)
- Print clear status: "Installing paperforge package..."
- On failure: show warning but don't block (user may install later)

**1b. Extend doctor importability check:**
- In `run_doctor()` lines 3111-3118, upgrade the "Agent 脚本" check from "directory exists" to actual importability
- Run `python -c "import paperforge.config"` via subprocess
- Also verify `ld_deep.py` can be imported: `python -c "import importlib.util; spec = importlib.util.spec_from_file_location('ld_deep', ld_deep_script); mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)"`
- Status: pass = "paperforge and ld_deep importable", warn = "directory exists but import failed"

**1c. Test:**
- `test_setup_wizard_pip_install`: verify setup wizard calls pip install -e .
- `test_doctor_importability_check`: verify doctor checks import, not just directory existence

---

### Task 2: Create OCR-Complete Fixture (DEEP-06)

**Files:**
- New: `tests/sandbox/ocr-complete/TSTONE001/fulltext.md`
- New: `tests/sandbox/ocr-complete/TSTONE001/meta.json`
- New: `tests/sandbox/ocr-complete/TSTONE001/figure-map.json`
- New: `tests/sandbox/ocr-complete/TSTONE001/chart-type-map.json`

**Fixture content:**
- `fulltext.md`: Synthetic markdown with realistic figure captions (e.g., "Figure 1: Biomechanical comparison of suture anchor fixations...", "Table 1: Patient demographics...") and page markers (`<!-- page 1 -->`, `<!-- page 2 -->`).
- `meta.json`: `{ "zotero_key": "TSTONE001", "ocr_status": "done", "page_count": 3, "generated_at": "..." }`
- `figure-map.json`: Generated by running `python skills/literature-qa/scripts/ld_deep.py figure-map tests/sandbox/ocr-complete/TSTONE001/fulltext.md --key TSTONE001 --out tests/sandbox/ocr-complete/TSTONE001/figure-map.json`
- `chart-type-map.json`: Generated by running `python skills/literature-qa/scripts/ld_deep.py chart-type-scan tests/sandbox/ocr-complete/TSTONE001/figure-map.json --out tests/sandbox/ocr-complete/TSTONE001/chart-type-map.json`

**Note:** The fixture must be committed to git. Generate once, commit, and never regenerate (to keep tests deterministic).

**2a. Write fixture generation script (for one-time use):**
- Create `tests/sandbox/generate_ocr_fixture.py` that:
  1. Reads `tests/sandbox/exports/骨科.json` to get TSTONE001 metadata
  2. Writes synthetic `fulltext.md` with figure/table captions
  3. Runs `ld_deep.py figure-map` and `chart-type-scan` to produce JSON
  4. Writes `meta.json` with `ocr_status: done`

**2b. Generate and commit fixtures:**
- Run `python tests/sandbox/generate_ocr_fixture.py`
- Verify files are created
- Add `tests/sandbox/ocr-complete/` to `.gitignore`? No — commit fixtures (deterministic)
- But add `tests/sandbox/00_TestVault/` to `.gitignore` (generated vault output)

---

### Task 3: Extend Smoke Tests (REG-01, REG-02)

**Files:**
- Modify: `tests/test_smoke.py` (add new tests)
- New: `tests/test_prepare_rollback.py`

**3a. Test: ld_deep.py importability from deployed location:**
- Use `tests/sandbox/00_TestVault/` as the deployed vault
- Verify `python -c "import ld_deep"` from `00_TestVault/.opencode/skills/literature-qa/scripts/` succeeds
- Or use `spec_from_file_location` pattern from `test_ld_deep_config.py`

**3b. Test: ld_deep.py prepare produces scaffold:**
- Copy OCR-complete fixture into a `fixture_vault` under `99_System/PaperForge/ocr/TSTONE001/`
- Ensure library-record for TSTONE001 exists with `analyze: true`
- Ensure formal note for TSTONE001 exists
- Run `ld_deep.py prepare TSTONE001 --vault <fixture_vault>`
- Assert: returns status "ok"
- Assert: `figure-map.json` created
- Assert: `chart-type-map.json` created
- Assert: formal note contains `## 精读` section

**3c. Test: ld_deep.py queue:**
- Run `ld_deep.py queue --vault <fixture_vault> --format json`
- Assert: TSTONE001 appears in queue with `ocr_status: done`

**3d. REG-02 regression assertions (embedded in smoke tests):**
- `test_regression_doctor_env_name`: doctor checks `PADDLEOCR_API_TOKEN` (not old env name)
- `test_regression_per_domain_json`: doctor validates `tests/sandbox/exports/骨科.json` without false "missing library.json" error
- `test_regression_worker_path_json`: `paperforge paths --json` returns `worker_script` and `ld_deep_script` with correct paths
- `test_regression_bbt_pdf_path`: selection sync resolves `TSTONE001/TSTONE001.pdf` from `storage/TSTONE001/`
- `test_regression_metadata_fields`: library-record has non-empty `first_author` and `journal`
- `test_regression_queue_output`: `paperforge deep-reading --verbose` prints ready/waiting/blocked details (not just pending count)
- `test_regression_agent_importability`: deployed `ld_deep.py` is importable without `PYTHONPATH`

---

### Task 4: Add Doc Verification to Smoke Tests (REG-03)

**Files:**
- Modify: `tests/test_smoke.py` (add doc verification test)

**4a. Extract commands from docs:**
- Parse markdown code blocks from:
  - `README.md`
  - `docs/INSTALLATION.md`
  - `AGENTS.md`
  - `command/ld-deep.md`
  - `command/lp-ocr.md`
  - `command/lp-status.md`
  - `command/lp-selection-sync.md`
  - `command/lp-index-refresh.md`
- Filter for commands starting with `paperforge `, `python -m paperforge`, or `ld_deep.py `
- Replace `<vault>` placeholder with actual fixture vault path

**4b. Execute and validate:**
- For each extracted command:
  - Run in subprocess with `cwd=repo_root`
  - Assert: exit code 0 (or expected exit code)
  - Assert: no crash, no import error
- Report which doc files had unverifiable commands (with reason)

**4c. Test:**
- `test_doc_commands_executable`: runs the extraction + execution loop

---

### Task 5: Add Rollback to prepare_deep_reading (D-13..D-15)

**Files:**
- Modify: `skills/literature-qa/scripts/ld_deep.py` (prepare_deep_reading function)

**5a. Implement rollback:**
- Track files written during prepare:
  - `figure_map_path` (line 1089)
  - `chart_type_map_path` (line 1098)
  - `formal_note` (line 1121)
- Before writing, save original `formal_note` content
- Wrap steps 6-8 in try/except
- On any exception:
  1. Delete `figure_map_path` if created
  2. Delete `chart_type_map_path` if created
  3. Restore `formal_note` to original content
  4. Return error status

**5b. Test rollback:**
- `test_prepare_rollback_on_figure_map_failure`: mock `build_figure_map` to raise, assert no partial files remain
- `test_prepare_rollback_on_scaffold_failure`: mock `ensure_study_section` to raise, assert figure-map and chart-type-map are deleted, formal note restored
- `test_prepare_success_no_rollback`: normal flow, assert all files exist

---

## Test Plan

| Test | File | What it covers |
|------|------|---------------|
| `test_setup_wizard_pip_install` | test_smoke.py | setup wizard calls pip install -e . |
| `test_doctor_importability_check` | test_smoke.py | doctor checks import, not just directory |
| `test_ld_deep_import_from_deployed` | test_smoke.py | ld_deep.py importable from 00_TestVault |
| `test_prepare_produces_scaffold` | test_smoke.py | OCR fixture → figure-map + chart-type-map + 精读 |
| `test_queue_shows_ready_paper` | test_smoke.py | queue lists TSTONE001 with ocr_status done |
| `test_regression_doctor_env_name` | test_smoke.py | doctor checks PADDLEOCR_API_TOKEN |
| `test_regression_per_domain_json` | test_smoke.py | doctor validates per-domain exports |
| `test_regression_worker_path_json` | test_smoke.py | paths --json returns correct keys |
| `test_regression_bbt_pdf_path` | test_smoke.py | KEY/KEY.pdf resolves from storage/KEY/ |
| `test_regression_metadata_fields` | test_smoke.py | first_author and journal populated |
| `test_regression_queue_output` | test_smoke.py | deep-reading --verbose prints details |
| `test_regression_agent_importability` | test_smoke.py | ld_deep.py runs without PYTHONPATH |
| `test_doc_commands_executable` | test_smoke.py | extracted doc commands run successfully |
| `test_prepare_rollback_figure_map` | test_prepare_rollback.py | partial cleanup on figure-map failure |
| `test_prepare_rollback_scaffold` | test_prepare_rollback.py | full cleanup on scaffold failure |
| `test_prepare_success` | test_prepare_rollback.py | normal flow, no rollback |

---

## Verification

After all tasks:
1. Run `pytest tests/test_smoke.py -v` — all PASS
2. Run `pytest tests/test_prepare_rollback.py -v` — all PASS
3. Run `python -m paperforge doctor` in `tests/sandbox/00_TestVault/` — passes importability check
4. Verify `tests/sandbox/ocr-complete/` is committed to git
5. Verify no live PaddleOCR API calls in any test

---

*Plan created: 2026-04-24*
*Phase: 08-deep-helper-deployment*
