obsidian-plugin/lllin000_PaperForge
1
0
Fork 0
mirror of https://github.com/lllin000/PaperForge.git synced 2026-07-22 06:50:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs(57): create contract layer phase plans

Browse source 
4 plans, 12 tasks across 2 waves:
- 057-01: Core types (ErrorCode, PFResult/PFError) + round-trip tests
- 057-02: Status & doctor --json PFResult wrapping + contract tests
- 057-03: Sync & ocr --diagnose --json PFResult wrapping + contract tests
- 057-04: Dashboard command + plugin contract + contract tests

All plans pass frontmatter and structure validation.
This commit is contained in:
Research Assistant 2026-05-09 14:38:37 +08:00
parent a61f9fb6cb
commit 9a0843faec
7 changed files with 1387 additions and 393 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

452
.planning/ROADMAP.md
View file

@ -1,354 +1,98 @@
# Roadmap: PaperForge
**All milestones shipped up to v1.11.**
**Phase numbering:** Continuous. v1.10 ended at Phase 45. v1.11 ended at Phase 50. v2.0 starts at Phase 51.
---
## Milestones
- ✅ **v1.0 MVP** — Phases 1-5 (shipped 2026-04-23)
- ✅ **v1.1 Sandbox Onboarding** — Phases 6-8 (shipped 2026-04-24)
- ✅ **v1.2 Systematization & Cohesion** — Phases 9-10 (shipped 2026-04-24)
- ✅ **v1.3 Path Normalization & Architecture Hardening** — Phases 11-12 (shipped 2026-04-24)
- ✅ **v1.4 Code Health & UX Hardening** — Phases 13-19 (shipped 2026-04-28)
- ✅ **v1.5 Obsidian Plugin Setup Integration** — Phases 20-21 (shipped 2026-04-29)
- ✅ **v1.6 AI-Ready Literature Asset Foundation** — Phases 22-26 (shipped 2026-05-04)
- ✅ **v1.7 Context-Aware Dashboard** — Phases 27-30 (shipped 2026-05-04)
- ✅ **v1.8 AI Discussion & Deep-Reading Dashboard** — Phases 31-36 (shipped 2026-05-07)
- ✅ **v1.9 Frontmatter Rationalization & Library-Record Deprecation** — Phases 37-41 (shipped 2026-05-07)
- ✅ **v1.10 Dependency Cleanup** — Phases 42-45 (shipped 2026-05-07)
- ✅ **v1.11 Merge Gate — v1.9 Ripple Remediation** — Phases 46-50 (shipped 2026-05-07)
- 🚧 **v2.0 Testing Infrastructure — 6-Layer Quality Gates** — Phases 51-55 (planning)
*Archive: `.planning/milestones/`*
---
## Phases
<details>
<summary>✅ v1.6 AI-Ready Literature Asset Foundation (Phases 22-26) — SHIPPED 2026-05-04</summary>
- [x] Phase 22: Configuration Truth & Compatibility (3/3)
- [x] Phase 23: Canonical Asset Index & Safe Rebuilds (3/3)
- [x] Phase 24: Derived Lifecycle, Health & Maturity (2/2)
- [x] Phase 25: Surface Convergence, Doctor & Repair (3/3)
- [x] Phase 26: Traceable AI Context Packs (3/3)
</details>
<details>
<summary>✅ v1.7 Context-Aware Dashboard (Phases 27-30) — SHIPPED 2026-05-04</summary>
- [x] Phase 27: Component Library (2/2)
- [x] Phase 28: Dashboard Shell & Context Detection (2/2)
- [x] Phase 29: Per-Paper View (1/1)
- [x] Phase 30: Collection View (1/1)
</details>
<details>
<summary>✅ v1.8 AI Discussion & Deep-Reading Dashboard (Phases 31-36) — SHIPPED 2026-05-07</summary>
- [x] Phase 31: Bug Fixes — Restore version display; remove meaningless "ai" UI row
- [x] Phase 32: Deep-Reading Mode Detection — Plugin routes deep-reading.md to dedicated dashboard mode
- [x] Phase 33: Deep-Reading Dashboard Rendering — Status bar, Pass 1 summary, empty-state AI Q&A card
- [x] Phase 34: Jump to Deep Reading Button — Per-paper dashboard card links to deep-reading.md (completed 2026-05-06)
- [x] Phase 35: AI Discussion Recorder — Python module writes discussion.md + discussion.json into ai/ (completed 2026-05-06)
- [x] Phase 36: Integration Verification — End-to-end pipeline verified with CJK encoding and vault.adapter.read
</details>
<details>
<summary>✅ v1.9 Frontmatter Rationalization & Library-Record Deprecation (Phases 37-41) — SHIPPED 2026-05-07</summary>
- [x] Phase 37: Frontmatter Rationalization (1 plan)
- [x] Phase 38: Workspace Stabilization (1 plan)
- [x] Phase 39: Base View Fix
- [x] Phase 40: Library-Record Deprecation
- [x] Phase 41: Plugin Dashboard Sync
</details>
<details>
<summary>✅ v1.10 Dependency Cleanup (Phases 42-45) — SHIPPED 2026-05-07</summary>
**Milestone Goal:** Fix all code breakage and documentation staleness caused by v1.9's library-records deprecation and directory default changes.
- [x] **Phase 42: Core Pipeline Fix** — OCR, status, and sync workers read from formal notes, not library-records
- [x] **Phase 43: Repair & Directory Defaults** — Repair re-anchored; all hardcoded old directory defaults updated
- [x] **Phase 44: Documentation Update** — AGENTS.md, 5 skill files, and 3 docs reflect v1.9 structure
- [x] **Phase 45: Validation & Release Gate** — Tests pass; end-to-end OCR/status verification
</details>
### ✅ v1.11 Merge Gate — v1.9 Ripple Remediation (Shipped 2026-05-07)
**Milestone Goal:** Resolve all 27 findings from the v1.6-ai-ready-asset-foundation branch review before merging to master. Fix cascading v1.9 structural ripple across four root cause clusters: index path hardcoding, library-records residual traces, setup wizard TUI removal, and new module hardening gaps.
- [x] **Phase 46: Index Path Resolution** — 2 plans: config-resolved paths, env var/placeholder fixes (completed 2026-05-07)
- [x] **Phase 47: Library-Records Deprecation Cleanup** — Zero residual traces in production code and documentation (completed 2026-05-07)
- [x] **Phase 48: Textual TUI Removal** — 2 plans: TUI code removal, documentation updates (completed 2026-05-07)
- [x] **Phase 49: Module Hardening** — Production-grade safety guards in discussion.py, main.js, asset_state.py (completed 2026-05-07)
- [x] **Phase 50: Repair Blind Spots** — All 6 divergence types detected and handled by fix mode (completed 2026-05-07)
---
### 🚧 v2.0 Testing Infrastructure — 6-Layer Quality Gates (Planning)
**Milestone Goal:** Establish a multi-layer testing infrastructure covering version consistency (L0), Python unit tests (L1), CLI contracts (L2), plugin-backend integration (L3), temp vault E2E workflows (L4), user journey contracts (L5), and destructive scenarios (L6) — with CI matrix, golden datasets, and snapshot testing.
- [ ] **Phase 51: Testing Foundation** — Fixture hierarchy, L0 version checker, L1 unit test relocation, PR check CI
- [ ] **Phase 52: Golden Datasets & CLI Contracts** — Fixture files, CLI `--json` contract tests with snapshot assertions
- [ ] **Phase 53: Plugin Tests & Temp Vault E2E** — Vitest plugin tests, temp vault E2E workflows, Node 20 CI
- [x] **Phase 54: User Journey & Chaos Tests** — UX contract, journey scripts, destructive scenario tests, chaos CI (completed 2026-05-08)
- [ ] **Phase 55: CI Optimization & Consistency Audit** — Plasma matrix, full gate, path-filtered triggers, mock validation audit
---
## Phase Details
### Phase 42: Core Pipeline Fix
**Goal**: OCR, status, and sync workers read workflow state (do_ocr, analyze, ocr_status) from formal note frontmatter — same logic as the existing `get_analyze_queue()` pattern. Core workflow unbroken for new papers created post-v1.9.
**Depends on**: Nothing (first v1.10 phase; v1.9 shipped)
**Requirements**: WF-01, WF-02, WF-03, WF-04, SYN-01, SYN-02, SYN-03
**Success Criteria** (what must be TRUE):
1. Running `paperforge ocr` finds and processes papers whose formal note frontmatter has `do_ocr: true` — no library-records reads
2. `auto_analyze_after_ocr` writes `analyze: true` into the formal note frontmatter of the processed paper
3. `paperforge status` reports paper counts and OCR status counts sourced from formal notes + canonical index, not from library-records
4. `paperforge status` doctor checks (PDF path validation, wikilink format) sample from formal notes, not library-records
5. `paperforge sync` no longer creates empty library-records domain directories; `load_control_actions()` scans formal note frontmatter instead of library-records directory
6. Orphaned formal notes (no matching Zotero entry) are cleaned up from the Literature/ directory during sync
**Plans**: TBD
### Phase 43: Repair & Directory Defaults
**Goal**: Repair worker three-way divergence scan and path error detection re-anchored from library-records to formal notes + canonical index. All 14 hardcoded old directory defaults (`99_System`, `03_Resources`, `05_Bases`) updated across production code, setup wizard, validation script, .gitignore, and CLI help text to match `DEFAULT_CONFIG` (`System`, `Resources`, `Bases`).
**Depends on**: Phase 42 (formal note path model confirmed; repair needs the same scan pattern)
**Requirements**: REP-01, REP-02, REP-03, DEF-01, DEF-02, DEF-03, DEF-04, DEF-05, DEF-06, DEF-07
**Success Criteria** (what must be TRUE):
1. `paperforge repair` three-way divergence scan reads from formal note frontmatter, canonical index, and paper-meta.json — zero library-records reads
2. `paperforge repair --fix-paths` detects path errors by scanning formal notes and writes fixes into formal note frontmatter
3. All `cfg.get("system_dir", "99_System")` fallbacks in asset_index, sync, and repair use `"System"` instead
4. setup_wizard function signature defaults, validate_setup.py legacy fallbacks, and .gitignore patterns use clean directory names (`System`/`Resources`/`Bases`)
5. CLI `--help` text displays clean default directory names; setup_wizard no longer creates an empty control_dir
**Plans**: TBD
### Phase 44: Documentation Update
**Goal**: All user-facing and agent-facing documentation reflects the v1.9 simplified structure. Zero references to the deprecated library-records workflow remain in AGENTS.md, 5 skill files, or 3 docs files.
**Depends on**: Phase 43 (all code changes finalized; documentation describes the settled behavior)
**Requirements**: DOC-01, DOC-02, DOC-03, DOC-04, DOC-05, DOC-06, DOC-07, DOC-08, DOC-09
**Success Criteria** (what must be TRUE):
1. AGENTS.md contains zero references to library-records; frontmatter section shows only formal note fields
2. All 5 skill files (pf-sync, pf-ocr, pf-status, pf-paper, pf-deep) describe the formal-note-only workflow with no mention of library-records
3. docs/setup-guide.md directory structure diagram shows Literature/ workspace directories without library-records
4. docs/ARCHITECTURE.md data flow reflects the v1.9 simplified tracking layer (formal notes + canonical index, no library-record intermediate)
5. docs/COMMANDS.md sync description shows direct formal note generation without a two-phase library-record step
**Plans**: TBD
### Phase 45: Validation & Release Gate
**Goal**: All existing tests pass with zero regressions. End-to-end verification confirms OCR and status workers operate correctly on the new formal-note-based paths. Release gate met.
**Depends on**: Phase 44 (documentation may need test validation too; all code changes complete)
**Requirements**: VAL-01, VAL-02, VAL-03
**Success Criteria** (what must be TRUE):
1. Full test suite passes with zero failures (no silent regressions in OCR, repair, sync, or status workflows)
2. End-to-end: `paperforge ocr` correctly finds and processes a paper whose formal note has `do_ocr: true`
3. `paperforge status` output contains zero references to library-records (no `library_records: 0` or equivalent)
**Plans**: TBD
### Phase 46: Index Path Resolution
**Goal**: All 5 workspace-path fields in the canonical index (`paper_root`, `main_note_path`, `fulltext_path`, `deep_reading_path`, `ai_path`) use config-resolved `literature_dir` instead of hardcoded `"Literature/"`. All 11 downstream consumers resolve correct paths. Config env var typo and migration gaps fixed.
**Depends on**: Nothing (first v1.11 phase; v1.10 shipped)
**Requirements**: PATH-01, PATH-02, PATH-03, PATH-04, PATH-05, PATH-06
**Success Criteria** (what must be TRUE):
1. `paperforge sync` generates canonical index entries with `paper_root`, `main_note_path`, `fulltext_path`, `deep_reading_path`, and `ai_path` using the user's configured `literature_dir` — verified via `paperforge context <key> --json` showing correct paths
2. Plugin dashboard renders per-paper views using config-resolved paths from the index (not hardcoded `"Literature/"`) — verified by opening a paper dashboard after sync with a non-default literature_dir
3. Environment variable `PAPERFORGE_LITERATURE_DIR` correctly overrides `literature_dir` (no truncation to `PAPERFORGERATURE_DIR`) — verified via `paperforge paths --json`
4. Legacy `paperforge.json` with top-level `skill_dir` and `command_dir` settings migrates into `vault_config` on first sync — no orphaned top-level keys remain
5. Shipping `.base` templates contain zero `${LIBRARY_RECORDS}` placeholders — verified by inspecting generated Base files in a fresh vault
**Plans**: TBD
**Plans**: 2 plans
Plans:
- [x] 46-001-PLAN.md — Core path resolution: fix 5 hardcoded "Literature/" in asset_index.py, fix config.py (env var typo, library_records path, CONFIG_PATH_KEYS), fix test_config.py
- [x] 46-002-PLAN.md — Placeholder & Windows path cleanup: remove LIBRARY_RECORDS substitution in base_views.py, remove unnecessary backslash replace in discussion.py
**UI hint**: yes
### Phase 47: Library-Records Deprecation Cleanup
**Goal**: Zero library-records references remain in production code (status.py, sync.py, ld_deep.py), documentation (5 command skill files), or user-facing labels. Dead code removed, stale scan paths corrected, post-install instructions updated to single-command workflow.
**Depends on**: Phase 46 (sync.py docstring changes reference same config-resolved paths)
**Requirements**: LEGACY-01, LEGACY-02, LEGACY-03, LEGACY-04, LEGACY-05, LEGACY-06, LEGACY-07
**Success Criteria** (what must be TRUE):
1. `paperforge status` reports `formal_notes` count (label is `formal_notes`, not `library_records`) — output reflects post-v1.9 reality
2. Five command skill files (`pf-sync.md`, `pf-ocr.md`, `pf-status.md`, `pf-paper.md`, `pf-deep.md`) contain zero mentions of "library-records" — verified by `grep -r "library.record"` returning no hits in skill files
3. `paperforge sync` no longer constructs `record_path` or calls `parse_existing_library_record()` — dead code removed, sync completes without errors
4. Setup wizard post-install instructions describe a single `paperforge sync` workflow (not old `--selection`/`--index` two-phase flow)
5. `paperforge repair` docstring reads "Scan formal literature notes" (not "library-records") and `ld_deep.py` return dict contains only active path keys
**Plans**: 2 plans
Plans:
- [x] 47-001-PLAN.md — Python source cleanup: status.py label/scan path, sync.py dead code + docstrings, ld_deep.py records key, repair.py + discussion.py docstrings (LEGACY-01, 02, 03, 04, 07)
- [x] 47-002-PLAN.md — Documentation cleanup: setup_wizard.py post-install text, 10 command file copies in command/ + paperforge/command_files/ (LEGACY-05, 06)
### Phase 48: Textual TUI Removal
**Goal**: The broken Textual TUI setup wizard is removed entirely. `paperforge setup` (bare, no `--headless`) prints a help message redirecting users to `--headless` or the plugin settings tab. All TUI classes, import paths, and the `textual` optional dependency are purged. Documentation updated to reflect headless-only setup. `headless_setup()` and all shared utilities preserved intact.
**Depends on**: Nothing (standalone removal; no dependency on PATH or LEGACY phases)
**Requirements**: DEPR-01, DEPR-02, DEPR-03
**Success Criteria** (what must be TRUE):
1. Running `paperforge setup` (bare, without `--headless`) prints a clean help message redirecting to `paperforge setup --headless` or the Obsidian plugin settings tab — no `NameError` crash, no TUI launch attempt
2. `setup_wizard.py` contains zero Textual-related imports or classes — `WelcomeStep`, `DirOverviewStep`, `VaultStep`, `PlatformStep`, `DeployStep`, `DoneStep`, `SetupWizardApp`, `ContentSwitcher`, `StepScreen`, and all `from textual` import paths removed; verified by `rg "from textual" setup_wizard.py` returning no hits
3. All three documentation files (`docs/setup-guide.md`, `docs/INSTALLATION.md`, `README.md`) reference only `paperforge setup --headless` — no bare `paperforge setup` without `--headless` flag
4. Post-install instruction text and headless completion message describe headless-only workflow; `--non-interactive` CLI option removed; `textual` removed from project optional dependencies
5. `headless_setup()`, shared utilities (`EnvChecker`, `AGENT_CONFIGS`, `_copy_file_incremental`, `_merge_env_incremental`) preserved and fully functional — zero behavior change for the headless code path
**Plans**: 2 plans
- [x] `48-001-PLAN.md` — TUI code removal (DEPR-01, DEPR-03): remove textual imports/classes from setup_wizard.py, replace main() with help message, update cli.py help text, remove textual from pyproject.toml
- [x] `48-002-PLAN.md` — Documentation updates (DEPR-02): update setup-guide.md and INSTALLATION.md for headless-only workflow
### Phase 49: Module Hardening
**Goal**: New modules built during v1.6-v1.8 (discussion.py, asset_state.py, main.js) have production-grade safety guards: file locking prevents concurrent write corruption, markdown special characters are escaped, timestamps use UTC, API keys pass via environment not CLI args, DOM rendering avoids XSS vectors, and empty-state outputs are safe JSON.
**Depends on**: Phase 47 (discussion.py docstring fixes share file context with LEGACY-07)
**Requirements**: HARDEN-01, HARDEN-02, HARDEN-03, HARDEN-04, HARDEN-05, HARDEN-06, HARDEN-07
**Success Criteria** (what must be TRUE):
1. Two concurrent `/pf-paper` calls for the same paper do not corrupt `discussion.json` or `discussion.md` — file locking prevents interleaved write operations
2. Markdown special characters (`*`, `#`, `[`, `_`, `` ` ``) in QA question/answer fields are escaped before writing to `discussion.md` — no broken formatting when rendered in Obsidian
3. All timestamps in `discussion.json` use UTC (`datetime.now(timezone.utc)`) — no CST/UTC+8 hardcoded offset; verified by inspecting a newly created discussion session timestamp
4. Obsidian plugin spawns OCR subprocess with `PADDLEOCR_API_TOKEN` in environment variable (not command-line argument) — API key not visible in process list via Task Manager
5. Plugin directory tree renders via `createEl()` DOM API (not `innerHTML` assignment) — no XSS vector from user-configured directory names containing HTML/script tags
6. `paperforge status --json` returns `lifecycle_level_counts`, `health_aggregate`, and `maturity_distribution` as empty dicts `{}` (not `null`) when no canonical index exists — downstream JSON parsers do not crash on field access
**Plans**: 3 plans
Plans:
- [x] `49-001-PLAN.md` — discussion.py hardening: UTC timestamps (HARDEN-03), markdown escaping (HARDEN-02), file locking (HARDEN-01)
- [x] `49-002-PLAN.md` — main.js hardening: API key via env var (HARDEN-04), createEl() not innerHTML (HARDEN-05)
- [x] `49-003-PLAN.md` — asset_state.py + status.py hardening: reorder next_step checks (HARDEN-06), empty dicts not null (HARDEN-07)
**UI hint**: yes
### Phase 50: Repair Blind Spots
**Goal**: Repair worker three-way divergence detection covers all 6 divergence types (was missing the `ocr_status: pending` vs `meta done/failed` case). `--fix` mode handles every detected condition or produces explicit warnings for unhandled types. Silent exception swallowing replaced with logged warnings. Dead code removed.
**Depends on**: Phase 49 (repair.py shares logging patterns with discussion.py hardening)
**Requirements**: REPAIR-01, REPAIR-02, REPAIR-03, REPAIR-04
**Success Criteria** (what must be TRUE):
1. `paperforge repair` detects condition 4 divergence: `ocr_status: pending` in formal note frontmatter vs `done`/`failed` in `meta.json` — output includes these findings (previously silently skipped)
2. `paperforge repair --fix` handles all 6 detected divergence types — no silently skipped conditions; any unhandled type produces an explicit `[WARNING]` line in console output
3. `paperforge repair --fix` logs (rather than silently ignores) index write failures during fix operations — `logger.warning()` calls replace bare `except Exception: pass` blocks
4. Dead `load_domain_config` call and unused dict comprehension removed from `repair.py:196` — no unreachable code or unused imports
**Plans**: 1 plan
Plans:
- [x] 50-001-PLAN.md — All 4 REPAIR fixes: dead code removal, condition 4 detection, --fix mode coverage, silent exception logging
### Phase 51: Testing Foundation
**Goal**: Establish the testing framework — version consistency checker (L0), existing unit test relocation to `tests/unit/` (L1), 5-level hierarchical pytest fixtures, and PR check CI pipeline.
**Depends on**: Nothing (first v2.0 phase; v1.11 shipped)
**Requirements**: VC-01, VC-02, UNIT-01, UNIT-02, UNIT-03, UNIT-04, UNIT-05, UNIT-06, UNIT-07, UNIT-08, CI-01
**Success Criteria** (what must be TRUE):
1. `scripts/check_version_sync.py` validates all 6+ version declarations (`__init__.__version__`, manifest.json, versions.json, CHANGELOG) and fails on mismatch — CI gate blocks push on version drift
2. All existing 473+ tests pass under `tests/unit/` directory structure with zero behavior modifications — `pytest tests/unit/` succeeds
3. `tests/conftest.py` provides 5-level fixture hierarchy (`empty_vault` -> `config_vault` -> `vault_with_export` -> `vault_with_ocr` -> `full_test_vault`) — each level usable independently by downstream test layers
4. `ci-pr-checks.yml` runs L0 (version check on ubuntu) + L1 (unit tests on 3 OS x 3 Python matrix) with total wall-clock under 2 minutes
5. `pyproject.toml` updated with test markers (`unit`, `cli`, `e2e`, `journey`, `chaos`, `slow`), testpaths, and new dependencies (pytest-snapshot, pytest-timeout, pytest-mock, responses, coverage)
**Plans**: TBD
### Phase 52: Golden Datasets & CLI Contracts
**Goal**: Build the shared `fixtures/` golden dataset (Zotero JSON, PDF samples, mock OCR responses, expected snapshots) and CLI contract tests (L2) with subprocess invoker and shape-specific snapshot assertions.
**Depends on**: Phase 51 (fixture hierarchy conftest provides base structure; test runner configured)
**Requirements**: FIX-01, FIX-02, FIX-03, FIX-04, FIX-05, CLI-01, CLI-02, CLI-03
**Success Criteria** (what must be TRUE):
1. `fixtures/` directory contains 8+ Zotero JSON variants (valid, empty, malformed, missing keys, CJK content, multi-attachment, 3 path formats), 4 minimal valid PDFs (including CJK filenames), 5 mock OCR response fixtures (submit, poll, result, error, timeout), and expected output snapshots — all tracked in `MANIFEST.json` with `used_by`, `generated`, `desc` fields
2. All 7 CLI commands (`status`, `sync`, `ocr`, `doctor`, `repair`, `context`, `setup`) return stable `--json` output with consistent schema — error responses use `ok`, `error_code`, `message`, `details`, `suggestions` fields
3. `pytest-snapshot` tests pass with shape-specific assertions (normalized dynamic fields, subset matching) — snapshot updates require explicit `--snapshot-update` flag and deliberate commit
4. Mock OCR backend using `responses` library produces deterministic, replayable PaddleOCR responses for all API states — no external HTTP calls during test execution
**Plans**: 2 plans
Plans:
- [ ] 52-001-PLAN.md — Golden Datasets: Zotero JSON variants (10), PDF fixtures (4 generated), mock OCR responses (6+), expected snapshots, MANIFEST.json, vault_builder.py (FIX-01, FIX-02, FIX-03, FIX-04, FIX-05)
- [ ] 52-002-PLAN.md — CLI Contract Tests: conftest with cli_invoker + mock_ocr_backend, pytest-snapshot integration, contract tests for all 7 commands + error codes (CLI-01, CLI-02, CLI-03)
### Phase 53: Plugin Tests & Temp Vault E2E
**Goal**: Build plugin-backend integration tests (L3) with Vitest + obsidian-test-mocks, and full temp vault end-to-end tests (L4) covering sync, OCR, status, doctor, and repair workflows.
**Depends on**: Phase 52 (CLI contract outputs define the interface L3 plugin tests validate against; golden datasets provide E2E input data)
**Requirements**: PLUG-01, PLUG-02, PLUG-03, E2E-01, E2E-02, E2E-03, E2E-04, E2E-05, CI-04
**Success Criteria** (what must be TRUE):
1. `tests/plugin/` runs on Vitest + obsidian-test-mocks + jsdom — `resolvePythonExecutable`, `getPluginVersion`, and `checkRuntimeVersion` have passing tests
2. Plugin error classification covers all 5 error patterns (Python missing, import failed, version mismatch, pip install failure, timeout) — `buildRuntimeInstallCommand` and `parseRuntimeStatus` dispatch tests pass
3. Temp vault fixture (`tmp_path`-based) produces a disposable Vault with config, directories, mock Zotero data, and mock OCR state — usable by all E2E tests
4. Full E2E pipeline test: BBT JSON -> formal notes -> canonical index -> Base views completes in temp vault without external dependencies
5. OCR E2E test: mock PaddleOCR backend via `responses` HTTP interception processes `do_ocr: true` paper through pending -> processing -> done states
6. Multi-domain sync test verifies multiple Zotero collections sync correctly, producing domain-separated formal notes and index entries
7. Node 20 CI runner executes all plugin Vitest tests in `ci.yml` — L3 gate passes on PR to main
**Plans**: 2 plans
Plans:
- [ ] 53-001-PLAN.md — Plugin source extraction & Vitest tests (L3): extract src/runtime.js, src/errors.js, src/commands.js; set up Vitest + obsidian-test-mocks + jsdom; write & pass plugin tests; add Node 20 CI runner
- [ ] 53-002-PLAN.md — Temp vault E2E tests (L4): E2E conftest with temp vault fixture; sync pipeline, multi-domain sync, OCR mock E2E, status/doctor/repair E2E tests
**UI hint**: yes
### Phase 54: User Journey & Chaos Tests
**Goal**: Document and implement user journey tests (L5) against verifiable UX contracts, plus destructive/abnormal scenario tests (L6) with safety contracts, Docker isolation, and weekly CI schedule.
**Depends on**: Phase 53 (E2E vault infrastructure reused by journey tests; mock systems shared with chaos tests)
**Requirements**: JNY-01, JNY-02, JNY-03, CHAOS-01, CHAOS-02, CHAOS-03, CHAOS-04, CI-05
**Success Criteria** (what must be TRUE):
1. `docs/ux-contract.md` defines concrete, verifiable step sequences for installation, sync, OCR, and dashboard workflows — each step has a single measurable outcome
2. New user onboarding journey test (`install -> sync -> OCR -> analyze -> deep-read`) completes in temp vault with journey fixture pack at each stage
3. Daily workflow journey test (`existing user adds paper -> syncs -> OCRs -> reads`) completes in pre-configured temp vault with existing papers
4. `CHAOS_MATRIX.md` documents all destructive scenarios with triggers, expected behavior, and safety contracts — no undocumented failure modes
5. Corrupted input tests (malformed JSON, corrupt PDF, broken meta.json, missing frontmatter) produce graceful error messages — no unhandled crashes
6. Network failure tests (OCR API timeout, HTTP 401, 500, DNS unreachable) use mock backend and produce actionable error messages
7. Filesystem error tests (permission denied, locked files, missing directories) use isolation assertion (`assert "tmp" in str(vault)`) — no real vault damage
8. `ci-chaos.yml` runs on weekly schedule + manual trigger with Docker isolation — chaos tests excluded from regular CI gate
**Plans**: 3 plans
Plans:
- [x] 54-001-PLAN.md — UX Contract + Journey Tests: docs/ux-contract.md, journey fixture pack, onboarding + daily workflow tests (JNY-01, JNY-02, JNY-03)
- [x] 54-002-PLAN.md — Chaos Matrix + Chaos Tests: CHAOS_MATRIX.md, corrupted input, network failure, filesystem error tests with isolation guards (CHAOS-01, CHAOS-02, CHAOS-03, CHAOS-04)
- [x] 54-003-PLAN.md — CI Chaos Workflow: ci-chaos.yml with weekly schedule + manual trigger (CI-05)
### Phase 55: CI Optimization & Consistency Audit
**Goal**: Harden CI with plasma matrix strategy, full L0-L4 merge gate, path-filtered triggers, and cross-layer consistency audit that validates L1 mocks against L4 ground truth.
**Depends on**: Phase 54 (all test layers L0-L6 exist in CI; optimization decisions informed by actual run data)
**Requirements**: CI-02, CI-03
**Success Criteria** (what must be TRUE):
1. `ci.yml` full gate runs L0 through L4 on merge to main — `re-actors/alls-green` provides single-status check for branch protection
2. Plasma CI matrix: L1 on 3 OS x 3 Python (fast); L2 on 2 Python x 1 OS; L3-L5 on single config — total CI budget under configured concurrent runner limit
3. Path-filtered CI triggers prevent unnecessary jobs: changes to `paperforge/ocr.py` trigger L1+L2+L4; changes to `paperforge/plugin/main.js` trigger L3 only; version files trigger L0 only
4. Consistency audit test validates L1 mock expectations against L4 real pipeline output — `pytest tests/audit/` detects mock drift before it reaches production
**Plans**: 2 plans
Plans:
- [ ] 55-001-PLAN.md — Consistency Audit Tests: cross-layer consistency validation detecting L1 mock drift against L4 golden dataset ground truth
- [ ] 55-002-PLAN.md — Plasma Matrix CI Pipeline: ci.yml rewrite with L0-L5 merge gate, plasma matrix, path-filtered triggers, alls-green aggregator
---
## Progress
| Phase | Milestone | Plans Complete | Status | Completed |
|-------|-----------|----------------|--------|-----------|
| 31. Bug Fixes | v1.8 | — | Complete | 2026-05-07 |
| 32. Deep-Reading Mode Detection | v1.8 | — | Complete | 2026-05-07 |
| 33. Deep-Reading Dashboard Rendering | v1.8 | — | Complete | 2026-05-07 |
| 34. Jump to Deep Reading Button | v1.8 | 1/1 | Complete | 2026-05-06 |
| 35. AI Discussion Recorder | v1.8 | 1/1 | Complete | 2026-05-06 |
| 36. Integration Verification | v1.8 | — | Complete | 2026-05-07 |
| 37. Frontmatter Rationalization | v1.9 | 1/1 | Complete | 2026-05-07 |
| 38. Workspace Stabilization | v1.9 | 1/1 | Complete | 2026-05-07 |
| 39. Base View Fix | v1.9 | — | Complete | 2026-05-07 |
| 40. Library-Record Deprecation | v1.9 | — | Complete | 2026-05-07 |
| 41. Plugin Dashboard Sync | v1.9 | — | Complete | 2026-05-07 |
| 42. Core Pipeline Fix | v1.10 | — | Complete | 2026-05-07 |
| 43. Repair & Directory Defaults | v1.10 | — | Complete | 2026-05-07 |
| 44. Documentation Update | v1.10 | — | Complete | 2026-05-07 |
| 45. Validation & Release Gate | v1.10 | — | Complete | 2026-05-07 |
| 46. Index Path Resolution | v1.11 | 2/2 | Complete | 2026-05-07 |
| 47. Library-Records Deprecation Cleanup | v1.11 | 2/2 | Complete | 2026-05-07 |
| 48. Textual TUI Removal | v1.11 | 3/2 | Complete | 2026-05-07 |
| 49. Module Hardening | v1.11 | 3/3 | Complete | 2026-05-07 |
| 50. Repair Blind Spots | v1.11 | 1/1 | Complete | 2026-05-07 |
| 51. Testing Foundation | v2.0 | 0/0 | Not started | - |
| 52. Golden Datasets & CLI Contracts | v2.0 | 2/2 | Planning | 2026-05-08 |
| 53. Plugin Tests & Temp Vault E2E | v2.0 | 0/2 | Planning | - |
| 54. User Journey & Chaos Tests | v2.0 | 3/3 | Complete | 2026-05-08 |
| 55. CI Optimization & Consistency Audit | v2.0 | 0/2 | Planning | - |
---
*Roadmap updated: 2026-05-08 — v2.0 milestone phases created*
# Roadmap: PaperForge v2.1 — Contract-Driven Architecture & Engineering Hardening
## Overview
v2.1 transforms PaperForge from feature-stacked monoliths into a contract-driven system. Each phase builds on the last: quick consistency fixes unblock the work (Phase 56), stable JSON contracts define the machine-readable API surface (Phase 57), sync.py decomposes into testable adapters and a service layer (Phase 58), the state machine formalizes with enums and a field registry (Phase 59), and setup_wizard modularizes into focused classes with per-step JSON output (Phase 60). Every requirement maps to exactly one phase.
## Phases
- [ ] **Phase 56: Stop the Bleeding** — Quick consistency fixes: version sync, PyYAML, README/install unification, plugin version pinning
- [ ] **Phase 57: Contract Layer** — PFResult/PFError dataclasses, ErrorCode enum, CLI --json contracts (status/doctor/sync/ocr/dashboard) [Planned, 4 plans]
- [ ] **Phase 58: Service Extraction** — Decompose sync.py into adapters (bbt/zotero_paths/obsidian_frontmatter) and SyncService
- [ ] **Phase 59: State Machine & Field Registry** — PdfStatus/OcrStatus/Lifecycle enums, ALLOWED_TRANSITIONS, field_registry.yaml, doctor field checks
- [ ] **Phase 60: Setup Modularization** — SetupPlan/Checker/RuntimeInstaller/VaultInitializer/AgentInstaller/ConfigWriter classes, setup --headless --json
## Phase Details
### Phase 56: Stop the Bleeding
**Goal**: Quick consistency fixes that unblock all subsequent phases — version declarations synchronized, dependency declarations resolved, install documentation unified, and plugin runtime version pinned.
**Depends on**: Nothing (v2.0 complete)
**Requirements**: BLEED-01, BLEED-02, BLEED-03, BLEED-04
**Success Criteria** (what must be TRUE):
1. `scripts/check_version_sync.py` passes as a CI gate — all 6+ version declarations (__init__.py, manifest.json, versions.json, pyproject.toml, plugin manifest, docs) return consistent
2. PyYAML dependency is resolved — either added to pyproject.toml as explicit dependency, or the yaml module existence check is removed from `paperforge doctor` (single source of truth — no "optional dependency" ambiguity)
3. README, INSTALLATION, and setup-guide present a single unified primary install path — no conflicting recommendations across the three documents
4. Plugin runtime `pip install paperforge` pins to the plugin manifest version — no version drift possible between the Obsidian plugin and the Python package
**Plans**: 2 plans
Plans:
- [ ] 056-01-PLAN.md — Version sync CI gate script + PyYAML doctor hardening (BLEED-01, BLEED-02)
- [ ] 056-02-PLAN.md — Install doc unification + plugin version pinning (BLEED-03, BLEED-04)
### Phase 57: Contract Layer
**Goal**: Stable JSON contracts (PFResult/PFError dataclasses, ErrorCode enum) that CLI commands produce and the plugin consumes — defining the machine-readable API surface consumed by SYNC, STAT, and SETP phases.
**Depends on**: Phase 56
**Requirements**: CTRT-01, CTRT-02, CTRT-03, CTRT-04, CTRT-05, CTRT-06, CTRT-07, CTRT-08
**Success Criteria** (what must be TRUE):
1. `PFResult` and `PFError` dataclasses are defined in `paperforge/core/result.py` with `to_json()` serialization that survives round-trip (serialize → deserialize → equal)
2. All error codes are centralized in `ErrorCode` enum in `paperforge/core/errors.py` — no scattered string-based error codes remain in any production module
3. `paperforge status --json`, `doctor --json`, `sync --json`, and `ocr --diagnose --json` all return PFResult-format output with consistent `{ok, command, version, data, error}` shape
4. `paperforge dashboard --json` returns a stable UI contract with stats (papers, pdf_health, ocr_health, domain_counts) and actionable permissions (can_sync, can_ocr, can_copy_context)
5. Plugin reads dashboard data via `paperforge dashboard --json` CLI contract; fallback to direct `formal-library.json` reading remains available during the transition period (removed after 2 release cycles of stable PFResult)
**Plans**: 4 plans
**UI hint**: yes
Plans:
- [ ] 057-01-PLAN.md — Core contract types: ErrorCode enum + PFResult/PFError dataclasses + round-trip tests (CTRT-01, CTRT-02)
- [ ] 057-02-PLAN.md — Status & doctor --json PFResult wrapping + contract tests (CTRT-03, CTRT-04)
- [ ] 057-03-PLAN.md — Sync & ocr --diagnose --json PFResult wrapping + contract tests (CTRT-05, CTRT-06)
- [ ] 057-04-PLAN.md — Dashboard command + plugin contract integration + contract tests (CTRT-07, CTRT-08)
### Phase 58: Service Extraction
**Goal**: Monolithic sync.py decomposed into focused adapters and a SyncService class — each module independently testable with its own unit tests, sync.py reduced to a thin CLI dispatch layer.
**Depends on**: Phase 57
**Requirements**: SYNC-01, SYNC-02, SYNC-03, SYNC-04, SYNC-05
**Success Criteria** (what must be TRUE):
1. `paperforge/adapters/bbt.py` contains all BBT JSON parsing functions (`load_export_rows`, `_normalize_attachment_path`, `_identify_main_pdf`, `extract_authors`, `resolve_item_collection_paths`) — importable and testable in isolation from sync.py
2. `paperforge/adapters/zotero_paths.py` contains all path resolution functions (`obsidian_wikilink_for_pdf`, `absolutize_vault_path`, `obsidian_wikilink_for_path`) — no path resolution logic remains in sync.py
3. `paperforge/adapters/obsidian_frontmatter.py` contains frontmatter read/write/update operations using YAML parser (replacing regex-based parsing where possible) — importable and testable in isolation
4. `paperforge/services/sync_service.py` exists as a `SyncService` class that wraps the decomposed adapter modules; `worker/sync.py` becomes a thin dispatch layer with no business logic beyond orchestration
5. All extracted modules have passing unit tests covering BBT JSON variants, path formats (storage:/, absolute Windows, CJK filenames, spaces), and frontmatter edge cases — existing sync behavior preserved (no regressions)
**Plans**: TBD
### Phase 59: State Machine & Field Registry
**Goal**: Formalized state machine with explicit enums and allowed transitions, plus a field registry that `paperforge doctor` can validate against — making state changes predictable and field drift detectable.
**Depends on**: Phase 57
**Requirements**: STAT-01, STAT-02, STAT-03, STAT-04, STAT-05
**Success Criteria** (what must be TRUE):
1. `PdfStatus`, `OcrStatus`, and `Lifecycle` enums are defined in `paperforge/core/state.py` with explicit string values — every existing state string in the codebase maps to an enum member
2. `ALLOWED_TRANSITIONS` table defines legal state migrations; workers validate transitions before writing state changes — illegal transitions are rejected with clear, actionable error messages
3. `paperforge/schema/field_registry.yaml` defines every field in the system (formal note frontmatter, index entries, paper-meta.json) with public/required/type/owner/description metadata — complete coverage of all field-carrying structures
4. `paperforge doctor` checks field completeness against the registry — missing required fields produce actionable warnings with migration suggestions; unknown/invalid fields produce warnings to flag drift
5. Edge case: fields present in data but absent from registry trigger a "drift detected" diagnostic; fields in registry but absent from data trigger "missing required field" with severity levels
**Plans**: TBD
### Phase 60: Setup Modularization
**Goal**: Monolithic setup_wizard.py decomposed into six focused classes with explicit dependencies — enabling `paperforge setup --headless --json` to return per-step status that the Obsidian plugin can render as progress.
**Depends on**: Phase 57
**Requirements**: SETP-01, SETP-02, SETP-03, SETP-04, SETP-05, SETP-06, SETP-07
**Success Criteria** (what must be TRUE):
1. `SetupPlan`, `SetupChecker`, `RuntimeInstaller`, `VaultInitializer`, `AgentInstaller`, and `ConfigWriter` classes each exist with a single responsibility and explicit interface — no class exceeds its boundary
2. `paperforge setup --headless --json` returns per-step status — each step has independent `{ok, error, message}` fields; the plugin UI can render step-by-step progress from the JSON output
3. `ConfigWriter` writes `paperforge.json` atomically using tempfile + os.replace — no partial or corrupted config files possible on crash or interrupt
4. `RuntimeInstaller` handles `pip install` with explicit version pinning (from plugin manifest), progress output via callback, and classified errors using the ErrorCode enum from Phase 57
5. `SetupChecker` validates all preconditions (Python executable, pip availability, dependency health) before any installation step begins — failures produce classified, user-readable diagnostics
**Plans**: TBD
**UI hint**: yes
## Progress
**Execution Order:** 56 → 57 → 58 → 59 → 60
| Phase | Plans Complete | Status | Completed |
|-------|----------------|--------|-----------|
| 56. Stop the Bleeding | 0/2 | Ready to execute | — |
| 57. Contract Layer | 0/TBD | Not started | — |
| 58. Service Extraction | 0/TBD | Not started | — |
| 59. State Machine & Field Registry | 0/TBD | Not started | — |
| 60. Setup Modularization | 0/TBD | Not started | — |

56
.planning/STATE.md
View file

@ -1,19 +1,3 @@
---
gsd_state_version: 1.0
milestone: v2.0
milestone_name: Testing Infrastructure — 6-Layer Quality Gates
status: verifying
stopped_at: "Completed Phase 54: User Journey & Chaos Tests"
last_updated: "2026-05-08T16:53:18.641Z"
last_activity: 2026-05-08
progress:
total_phases: 14
completed_phases: 6
total_plans: 19
completed_plans: 19
percent: 25
---
# Project State
## Project Reference
@ -21,39 +5,34 @@ progress:
See: .planning/PROJECT.md (updated 2026-05-08)
**Core value:** Researchers always know what papers they have, what state those papers are in, and whether each paper is reliably usable by AI with traceable fulltext, figures, notes, and source links.
**Current focus:** Phase 53 — Plugin Tests & Temp Vault E2E
**Current focus:** Phase 56 — Stop the Bleeding
## Current Position
Phase: 53 of 55 (Plugin Tests & Temp Vault E2E)
Plan: 2 of 2 in current phase
Status: Phase complete — ready for verification
Last activity: 2026-05-08
Phase: 57 of 60 (Contract Layer)
Plan: 0 of 4 in current phase
Status: Ready to execute
Last activity: 2026-05-09 — Phase 57 plans created (4 plans, 12 tasks, 2 waves)
Progress: [##### ] 25%
Progress: [░░░░░░░░░░░░░░░░░░░░] 0%
## Performance Metrics
**Velocity:**
- Total plans completed: 2
- Average duration: ~45min
- Total execution time: ~90min
- Total plans completed: 0
- Average duration: — min
- Total execution time: 0.0 hours
**By Phase:**
| Phase | Plans | Total | Avg/Plan |
|-------|-------|-------|----------|
| 53 | 2 | ~90min | ~45min |
| — | — | — | — |
**Recent Trend:**
- Last 5 plans: 2 completed (53-001, 53-002)
- Trend: Testing/infrastructure phase
- No plans executed yet.
*Updated after each plan completion*
| Phase 54 P001-003 | 89 | 16 tasks | 12 files |
## Accumulated Context
@ -62,11 +41,10 @@ Progress: [##### ] 25%
Decisions are logged in PROJECT.md Key Decisions table.
Recent decisions affecting current work:
- All CI decisions from research (plasma matrix, path-filtered jobs, snapshot strategy) applied to phase design
- FIX requirements distributed: fixture hierarchy (conftest) in Phase 51, fixture data files (golden datasets) in Phase 52
- CI requirements distributed: PR check in Phase 51, Node runner in Phase 53, chaos workflow in Phase 54, optimization in Phase 55
- **53-001**: Used dependency injection pattern for src/ modules to work around vitest v2.1.x CJS/ESM mock limitations
- **53-002**: Moved test files into `paperforge/plugin/tests/` for vitest compatibility; e2e_cli_invoker returns (invoker, vault_path) tuple
- [v2.1]: Incremental extraction from sync.py — adapters first, then SyncService, keeping sync.py as thin shell. No full rewrite.
- [v2.1]: Dual JSON output during contract transition — old format + new PFResult side-by-side until 2 release cycles of stability.
- [v2.1]: Plugin keeps fallback to direct index reading during PFResult transition — removed after 2 stable release cycles.
- [v2.1]: Tests must not be modified to pass if code is broken — tests verify contracts, not implementation convenience.
### Pending Todos
@ -78,6 +56,6 @@ None yet.
## Session Continuity
Last session: 2026-05-08T16:53:18.638Z
Stopped at: Completed Phase 54: User Journey & Chaos Tests
Last session: 2026-05-09
Stopped at: Roadmap created — files written (ROADMAP.md, STATE.md, REQUIREMENTS.md traceability)
Resume file: None

231
.planning/phases/057-contract-layer/057-01-PLAN.md Normal file
View file

@ -0,0 +1,231 @@
---
phase: 57-contract-layer
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- paperforge/core/__init__.py
- paperforge/core/errors.py
- paperforge/core/result.py
- tests/cli/test_contract_roundtrip.py
autonomous: true
requirements:
- CTRT-01
- CTRT-02
must_haves:
truths:
- "ErrorCode enum exists with all 8 required members (PYTHON_NOT_FOUND, VERSION_MISMATCH, BBT_EXPORT_NOT_FOUND, OCR_TOKEN_MISSING, SYNC_FAILED, VALIDATION_ERROR, INTERNAL_ERROR, UNKNOWN)"
- "PFResult dataclass serializes to JSON with {ok, command, version, data, error} shape"
- "PFError dataclass serializes to JSON with {code, message, details} shape"
- "JSON round-trip: serialize PFResult -> JSON string -> deserialize -> equal to original"
artifacts:
- path: "paperforge/core/__init__.py"
provides: "Core package init"
min_lines: 1
- path: "paperforge/core/errors.py"
provides: "ErrorCode enum with all error codes"
exports: ["ErrorCode"]
- path: "paperforge/core/result.py"
provides: "PFResult and PFError dataclasses with to_dict/to_json/from_json"
exports: ["PFResult", "PFError"]
- path: "tests/cli/test_contract_roundtrip.py"
provides: "Round-trip unit tests for PFResult/PFError serialization"
min_lines: 40
key_links:
- from: "paperforge/core/result.py"
to: "paperforge/core/errors.py"
via: "import ErrorCode"
pattern: "from paperforge.core.errors import ErrorCode"
- from: "tests/cli/test_contract_roundtrip.py"
to: "paperforge/core/result.py"
via: "import PFResult"
pattern: "from paperforge.core.result import PFResult"
---
<objective>
Create the core contract types (PFResult, PFError dataclasses and ErrorCode enum) that all subsequent CLI --json wrapping and the dashboard command depend on. This is the foundational layer for the contract-driven architecture.
Purpose: Define the machine-readable API surface that every CLI command will produce and the plugin will consume.
Output: paperforge/core/errors.py (ErrorCode enum), paperforge/core/result.py (PFResult/PFError), round-trip unit tests
</objective>
<execution_context>
@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md
@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/057-contract-layer/057-CONTEXT.md
<interfaces>
From paperforge/__init__.py:
```python
__version__ = "1.4.17rc3"
```
From existing tests/cli/test_contract_helpers.py (assert_json_shape, normalize_snapshot, assert_valid_json available for reuse):
```python
def assert_json_shape(data: dict, required_keys: set[str], optional_keys: set[str] | None = None)
def assert_valid_json(output: str) -> dict
def normalize_snapshot(output: str, vault_path: str | None = None) -> str
```
</interfaces>
</context>
<tasks>
<task type="auto">
<name>Task 1: Create ErrorCode enum in paperforge/core/errors.py</name>
<files>
paperforge/core/__init__.py
paperforge/core/errors.py
</files>
<action>
Create the paperforge/core/ directory with paperforge/core/__init__.py containing a minimal package docstring.
Create paperforge/core/errors.py with:
ErrorCode(str, Enum) -- string enum using SCREAMING_SNAKE_CASE naming with values equal to names. The 8 members per user decision (D-01, D-02, D-03):
- PYTHON_NOT_FOUND: Python interpreter not found by the plugin
- VERSION_MISMATCH: Version mismatch between plugin manifest and installed package
- BBT_EXPORT_NOT_FOUND: Better BibTeX JSON export file missing or invalid
- OCR_TOKEN_MISSING: PaddleOCR API token not configured in .env
- SYNC_FAILED: Sync operation failed (selection-sync or index-refresh)
- VALIDATION_ERROR: Validation error -- use details.field for field-level granularity
- INTERNAL_ERROR: Unexpected internal error -- use details for trace/context info
- UNKNOWN: Catch-all fallback for unmapped/unexpected errors
Each member MUST have a docstring describing when it is used.
Add helper: @classmethod def from_exception(cls, exc: Exception) -> ErrorCode that maps common exception types (FileNotFoundError -> BBT_EXPORT_NOT_FOUND, ValueError -> VALIDATION_ERROR, etc.) to appropriate codes. Unknown exceptions map to UNKNOWN.
Per user decision (CONTEXT.md): ErrorCode enum ONLY -- no JSON schema generation, no auto-generation from field_registry.yaml (deferred to Phase 59).
</action>
<verify>
<automated>python -c "from paperforge.core.errors import ErrorCode; e = ErrorCode.UNKNOWN; assert isinstance(e, ErrorCode); assert e.value == 'UNKNOWN'; print('OK enum:', e.value)"</automated>
</verify>
<done>ErrorCode enum exists in paperforge/core/errors.py with 8 members, all SCREAMING_SNAKE_CASE, each with docstring, plus from_exception helper.</done>
</task>
<task type="auto">
<name>Task 2: Create PFResult and PFError dataclasses in paperforge/core/result.py</name>
<files>
paperforge/core/result.py
</files>
<action>
Create paperforge/core/result.py with two dataclasses using `from __future__ import annotations`.
PFError dataclass:
- code: str -- Error code string (e.g., "SYNC_FAILED", "VALIDATION_ERROR")
- message: str -- Human-readable error description
- details: dict = field(default_factory=dict) -- Optional structured payload (e.g., {"field": "pdf_path", "reason": "not found"})
- to_dict() -> dict: Returns {"code": code, "message": message, "details": details}
- WARNING: details may contain non-JSON-serializable types -- coerce with str() on serialization.
PFResult dataclass:
- ok: bool -- True when operation succeeded (even with partial results + warnings in data); False only when error is set
- command: str -- CLI command name (e.g., "status", "sync")
- version: str -- Version string from __version__. Use lazy import inside methods (not at module level) to avoid circular imports.
- data: Any = None -- Payload (dict for commands with structured output, list for collections, None for errors)
- error: PFError | None = None -- Set only when ok is False
__post_init__ validation: if ok is True and error is not None -> raise ValueError("ok=True requires error=None"). if ok is False and error is None -> raise ValueError("ok=False requires an error").
Methods:
- to_dict() -> dict: Returns {"ok": ok, "command": command, "version": version, "data": data, "error": error.to_dict() if error else None}
- to_json(indent: int = 2) -> str: JSON-dumps to_dict() with ensure_ascii=False, indent=indent, default=str for non-serializable values
- @staticmethod from_json(json_str: str) -> PFResult: Parse JSON string back to PFResult. If the "error" field is not None, construct PFError from the dict.
Module-level __all__ = ["PFResult", "PFError"].
Per user decisions:
- Top-level fields MUST be {ok, command, version, data, error} following Google JSON Style Guide
- Version: semver string from __version__ (e.g., "1.4.17rc3")
- ok=false only when error is set; ok=true even with partial results (add warnings sub-field in data)
- Error shape: {"code":"ERR_CODE","message":"Human text","details":{}}
</action>
<verify>
<automated>python -c "
from paperforge.core.result import PFResult, PFError
# Test OK result
r = PFResult(ok=True, command='status', version='1.4.17rc3', data={'papers': 5})
d = r.to_dict()
assert d['ok'] == True
assert d['command'] == 'status'
assert d['error'] is None
# Test error result
er = PFResult(ok=False, command='sync', version='1.4.17rc3', error=PFError(code='SYNC_FAILED', message='Sync failed', details={'key': 'val'}))
d2 = er.to_dict()
assert d2['ok'] == False
assert d2['error']['code'] == 'SYNC_FAILED'
# Test round-trip
j = r.to_json()
r2 = PFResult.from_json(j)
assert r2.ok == r.ok
assert r2.command == r.command
# Test validation
try:
PFResult(ok=True, command='test', version='1.0', error=PFError(code='X', message='y'))
assert False, 'Should have raised'
except ValueError:
pass
print('OK: round-trip + validation valid')
"</automated>
</verify>
<done>PFResult and PFError dataclasses in paperforge/core/result.py with to_dict(), to_json(), from_json(), and __post_init__ validation. Round-trip (serialize->deserialize->equal) verified.</done>
</task>
<task type="auto">
<name>Task 3: Write round-trip unit tests in tests/cli/test_contract_roundtrip.py</name>
<files>
tests/cli/test_contract_roundtrip.py
</files>
<action>
Create tests/cli/test_contract_roundtrip.py with pytest tests covering:
1. TestPFResultShape: Verify PFResult.to_dict() has exactly {ok, command, version, data, error} keys. Test both success and error states.
2. TestPFErrorShape: Verify PFError.to_dict() has exactly {code, message, details} keys. Test with empty dict, with populated details.
3. TestPFResultRoundTrip: Full round-trip tests:
- Serialize PFResult (ok=True, with data dict) -> JSON string -> deserialize -> assert equal
- Serialize PFResult (ok=False, with PFError) -> JSON string -> deserialize -> assert equal
- Serialize PFResult with list data -> JSON -> deserialize -> assert equal
4. TestPFResultValidation: Verify __post_init__ raises ValueError for:
- ok=True with error set
- ok=False without error
Use existing fixtures/conftest patterns (from tests/cli/conftest.py). No vault needed for these pure unit tests -- they test the dataclasses directly.
Keep tests compact: 4 test classes, no more than 5 assertions each.
</action>
<verify>
<automated>python -m pytest tests/cli/test_contract_roundtrip.py -v --tb=short</automated>
</verify>
<done>All round-trip tests pass: shape verification, serialization round-trip, validation error cases.</done>
</task>
</tasks>
<verification>
- python -c "from paperforge.core.errors import ErrorCode; print(list(ErrorCode))" shows all 8 members
- python -c "from paperforge.core.result import PFResult, PFError; print(PFResult(ok=True, command='status', version='1.0', data={}).to_json())" produces valid JSON
- pytest tests/cli/test_contract_roundtrip.py -v passes all tests
</verification>
<success_criteria>
- [ ] paperforge/core/errors.py exists with ErrorCode enum (8 members, SCREAMING_SNAKE_CASE, docstrings, from_exception)
- [ ] paperforge/core/result.py exists with PFResult and PFError dataclasses (to_dict, to_json, from_json, __post_init__ validation)
- [ ] Round-trip unit tests pass without requiring a vault
- [ ] All downstream phases (57-02, 57-03, 57-04) can import from paperforge.core
</success_criteria>
<output>
After completion, create .planning/phases/057-contract-layer/057-01-SUMMARY.md
</output>

281
.planning/phases/057-contract-layer/057-02-PLAN.md Normal file
View file

@ -0,0 +1,281 @@
---
phase: 57-contract-layer
plan: 02
type: execute
wave: 2
depends_on:
- 57-01
files_modified:
- paperforge/worker/status.py
- paperforge/cli.py
- paperforge/commands/status.py
- tests/cli/test_json_contracts.py
autonomous: true
requirements:
- CTRT-03
- CTRT-04
must_haves:
truths:
- "paperforge status --json returns PFResult envelope with data containing current flat dict shape"
- "paperforge doctor --json returns PFResult envelope with structured checks in data"
- "All PFResult output from status and doctor has valid {ok, command, version, data, error} shape"
artifacts:
- path: "paperforge/worker/status.py"
provides: "Modified run_status (PFResult wrapping) + run_doctor with --json support"
contains: "from paperforge.core.result import"
- path: "paperforge/cli.py"
provides: "doctor subparser with --json flag"
contains: "add_argument.*--json"
- path: "tests/cli/test_json_contracts.py"
provides: "Updated PFResult shape tests for status --json, new tests for doctor --json"
min_lines: 140
key_links:
- from: "paperforge/worker/status.py"
to: "paperforge/core/result.py"
via: "PFResult wrapper in JSON output path"
- from: "paperforge/worker/status.py"
to: "paperforge/core/errors.py"
via: "ErrorCode entries for doctor checks"
- from: "tests/cli/test_json_contracts.py"
to: "paperforge/worker/status.py"
via: "CLI subprocess invoker exercises both commands"
---
<objective>
Wrap the existing `paperforge status --json` output in a PFResult envelope (CTRT-03) and add `--json` output to `paperforge doctor` (CTRT-04). The existing flat dict output is preserved inside the `data` field (backward compatible). Doctor gets new structured JSON output with check results.
Purpose: All status and doctor commands produce machine-readable PFResult-format JSON with consistent shape.
Output: Modified status.py (PFResult wrapping for run_status, JSON for run_doctor), CLI updates, contract tests
</objective>
<execution_context>
@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md
@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/057-contract-layer/057-CONTEXT.md
<interfaces>
From paperforge/core/result.py (after Plan 57-01):
```python
@dataclass
class PFResult:
ok: bool
command: str
version: str
data: Any = None
error: PFError | None = None
def to_dict(self) -> dict: ...
def to_json(self, indent: int = 2) -> str: ...
@staticmethod def from_json(json_str: str) -> PFResult: ...
@dataclass
class PFError:
code: str
message: str
details: dict
```
From paperforge/core/errors.py (after Plan 57-01):
```python
class ErrorCode(str, Enum):
PYTHON_NOT_FOUND = "PYTHON_NOT_FOUND"
VERSION_MISMATCH = "VERSION_MISMATCH"
...
```
Existing status --json output (paperforge/worker/status.py:973-998):
```python
data = {
"version": __import__("paperforge").__version__,
"vault": str(vault),
"system_dir": cfg["system_dir"],
"resources_dir": cfg["resources_dir"],
"exports": len(export_files),
"domains": len(config.get("domains", [])),
"total_papers": record_count,
"formal_notes": note_count,
"bases": base_count,
"ocr": {"total": ..., "pending": ..., "done": ..., "failed": ...},
"path_errors": ...,
"env_configured": ...,
"lifecycle_level_counts": ...,
"health_aggregate": ...,
"maturity_distribution": ...,
}
print(_json.dumps(data, indent=2, ensure_ascii=False))
```
Existing doctor command dispatch (paperforge/cli.py:449-453):
```python
if args.command == "doctor":
from paperforge.worker.status import run_doctor
return run_doctor(vault)
```
Existing run_doctor signature (worker/status.py:359):
```python
def run_doctor(vault: Path, verbose: bool = False) -> int:
```
</interfaces>
</context>
<tasks>
<task type="auto">
<name>Task 1: Wrap status --json output in PFResult envelope</name>
<files>
paperforge/worker/status.py
</files>
<action>
In paperforge/worker/status.py, modify the `run_status` function's JSON output block (around line 973):
1. Import PFResult at the top of the function (lazy import, inside the `if json_output:` block): `from paperforge.core.result import PFResult`
2. Build the same `data` dict as currently, preserving its exact shape (backward compatibility of the data payload).
3. Wrap in PFResult:
```python
result = PFResult(
ok=True,
command="status",
version=data["version"], # from the existing data dict
data=data,
)
print(result.to_json())
```
4. Remove the `import json as _json` at the function top if it becomes unused (check -- _json may still be used elsewhere in the function). If _json is only used for the print, replace with PFResult. Otherwise keep _json and suppress the warning.
5. Handle error case: if any unrecoverable error occurs during status computation (e.g., vault doesn't exist), wrap in PFResult with ok=False and appropriate ErrorCode. Use try/except around the existing function body, catching broad exceptions and wrapping in PFResult(ok=False, command="status", error=PFError(code=ErrorCode.INTERNAL_ERROR.value, message=str(e), details={"vault": str(vault)})).
IMPORTANT: Do NOT change the `data` dict structure. The existing flat dict is the backward-compatible payload. Only wrap it in PFResult.
</action>
<verify>
<automated>python -m paperforge status --json --vault C:\Users\Lin\AppData\Local\Temp\opencode 2>&1 || true</automated>
</verify>
<done>
status --json output is wrapped in PFResult: {"ok": true, "command": "status", "version": "1.4.17rc3", "data": {existing flat dict}, "error": null}
Errors produce PFResult with ok=false.
</done>
</task>
<task type="auto">
<name>Task 2: Add --json output to doctor command</name>
<files>
paperforge/cli.py
paperforge/worker/status.py
paperforge/commands/status.py
</files>
<action>
Two modifications:
A. Add --json flag to doctor subparser in paperforge/cli.py:
```python
doctor_p = sub.add_parser("doctor", help="Validate PaperForge setup and configuration")
doctor_p.add_argument("--json", action="store_true", dest="json_output", help="Output JSON")
```
B. Modify run_doctor function in paperforge/worker/status.py:
1. Add `json_output: bool = False` parameter to `def run_doctor(vault: Path, verbose: bool = False, json_output: bool = False) -> int:`
2. At the end of the function (after verdict is computed but before print statements), add:
```python
if json_output:
from paperforge.core.result import PFResult, PFError
from paperforge.core.errors import ErrorCode
from paperforge import __version__
if has_fail:
result = PFResult(
ok=False,
command="doctor",
version=__version__,
data={"checks": checks_data, "verdict": "FAIL"},
error=PFError(code=ErrorCode.INTERNAL_ERROR.value, message=f"{verdict}: {next_action}", details={"fail_count": fail_count, "categories": list(fail_categories)}),
)
else:
result = PFResult(
ok=True,
command="doctor",
version=__version__,
data={"checks": checks_data, "verdict": verdict, "verdict_message": next_action},
)
print(result.to_json())
return 1 if has_fail else 0
```
3. Build `checks_data` as a list of dicts: `[{"category": cat, "status": status, "message": msg, "fix": fix} for cat, status, msg, fix in checks]`
4. Compute `fail_count = sum(1 for _, s, _, _ in checks if s == "fail")` near the existing `has_fail` computation.
5. Keep the existing text output path unchanged (everything before the `if json_output:` block).
C. Update cli.py doctor dispatch to pass json_output:
```python
if args.command == "doctor":
from paperforge.worker.status import run_doctor
return run_doctor(vault, verbose=getattr(args, "verbose", False), json_output=getattr(args, "json_output", False))
```
</action>
<verify>
<automated>python -m paperforge doctor --json --vault C:\Users\Lin\AppData\Local\Temp\opencode 2>&1 || true</automated>
</verify>
<done>doctor --json produces PFResult with structured checks in data field. Text output path unchanged. CLI passes json_output to run_doctor.</done>
</task>
<task type="auto">
<name>Task 3: Update CLI contract tests for PFResult-shaped status and doctor</name>
<files>
tests/cli/test_json_contracts.py
</files>
<action>
Update tests/cli/test_json_contracts.py:
1. In TestStatusJson, add a new method `test_status_json_pfresult_shape` that verifies the output is wrapped in PFResult:
- Call cli_invoker(["status", "--json"])
- assert_valid_json(result.stdout)
- Parse into dict
- Assert top-level keys are {"ok", "command", "version", "data", "error"} using assert_json_shape
- Assert "ok" is True
- Assert "command" == "status"
- Assert "version" is a non-empty string
- Assert "data" is a dict with at least {"version", "vault"} keys
- Assert "error" is None
2. Add TestDoctorJson class (mirroring TestStatusJson pattern) with:
- REQUIRED_KEYS = {"ok", "command", "version", "data", "error"}
- test_doctor_json_pfresult_shape: verifies PFResult envelope
- test_doctor_json_data_has_checks: verifies data.checks is a list
- test_doctor_json_no_vault: doctor --json on nonexistent vault exits non-zero
3. Keep all existing tests (do NOT remove test_status_json_value_semantics etc. — they must continue to pass on the wrapped output, since the data field preserves the old shape)
Do NOT modify test_status_json_value_semantics and test_status_json_snapshot_regression — these tests read the PFResult.data sub-dict and must still pass.
</action>
<verify>
<automated>python -m pytest tests/cli/test_json_contracts.py -v --tb=short -k "pfresult or doctor" 2>&1 || true</automated>
</verify>
<done>New PFResult shape tests pass. Existing contract tests (old flat dict shape inside data field) continue to pass.</done>
</task>
</tasks>
<verification>
- paperforge status --json output starts with {"ok": true, "command": "status", ...}
- paperforge doctor --json output starts with {"ok": true/false, "command": "doctor", ...}
- All existing tests still pass (backward compatible data payload preserved)
- New PFResult shape tests pass
</verification>
<success_criteria>
- [ ] status --json wrapped in PFResult with backward-compatible data payload
- [ ] doctor --json produces PFResult with structured checks in data
- [ ] doctor subparser in cli.py has --json flag
- [ ] run_doctor accepts json_output parameter
- [ ] New PFResult contract tests pass alongside existing contract tests
</success_criteria>
<output>
After completion, create .planning/phases/057-contract-layer/057-02-SUMMARY.md
</output>

253
.planning/phases/057-contract-layer/057-03-PLAN.md Normal file
View file

@ -0,0 +1,253 @@
---
phase: 57-contract-layer
plan: 03
type: execute
wave: 2
depends_on:
- 57-01
files_modified:
- paperforge/commands/sync.py
- paperforge/commands/ocr.py
- paperforge/ocr_diagnostics.py
- paperforge/worker/sync.py
- tests/cli/test_json_contracts.py
autonomous: true
requirements:
- CTRT-05
- CTRT-06
must_haves:
truths:
- "paperforge sync --json returns PFResult with sync result data"
- "paperforge ocr --diagnose --json returns PFResult with diagnostic data"
- "Both outputs have valid {ok, command, version, data, error} PFResult shape"
artifacts:
- path: "paperforge/commands/sync.py"
provides: "--json flag on sync command with PFResult output"
contains: "PFResult"
- path: "paperforge/commands/ocr.py"
provides: "--json flag on ocr --diagnose with PFResult output"
contains: "json_output"
- path: "paperforge/ocr_diagnostics.py"
provides: "Optional enhanced ocr_doctor with json-friendly output"
- path: "tests/cli/test_json_contracts.py"
provides: "PFResult shape tests for sync --json and ocr --diagnose --json"
key_links:
- from: "paperforge/commands/sync.py"
to: "paperforge/core/result.py"
via: "import PFResult"
- from: "paperforge/commands/ocr.py"
to: "paperforge/core/result.py"
via: "import PFResult"
---
<objective>
Add --json flag to sync and ocr --diagnose commands, wrapping output in PFResult envelope (CTRT-05, CTRT-06). Sync reports operation status (selection-sync + index-refresh). OCR diagnose wraps existing diagnostic dict.
Purpose: All core CLI commands produce consistent PFResult JSON output.
Output: Modified sync.py, ocr.py with --json flags; contract tests
</objective>
<execution_context>
@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md
@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/057-contract-layer/057-CONTEXT.md
<interfaces>
paperforge/commands/sync.py run(args):
- Calls run_selection_sync(vault, verbose) and run_index_refresh(vault, verbose, rebuild_index)
- Both return int exit codes
- Current output: text logs via print/logger, no structured JSON
paperforge/commands/ocr.py run(args):
- Calls _diagnose(vault, live) for --diagnose
- _diagnose calls ocr_doctor(config, live) which returns dict with {level, passed, error, fix, raw_response}
- Current output: human-readable text via print, no --json flag
paperforge/ocr_diagnostics.py ocr_doctor(config, live):
- Returns dict: {"level": int, "passed": bool, "error": str, "fix": str, "raw_response": str (optional)}
- Already a structured result — ideal for PFResult data payload
</interfaces>
</context>
<tasks>
<task type="auto">
<name>Task 1: Add --json to sync command with PFResult output</name>
<files>
paperforge/commands/sync.py
</files>
<action>
In paperforge/commands/sync.py:
1. Add `--json` argument at the top of the module (after domain arg):
```python
# The --json flag is passed via args — sync command currently gets its args from cli.py dispatch.
```
Actually, the sync parser is in cli.py. Add to the sync subparser in paperforge/cli.py:
```python
p_sync.add_argument("--json", action="store_true", dest="json_output", help="Output JSON")
```
2. In sync.py `run()` function, after both selection_sync and index_refresh complete, add:
```python
if getattr(args, "json_output", False):
from paperforge.core.result import PFResult, PFError
from paperforge.core.errors import ErrorCode
from paperforge import __version__
data = {
"selection_sync": {"exit_code": selection_code} if selection_code is not None else None,
"index_refresh": {"exit_code": index_code} if index_code is not None else None,
"vault": str(vault),
}
if exit_code == 0:
result = PFResult(ok=True, command="sync", version=__version__, data=data)
else:
result = PFResult(
ok=False, command="sync", version=__version__, data=data,
error=PFError(code=ErrorCode.SYNC_FAILED.value, message="Sync completed with errors", details={"selection_code": selection_code, "index_code": index_code}),
)
print(result.to_json())
return exit_code
```
3. Track the individual exit codes: store `selection_code` and `index_code` variables instead of reusing `code`.
```python
selection_code = None
index_code = None
if not index_only:
run_selection_sync = _get_run_selection_sync()
selection_code = run_selection_sync(vault, verbose=getattr(args, "verbose", False))
if selection_code != 0:
exit_code = selection_code
if not selection_only:
run_index_refresh = _get_run_index_refresh()
index_code = run_index_refresh(vault, verbose=getattr(args, "verbose", False), rebuild_index=rebuild_index)
if index_code != 0 and exit_code == 0:
exit_code = index_code
```
4. IMPORTANT: The --json block must come AFTER both workers complete but BEFORE the `return exit_code`. The text output path (existing behavior) must remain unchanged.
Capture worker std output if possible, or just report exit codes. Do NOT change worker return types.
</action>
<verify>
<automated>python -m paperforge sync --json --vault C:\Users\Lin\AppData\Local\Temp\opencode 2>&1 || true</automated>
</verify>
<done>sync --json produces PFResult with selection_sync and index_refresh exit codes in data. Text output path unchanged.</done>
</task>
<task type="auto">
<name>Task 2: Add --json to ocr --diagnose with PFResult output</name>
<files>
paperforge/commands/ocr.py
paperforge/cli.py
</files>
<action>
Two modifications:
A. In paperforge/cli.py, add --json to the ocr parser:
```python
p_ocr.add_argument("--json", action="store_true", dest="json_output", help="Output JSON")
```
(The --json flag applies to both ocr run and ocr diagnose, but only diagnose uses it in this phase.)
B. In paperforge/commands/ocr.py, modify _diagnose function:
1. Add `json_output: bool = False` parameter
2. Inside _diagnose, after getting the `result` dict from ocr_doctor, add:
```python
if json_output:
from paperforge.core.result import PFResult, PFError
from paperforge.core.errors import ErrorCode
from paperforge import __version__
passed = result.get("passed", False)
if passed:
pf_result = PFResult(
ok=True, command="ocr-diagnose", version=__version__,
data={"level": result.get("level"), "passed": True, "message": result.get("message", "All checks passed")},
)
else:
pf_result = PFResult(
ok=False, command="ocr-diagnose", version=__version__,
data={"level": result.get("level"), "passed": False},
error=PFError(
code=ErrorCode.OCR_TOKEN_MISSING.value if result.get("level") == 1 else ErrorCode.INTERNAL_ERROR.value,
message=result.get("error", "Unknown failure"),
details={"fix": result.get("fix", ""), "raw_response": result.get("raw_response", "")},
),
)
print(pf_result.to_json())
return 0 if passed else 1
```
3. Keep the existing text output path unchanged (before the `if json_output:` block).
C. Update the call site in ocr.py `run()`:
```python
diagnose_only = getattr(args, "diagnose", False) or ocr_action == "doctor"
json_output = getattr(args, "json_output", False)
if diagnose_only:
return _diagnose(vault, live=live, json_output=json_output)
```
</action>
<verify>
<automated>python -m paperforge ocr --diagnose --json --vault C:\Users\Lin\AppData\Local\Temp\opencode 2>&1 || true</automated>
</verify>
<done>ocr --diagnose --json produces PFResult with diagnostic level/passed/message in data. Text path unchanged.</done>
</task>
<task type="auto">
<name>Task 3: Add CLI contract tests for sync --json and ocr --diagnose --json</name>
<files>
tests/cli/test_json_contracts.py
</files>
<action>
In tests/cli/test_json_contracts.py, add:
1. TestSyncJson class:
- REQUIRED_KEYS = {"ok", "command", "version", "data", "error"}
- test_sync_json_pfresult_shape: invokes sync --json on empty vault, verifies PFResult shape
- test_sync_json_data_has_keys: verifies data has selection_sync and index_refresh keys
- test_sync_json_no_vault: nonexistent vault exits non-zero
2. TestOcrDiagnoseJson class:
- REQUIRED_KEYS = {"ok", "command", "version", "data", "error"}
- test_ocr_diagnose_json_pfresult_shape: invokes ocr --diagnose --json, verifies PFResult shape
- test_ocr_diagnose_json_data: verifies data has level and passed keys
- test_ocr_diagnose_json_no_vault: nonexistent vault exits non-zero
Reuse the existing assert_json_shape and assert_valid_json helpers from test_contract_helpers.py.
Keep tests compact: each test class with 3 methods.
</action>
<verify>
<automated>python -m pytest tests/cli/test_json_contracts.py -v --tb=short -k "sync or ocr" 2>&1 || true</automated>
</verify>
<done>New PFResult contract tests for sync and ocr pass alongside existing tests.</done>
</task>
</tasks>
<verification>
- paperforge sync --json produces valid PFResult JSON
- paperforge ocr --diagnose --json produces valid PFResult JSON
- All existing sync/ocr text output paths unchanged
- All tests pass
</verification>
<success_criteria>
- [ ] sync --json added with PFResult output (data contains selection_sync and index_refresh codes)
- [ ] ocr --diagnose --json added with PFResult output (data contains level/passed/message)
- [ ] Both commands' text output paths preserved
- [ ] Contract tests pass for both commands
</success_criteria>
<output>
After completion, create .planning/phases/057-contract-layer/057-03-SUMMARY.md
</output>

408
.planning/phases/057-contract-layer/057-04-PLAN.md Normal file
View file

@ -0,0 +1,408 @@
---
phase: 57-contract-layer
plan: 04
type: execute
wave: 2
depends_on:
- 57-01
files_modified:
- paperforge/cli.py
- paperforge/commands/__init__.py
- paperforge/commands/dashboard.py
- paperforge/worker/asset_index.py
- paperforge/plugin/main.js
- tests/cli/test_json_contracts.py
autonomous: true
requirements:
- CTRT-07
- CTRT-08
must_haves:
truths:
- "paperforge dashboard --json returns PFResult with stats (papers, pdf_health, ocr_health, domain_counts)"
- "paperforge dashboard --json returns PFResult with permissions (can_sync, can_ocr, can_copy_context)"
- "Plugin reads dashboard data via paperforge dashboard --json CLI contract"
- "Plugin fallback to direct formal-library.json reading still works"
artifacts:
- path: "paperforge/commands/dashboard.py"
provides: "Dashboard CLI command with --json output"
exports: ["run"]
- path: "paperforge/commands/__init__.py"
provides: "dashboard entry in command registry"
- path: "paperforge/plugin/main.js"
provides: "Updated plugin _fetchStats to read dashboard --json"
key_links:
- from: "paperforge/commands/dashboard.py"
to: "paperforge/core/result.py"
via: "PFResult wrapping"
- from: "paperforge/commands/dashboard.py"
to: "paperforge/worker/asset_index.py"
via: "read_index and summarize_index"
- from: "paperforge/plugin/main.js"
to: "paperforge/commands/dashboard.py"
via: "execFile python -m paperforge dashboard --json"
---
<objective>
Implement `paperforge dashboard --json` command with stats (papers, pdf_health, ocr_health, domain_counts) and permissions (can_sync, can_ocr, can_copy_context). Update plugin to read dashboard data via CLI contract with fallback to direct index reading.
Purpose: Expose machine-readable dashboard data in PFResult format for the Obsidian plugin and other consumers.
Output: New commands/dashboard.py module, CLI registration, plugin _fetchStats update, contract tests
</objective>
<execution_context>
@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md
@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/057-contract-layer/057-CONTEXT.md
<interfaces>
Per CONTEXT.md Dashboard Contract:
- Stats: papers (total count), pdf_health (healthy/broken/missing), ocr_health (pending/done/failed), domain_counts (dict of domain -> count)
- Permissions: can_sync (BBT exports exist), can_ocr (PaddleOCR configured), can_copy_context (system_dir writable)
- Version: from PFResult envelope (not duplicated in data)
Existing plugin _fetchStats (main.js:338-430):
- Reads formal-library.json directly
- Falls back to `python -m paperforge status --json`
- Computes stats from index items (lifecycle, health, OCR counts)
- Needs to be updated to try dashboard --json first, then fallback
Existing paperforge/worker/asset_index.py provides:
- read_index(vault) -> dict with {"items": [...], "paperforge_version": "..."}
- summarize_index(vault) -> dict with paper_count, lifecycle_level_counts, health_aggregate, maturity_distribution
Existing config checks (from paperforge/worker/status.py for permissions):
- BBT exports: paths["exports"].glob("*.json")
- OCR token: os.environ.get("PADDLEOCR_API_TOKEN")
- System dir writable: os.access(paths["system"], os.W_OK)
</interfaces>
</context>
<tasks>
<task type="auto">
<name>Task 1: Create dashboard command with --json PFResult output</name>
<files>
paperforge/commands/dashboard.py
paperforge/commands/__init__.py
paperforge/cli.py
</files>
<action>
Step A: Create paperforge/commands/dashboard.py:
```python
"""Dashboard command -- expose stats and permissions for the plugin UI."""
from __future__ import annotations
import argparse
import os
import logging
from pathlib import Path
logger = logging.getLogger(__name__)
def _get_index_summary(vault: Path) -> dict | None:
"""Read canonical index summary. Returns None if unavailable."""
try:
from paperforge.worker.asset_index import summarize_index, read_index
summary = summarize_index(vault)
index_data = read_index(vault)
domain_counts: dict[str, int] = {}
if index_data and "items" in index_data:
for item in index_data["items"]:
domain = item.get("domain", "unknown")
domain_counts[domain] = domain_counts.get(domain, 0) + 1
if summary:
summary["domain_counts"] = domain_counts
return summary
except Exception:
return None
def _check_permissions(vault: Path, cfg: dict) -> dict:
"""Check system permissions for dashboard."""
from paperforge.config import paperforge_paths
paths = paperforge_paths(vault, cfg)
exports_dir = paths.get("exports", vault / cfg["system_dir"] / "PaperForge" / "exports")
system_dir = paths.get("system", vault / cfg["system_dir"])
bbt_exports = sorted(exports_dir.glob("*.json")) if exports_dir.exists() else []
can_sync = len(bbt_exports) > 0
ocr_token = os.environ.get("PADDLEOCR_API_TOKEN") or os.environ.get("PADDLEOCR_API_KEY") or ""
can_ocr = bool(ocr_token.strip())
can_copy_context = os.access(str(system_dir), os.W_OK) if system_dir.exists() else False
return {
"can_sync": can_sync,
"can_ocr": can_ocr,
"can_copy_context": can_copy_context,
}
def run(args: argparse.Namespace) -> int:
"""Run dashboard command."""
vault = getattr(args, "vault_path", None)
if vault is None:
from paperforge.config import resolve_vault
vault = resolve_vault(cli_vault=getattr(args, "vault", None))
cfg = getattr(args, "cfg", None)
if cfg is None:
from paperforge.config import load_vault_config
cfg = load_vault_config(vault)
from paperforge.config import paperforge_paths
paths = paperforge_paths(vault, cfg)
json_output = getattr(args, "json_output", False)
# Build dashboard data
summary = _get_index_summary(vault)
permissions = _check_permissions(vault, cfg)
# Count papers from index or fallback to literature dir
papers = summary.get("paper_count", 0) if summary else 0
domain_counts = summary.get("domain_counts", {}) if summary else {}
# PDF health from health_aggregate
health = summary.get("health_aggregate", {}) if summary else {}
pdf_health = health.get("pdf_health", {"healthy": 0, "unhealthy": 0})
ocr_health_agg = health.get("ocr_health", {"healthy": 0, "unhealthy": 0})
# OCR breakdown from direct scan
ocr_total = 0
ocr_pending = 0
ocr_done = 0
ocr_failed = 0
if vault and paths.get("literature"):
import re
do_ocr_keys: set[str] = set()
for note_path in paths["literature"].rglob("*.md"):
if note_path.name in ("fulltext.md", "deep-reading.md", "discussion.md"):
continue
try:
text = note_path.read_text(encoding="utf-8")
if re.search(r"^do_ocr:\s*true\s*$", text, re.MULTILINE):
m = re.search(r"^zotero_key:\s*(\S+)", text, re.MULTILINE)
if m:
do_ocr_keys.add(m.group(1))
except Exception:
continue
for key in do_ocr_keys:
ocr_total += 1
meta_path = paths.get("ocr", vault / cfg["system_dir"] / "PaperForge" / "ocr") / key / "meta.json"
if not meta_path.exists():
ocr_pending += 1
continue
try:
from paperforge.worker._utils import read_json
meta = read_json(meta_path)
status = str(meta.get("ocr_status", "")).strip().lower()
if status == "done":
ocr_done += 1
else:
ocr_failed += 1
except Exception:
ocr_failed += 1
data = {
"papers": papers,
"pdf_health": {
"healthy": pdf_health.get("healthy", 0),
"broken": pdf_health.get("unhealthy", 0),
"missing": 0,
},
"ocr_health": {
"total": ocr_total,
"pending": ocr_pending,
"done": ocr_done,
"failed": ocr_failed,
},
"domain_counts": domain_counts,
"permissions": permissions,
}
if json_output:
from paperforge.core.result import PFResult
from paperforge import __version__
result = PFResult(ok=True, command="dashboard", version=__version__, data=data)
print(result.to_json())
else:
print(f"Papers: {papers}")
print(f"PDF Health: {data['pdf_health']['healthy']} healthy, {data['pdf_health']['broken']} broken, {data['pdf_health']['missing']} missing")
print(f"OCR Health: {ocr_done}/{ocr_total} done (pending: {ocr_pending}, failed: {ocr_failed})")
print(f"Domains: {len(domain_counts)}")
print(f"Permissions: sync={'Yes' if permissions['can_sync'] else 'No'}, ocr={'Yes' if permissions['can_ocr'] else 'No'}, copy_context={'Yes' if permissions['can_copy_context'] else 'No'}")
return 0
```
Step B: Register in paperforge/commands/__init__.py:
```python
_COMMAND_REGISTRY: dict[str, str] = {
...
"dashboard": "paperforge.commands.dashboard",
}
```
Step C: Add subparser in paperforge/cli.py:
```python
p_dashboard = sub.add_parser("dashboard", help="Show dashboard with stats and permissions")
p_dashboard.add_argument("--json", action="store_true", dest="json_output", help="Output JSON")
```
Step D: Add dispatch in cli.py main():
```python
if args.command == "dashboard":
from paperforge.commands import dashboard
return dashboard.run(args)
```
Place this alongside the other command dispatch blocks (after the doctor block, before update).
Per user decision (CONTEXT.md): Dashboard contract has stats (papers, pdf_health as healthy/broken/missing, ocr_health as pending/done/failed, domain_counts) and permissions (can_sync, can_ocr, can_copy_context). Version from PFResult envelope only.
</action>
<verify>
<automated>python -m paperforge dashboard --json --vault C:\Users\Lin\AppData\Local\Temp\opencode 2>&1 || true</automated>
</verify>
<done>dashboard --json produces PFResult with stats (papers, pdf_health, ocr_health, domain_counts) and permissions (can_sync, can_ocr, can_copy_context). Text output also works.</done>
</task>
<task type="auto">
<name>Task 2: Update plugin to read dashboard via CLI contract</name>
<files>
paperforge/plugin/main.js
</files>
<action>
Modify the plugin's _fetchStats method (main.js:338-430) to support the PFResult dashboard contract:
1. Keep the existing direct index reading path (formal-library.json) as the PRIMARY path.
2. Add a new hybrid read strategy. After the existing try/catch for index reading fails (fallback path, line 404-428), add a SECOND fallback that calls `paperforge dashboard --json`:
```javascript
// In the catch block (line 404), after the existing status --json fallback:
// Before the existing status --json fallback, try dashboard --json first
```
Actually, modify the fallback order:
```
Primary: formal-library.json (existing)
Fallback 1: paperforge dashboard --json (new)
Fallback 2: paperforge status --json (existing)
```
The _fetchStats function currently looks like:
```
try:
read formal-library.json directly -> compute stats -> _renderStats(d)
catch (err):
fallback to status --json -> _renderStats(d)
```
Change to:
```
try:
read formal-library.json directly -> compute stats -> _renderStats(d)
catch (err):
try:
execFile python -m paperforge dashboard --json
d = JSON.parse(stdout)
// Check PFResult shape: use d.data if PFResult, or d directly if old format
const dashboardData = d.data || d;
this._cachedStats = {
version: d.version || d.version,
total_papers: dashboardData.papers || 0,
formal_notes: 0,
exports: 0,
bases: 0,
ocr: dashboardData.ocr_health || { total: 0, pending: 0, processing: 0, done: 0, failed: 0 },
path_errors: 0,
lifecycle_level_counts: {},
health_aggregate: {
pdf_health: { healthy: dashboardData.pdf_health?.healthy || 0, unhealthy: (dashboardData.pdf_health?.broken || 0) + (dashboardData.pdf_health?.missing || 0) },
ocr_health: { healthy: dashboardData.ocr_health?.done || 0, unhealthy: (dashboardData.ocr_health?.pending || 0) + (dashboardData.ocr_health?.failed || 0) },
note_health: { healthy: 0, unhealthy: 0 },
asset_health: { healthy: 0, unhealthy: 0 },
},
};
this._renderStats(this._cachedStats);
this._renderOcr(this._cachedStats);
catch (err2):
// Keep existing status --json fallback
execFile python -m paperforge status --json ...
```
The key conversion map:
- dashboard.papers -> total_papers
- dashboard.pdf_health.healthy + broken/missing -> health_aggregate.pdf_health
- dashboard.ocr_health -> ocr stats (for _renderOcr)
- dashboard.permissions -> store for potential UI use (not rendered yet, but available)
3. Ensure the _renderStats function (line 500) can handle both formats:
- PFResult shaped: {ok, command, version, data: {papers, pdf_health, ...}} -- extract data sub-object
- Old status format: {total_papers, version, ocr, ...} -- use directly
The current _renderStats reads d.total_papers, d.version, d.ocr, etc. After the dashboard data is mapped to _cachedStats with the old shape keys (total_papers, ocr, etc.), _renderStats works unchanged.
Per user decision: Plugin keeps fallback to direct index reading during PFResult transition -- removed after 2 stable release cycles.
</action>
<verify>
<automated>node -e "console.log('Plugin logic verification: see test file')"</automated>
</verify>
<done>Plugin _fetchStats falls back to dashboard --json when formal-library.json is unavailable. Old format reading still works. Status --json remains as final fallback.</done>
</task>
<task type="auto">
<name>Task 3: Add CLI contract tests for dashboard --json</name>
<files>
tests/cli/test_json_contracts.py
</files>
<action>
In tests/cli/test_json_contracts.py, add TestDashboardJson class:
- REQUIRED_KEYS = {"ok", "command", "version", "data", "error"}
- DATA_KEYS = {"papers", "pdf_health", "ocr_health", "domain_counts", "permissions"}
- PERM_KEYS = {"can_sync", "can_ocr", "can_copy_context"}
Tests:
1. test_dashboard_json_pfresult_shape: invokes dashboard --json, verifies PFResult envelope
2. test_dashboard_json_data_keys: verifies data contains all required stat keys
3. test_dashboard_json_permissions_shape: verifies permissions has can_sync/can_ocr/can_copy_context
4. test_dashboard_json_no_vault: nonexistent vault exits non-zero
Reuse existing assert_json_shape and assert_valid_json helpers.
</action>
<verify>
<automated>python -m pytest tests/cli/test_json_contracts.py -v --tb=short -k "dashboard" 2>&1 || true</automated>
</verify>
<done>Dashboard PFResult contract tests pass. All 4 dashboard test methods verify shape and content.</done>
</task>
</tasks>
<verification>
- python -m paperforge dashboard --json produces valid PFResult JSON
- python -m pytest tests/cli/test_json_contracts.py -k "dashboard" passes
- Plugin can parse dashboard --json output via execFile
- Text output path works: python -m paperforge dashboard
</verification>
<success_criteria>
- [ ] dashboard command registered in cli.py and commands/__init__.py
- [ ] dashboard --json returns PFResult with stats (papers, pdf_health, ocr_health, domain_counts)
- [ ] dashboard --json returns PFResult with permissions (can_sync, can_ocr, can_copy_context)
- [ ] Plugin fallback reads dashboard --json when formal-library.json is unavailable
- [ ] Plugin fallback to status --json remains as final option
- [ ] Contract tests pass for dashboard
</success_criteria>
<output>
After completion, create .planning/phases/057-contract-layer/057-04-SUMMARY.md
</output>

99
.planning/phases/057-contract-layer/057-CONTEXT.md Normal file
View file

@ -0,0 +1,99 @@
# Phase 57: Contract Layer - Context
**Gathered:** 2026-05-09
**Status:** Ready for planning
<domain>
## Phase Boundary
Stable JSON contracts (PFResult/PFError dataclasses, ErrorCode enum) that CLI commands produce and the plugin consumes — defining the machine-readable API surface consumed by SYNC, STAT, and SETP phases. Specifically:
1. `paperforge/core/result.py` — PFResult and PFError dataclasses with JSON round-trip serialization
2. `paperforge/core/errors.py` — ErrorCode enum centralizing all error codes
3. CLI commands (`status --json`, `doctor --json`, `sync --json`, `ocr --diagnose --json`) wrapped in PFResult format
4. `dashboard --json` contract with stats and permissions for plugin consumption
5. Plugin reads dashboard via CLI; fallback to index reading during transition
</domain>
<decisions>
## Implementation Decisions
### PFResult Contract Shape
- Top-level fields: `{ok, command, version, data, error}` following Google JSON Style Guide
- Version field: semver string from `__version__` (e.g., `"1.4.17rc3"`)
- Error shape: `{"code":"ERR_CODE","message":"Human text","details":{}}` — machine code + human message + optional payload
- `ok` field: `false` only when `error` is set; `true` when `data` has partial results + `warnings` sub-field
### ErrorCode Enum
- Naming convention: `SCREAMING_SNAKE_CASE` (Python enum standard)
- Initial codes: `PYTHON_NOT_FOUND`, `VERSION_MISMATCH`, `BBT_EXPORT_NOT_FOUND`, `OCR_TOKEN_MISSING`, `SYNC_FAILED`, `VALIDATION_ERROR`, `INTERNAL_ERROR`
- `ErrorCode.UNKNOWN` fallback catch-all required
- Validation errors: single `VALIDATION_ERROR` with `details.field` for field-level granularity
### Dashboard Contract
- Stats: `papers` (total count), `pdf_health` (healthy/broken/missing), `ocr_health` (pending/done/failed), `domain_counts` (dict of domain -> count)
- Permissions: `can_sync` (BBT exports exist), `can_ocr` (PaddleOCR configured), `can_copy_context` (system_dir writable)
- Version: from PFResult envelope (not duplicated in data)
- Transition: Fallback to direct `formal-library.json` reading allowed; removed after 2 release cycles of stable PFResult
### Implementation Structure
- PFResult/PFError in `paperforge/core/result.py`
- ErrorCode in `paperforge/core/errors.py`
- CLI migration: wrap existing JSON output in PFResult envelope first (data field retains old shape), then refactor per-command
- Testing: round-trip unit tests for PFResult serialization + CLI contract tests asserting shape
### the agent's Discretion
- Exact ErrorCode member names and descriptions (beyond the agreed categories)
- ErrorCode string values for serialization
- Dashboard field naming details in data payload
- CLI argument parsing for `--json` flag (follow existing patterns)
</decisions>
<code_context>
## Existing Code Insights
### Reusable Assets
- `paperforge/worker/status.py:973-998` — existing `--json` output as flat dict (wrapping target)
- `paperforge/commands/status.py` — thin CLI dispatch layer
- `paperforge/commands/ocr.py` — `_diagnose()` and `run()` patterns
- `paperforge/worker/sync.py` — sync worker (JSON output pattern TBD)
- `paperforge/worker/status.py:206` — existing dependency check list with `"import":"yaml"` entries
### Established Patterns
- CLI commands in `paperforge/commands/` with `run(args) -> int` signature
- Workers in `paperforge/worker/` with `run_status(vault, verbose, json_output)` pattern
- `json_output` flag passed to workers for format control
- `import json as _json` in worker code for JSON serialization
- Plugin reads from `formal-library.json` directly (transition away)
### Integration Points
- All `--json` output paths need to be wrapped in PFResult
- Plugin `main.js` reads dashboard data — needs PFResult-aware parsing
- `paperforge/commands/` dispatch layer — where JSON output formatting should be centralized
- `paperforge/core/` — new module location for contract types
- CLI tests at `tests/cli/` — contract verification tests
</code_context>
<specifics>
## Specific Ideas
### Prior Decision from REQUIREMENTS.md
- Dual JSON output during contract transition: old format + new PFResult side-by-side until 2 release cycles of stability
- Plugin keeps fallback to direct index reading during PFResult transition — removed after 2 stable release cycles
### Phase 56 Dependency
- Phase 57 depends on Phase 56 (Stop the Bleeding) — version sync must be stable before contract layer
- Phase 56 created `scripts/check_version_sync.py` — version is now read from `__version__` consistently
</specifics>
<deferred>
## Deferred Ideas
- ErrorCode auto-generation from field_registry.yaml (Phase 57 will) — deferred to Phase 59
- Full JSON schema generation for PFResult (e.g., JSON Schema for external consumers) — out of scope for this phase
- Plugin PFResult parsing utilities (separate phase: plugin contract consumption)
</deferred>