diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 25934f3d..f43a3a72 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -1,354 +1,98 @@ -# Roadmap: PaperForge - -**All milestones shipped up to v1.11.** -**Phase numbering:** Continuous. v1.10 ended at Phase 45. v1.11 ended at Phase 50. v2.0 starts at Phase 51. - ---- - -## Milestones - -- ✅ **v1.0 MVP** — Phases 1-5 (shipped 2026-04-23) -- ✅ **v1.1 Sandbox Onboarding** — Phases 6-8 (shipped 2026-04-24) -- ✅ **v1.2 Systematization & Cohesion** — Phases 9-10 (shipped 2026-04-24) -- ✅ **v1.3 Path Normalization & Architecture Hardening** — Phases 11-12 (shipped 2026-04-24) -- ✅ **v1.4 Code Health & UX Hardening** — Phases 13-19 (shipped 2026-04-28) -- ✅ **v1.5 Obsidian Plugin Setup Integration** — Phases 20-21 (shipped 2026-04-29) -- ✅ **v1.6 AI-Ready Literature Asset Foundation** — Phases 22-26 (shipped 2026-05-04) -- ✅ **v1.7 Context-Aware Dashboard** — Phases 27-30 (shipped 2026-05-04) -- ✅ **v1.8 AI Discussion & Deep-Reading Dashboard** — Phases 31-36 (shipped 2026-05-07) -- ✅ **v1.9 Frontmatter Rationalization & Library-Record Deprecation** — Phases 37-41 (shipped 2026-05-07) -- ✅ **v1.10 Dependency Cleanup** — Phases 42-45 (shipped 2026-05-07) -- ✅ **v1.11 Merge Gate — v1.9 Ripple Remediation** — Phases 46-50 (shipped 2026-05-07) -- 🚧 **v2.0 Testing Infrastructure — 6-Layer Quality Gates** — Phases 51-55 (planning) - -*Archive: `.planning/milestones/`* - ---- - -## Phases - -
-✅ v1.6 AI-Ready Literature Asset Foundation (Phases 22-26) — SHIPPED 2026-05-04 - -- [x] Phase 22: Configuration Truth & Compatibility (3/3) -- [x] Phase 23: Canonical Asset Index & Safe Rebuilds (3/3) -- [x] Phase 24: Derived Lifecycle, Health & Maturity (2/2) -- [x] Phase 25: Surface Convergence, Doctor & Repair (3/3) -- [x] Phase 26: Traceable AI Context Packs (3/3) - -
- -
-✅ v1.7 Context-Aware Dashboard (Phases 27-30) — SHIPPED 2026-05-04 - -- [x] Phase 27: Component Library (2/2) -- [x] Phase 28: Dashboard Shell & Context Detection (2/2) -- [x] Phase 29: Per-Paper View (1/1) -- [x] Phase 30: Collection View (1/1) - -
- -
-✅ v1.8 AI Discussion & Deep-Reading Dashboard (Phases 31-36) — SHIPPED 2026-05-07 - -- [x] Phase 31: Bug Fixes — Restore version display; remove meaningless "ai" UI row -- [x] Phase 32: Deep-Reading Mode Detection — Plugin routes deep-reading.md to dedicated dashboard mode -- [x] Phase 33: Deep-Reading Dashboard Rendering — Status bar, Pass 1 summary, empty-state AI Q&A card -- [x] Phase 34: Jump to Deep Reading Button — Per-paper dashboard card links to deep-reading.md (completed 2026-05-06) -- [x] Phase 35: AI Discussion Recorder — Python module writes discussion.md + discussion.json into ai/ (completed 2026-05-06) -- [x] Phase 36: Integration Verification — End-to-end pipeline verified with CJK encoding and vault.adapter.read - -
- -
-✅ v1.9 Frontmatter Rationalization & Library-Record Deprecation (Phases 37-41) — SHIPPED 2026-05-07 - -- [x] Phase 37: Frontmatter Rationalization (1 plan) -- [x] Phase 38: Workspace Stabilization (1 plan) -- [x] Phase 39: Base View Fix -- [x] Phase 40: Library-Record Deprecation -- [x] Phase 41: Plugin Dashboard Sync - -
- -
-✅ v1.10 Dependency Cleanup (Phases 42-45) — SHIPPED 2026-05-07 - -**Milestone Goal:** Fix all code breakage and documentation staleness caused by v1.9's library-records deprecation and directory default changes. - -- [x] **Phase 42: Core Pipeline Fix** — OCR, status, and sync workers read from formal notes, not library-records -- [x] **Phase 43: Repair & Directory Defaults** — Repair re-anchored; all hardcoded old directory defaults updated -- [x] **Phase 44: Documentation Update** — AGENTS.md, 5 skill files, and 3 docs reflect v1.9 structure -- [x] **Phase 45: Validation & Release Gate** — Tests pass; end-to-end OCR/status verification - -
- -### ✅ v1.11 Merge Gate — v1.9 Ripple Remediation (Shipped 2026-05-07) - -**Milestone Goal:** Resolve all 27 findings from the v1.6-ai-ready-asset-foundation branch review before merging to master. Fix cascading v1.9 structural ripple across four root cause clusters: index path hardcoding, library-records residual traces, setup wizard TUI removal, and new module hardening gaps. - -- [x] **Phase 46: Index Path Resolution** — 2 plans: config-resolved paths, env var/placeholder fixes (completed 2026-05-07) -- [x] **Phase 47: Library-Records Deprecation Cleanup** — Zero residual traces in production code and documentation (completed 2026-05-07) -- [x] **Phase 48: Textual TUI Removal** — 2 plans: TUI code removal, documentation updates (completed 2026-05-07) -- [x] **Phase 49: Module Hardening** — Production-grade safety guards in discussion.py, main.js, asset_state.py (completed 2026-05-07) -- [x] **Phase 50: Repair Blind Spots** — All 6 divergence types detected and handled by fix mode (completed 2026-05-07) - ---- - -### 🚧 v2.0 Testing Infrastructure — 6-Layer Quality Gates (Planning) - -**Milestone Goal:** Establish a multi-layer testing infrastructure covering version consistency (L0), Python unit tests (L1), CLI contracts (L2), plugin-backend integration (L3), temp vault E2E workflows (L4), user journey contracts (L5), and destructive scenarios (L6) — with CI matrix, golden datasets, and snapshot testing. - -- [ ] **Phase 51: Testing Foundation** — Fixture hierarchy, L0 version checker, L1 unit test relocation, PR check CI -- [ ] **Phase 52: Golden Datasets & CLI Contracts** — Fixture files, CLI `--json` contract tests with snapshot assertions -- [ ] **Phase 53: Plugin Tests & Temp Vault E2E** — Vitest plugin tests, temp vault E2E workflows, Node 20 CI -- [x] **Phase 54: User Journey & Chaos Tests** — UX contract, journey scripts, destructive scenario tests, chaos CI (completed 2026-05-08) -- [ ] **Phase 55: CI Optimization & Consistency Audit** — Plasma matrix, full gate, path-filtered triggers, mock validation audit - ---- - -## Phase Details - -### Phase 42: Core Pipeline Fix -**Goal**: OCR, status, and sync workers read workflow state (do_ocr, analyze, ocr_status) from formal note frontmatter — same logic as the existing `get_analyze_queue()` pattern. Core workflow unbroken for new papers created post-v1.9. -**Depends on**: Nothing (first v1.10 phase; v1.9 shipped) -**Requirements**: WF-01, WF-02, WF-03, WF-04, SYN-01, SYN-02, SYN-03 -**Success Criteria** (what must be TRUE): - 1. Running `paperforge ocr` finds and processes papers whose formal note frontmatter has `do_ocr: true` — no library-records reads - 2. `auto_analyze_after_ocr` writes `analyze: true` into the formal note frontmatter of the processed paper - 3. `paperforge status` reports paper counts and OCR status counts sourced from formal notes + canonical index, not from library-records - 4. `paperforge status` doctor checks (PDF path validation, wikilink format) sample from formal notes, not library-records - 5. `paperforge sync` no longer creates empty library-records domain directories; `load_control_actions()` scans formal note frontmatter instead of library-records directory - 6. Orphaned formal notes (no matching Zotero entry) are cleaned up from the Literature/ directory during sync -**Plans**: TBD - -### Phase 43: Repair & Directory Defaults -**Goal**: Repair worker three-way divergence scan and path error detection re-anchored from library-records to formal notes + canonical index. All 14 hardcoded old directory defaults (`99_System`, `03_Resources`, `05_Bases`) updated across production code, setup wizard, validation script, .gitignore, and CLI help text to match `DEFAULT_CONFIG` (`System`, `Resources`, `Bases`). -**Depends on**: Phase 42 (formal note path model confirmed; repair needs the same scan pattern) -**Requirements**: REP-01, REP-02, REP-03, DEF-01, DEF-02, DEF-03, DEF-04, DEF-05, DEF-06, DEF-07 -**Success Criteria** (what must be TRUE): - 1. `paperforge repair` three-way divergence scan reads from formal note frontmatter, canonical index, and paper-meta.json — zero library-records reads - 2. `paperforge repair --fix-paths` detects path errors by scanning formal notes and writes fixes into formal note frontmatter - 3. All `cfg.get("system_dir", "99_System")` fallbacks in asset_index, sync, and repair use `"System"` instead - 4. setup_wizard function signature defaults, validate_setup.py legacy fallbacks, and .gitignore patterns use clean directory names (`System`/`Resources`/`Bases`) - 5. CLI `--help` text displays clean default directory names; setup_wizard no longer creates an empty control_dir -**Plans**: TBD - -### Phase 44: Documentation Update -**Goal**: All user-facing and agent-facing documentation reflects the v1.9 simplified structure. Zero references to the deprecated library-records workflow remain in AGENTS.md, 5 skill files, or 3 docs files. -**Depends on**: Phase 43 (all code changes finalized; documentation describes the settled behavior) -**Requirements**: DOC-01, DOC-02, DOC-03, DOC-04, DOC-05, DOC-06, DOC-07, DOC-08, DOC-09 -**Success Criteria** (what must be TRUE): - 1. AGENTS.md contains zero references to library-records; frontmatter section shows only formal note fields - 2. All 5 skill files (pf-sync, pf-ocr, pf-status, pf-paper, pf-deep) describe the formal-note-only workflow with no mention of library-records - 3. docs/setup-guide.md directory structure diagram shows Literature/ workspace directories without library-records - 4. docs/ARCHITECTURE.md data flow reflects the v1.9 simplified tracking layer (formal notes + canonical index, no library-record intermediate) - 5. docs/COMMANDS.md sync description shows direct formal note generation without a two-phase library-record step -**Plans**: TBD - -### Phase 45: Validation & Release Gate -**Goal**: All existing tests pass with zero regressions. End-to-end verification confirms OCR and status workers operate correctly on the new formal-note-based paths. Release gate met. -**Depends on**: Phase 44 (documentation may need test validation too; all code changes complete) -**Requirements**: VAL-01, VAL-02, VAL-03 -**Success Criteria** (what must be TRUE): - 1. Full test suite passes with zero failures (no silent regressions in OCR, repair, sync, or status workflows) - 2. End-to-end: `paperforge ocr` correctly finds and processes a paper whose formal note has `do_ocr: true` - 3. `paperforge status` output contains zero references to library-records (no `library_records: 0` or equivalent) -**Plans**: TBD - -### Phase 46: Index Path Resolution -**Goal**: All 5 workspace-path fields in the canonical index (`paper_root`, `main_note_path`, `fulltext_path`, `deep_reading_path`, `ai_path`) use config-resolved `literature_dir` instead of hardcoded `"Literature/"`. All 11 downstream consumers resolve correct paths. Config env var typo and migration gaps fixed. -**Depends on**: Nothing (first v1.11 phase; v1.10 shipped) -**Requirements**: PATH-01, PATH-02, PATH-03, PATH-04, PATH-05, PATH-06 -**Success Criteria** (what must be TRUE): - 1. `paperforge sync` generates canonical index entries with `paper_root`, `main_note_path`, `fulltext_path`, `deep_reading_path`, and `ai_path` using the user's configured `literature_dir` — verified via `paperforge context --json` showing correct paths - 2. Plugin dashboard renders per-paper views using config-resolved paths from the index (not hardcoded `"Literature/"`) — verified by opening a paper dashboard after sync with a non-default literature_dir - 3. Environment variable `PAPERFORGE_LITERATURE_DIR` correctly overrides `literature_dir` (no truncation to `PAPERFORGERATURE_DIR`) — verified via `paperforge paths --json` - 4. Legacy `paperforge.json` with top-level `skill_dir` and `command_dir` settings migrates into `vault_config` on first sync — no orphaned top-level keys remain - 5. Shipping `.base` templates contain zero `${LIBRARY_RECORDS}` placeholders — verified by inspecting generated Base files in a fresh vault -**Plans**: TBD -**Plans**: 2 plans - -Plans: -- [x] 46-001-PLAN.md — Core path resolution: fix 5 hardcoded "Literature/" in asset_index.py, fix config.py (env var typo, library_records path, CONFIG_PATH_KEYS), fix test_config.py -- [x] 46-002-PLAN.md — Placeholder & Windows path cleanup: remove LIBRARY_RECORDS substitution in base_views.py, remove unnecessary backslash replace in discussion.py -**UI hint**: yes - -### Phase 47: Library-Records Deprecation Cleanup -**Goal**: Zero library-records references remain in production code (status.py, sync.py, ld_deep.py), documentation (5 command skill files), or user-facing labels. Dead code removed, stale scan paths corrected, post-install instructions updated to single-command workflow. -**Depends on**: Phase 46 (sync.py docstring changes reference same config-resolved paths) -**Requirements**: LEGACY-01, LEGACY-02, LEGACY-03, LEGACY-04, LEGACY-05, LEGACY-06, LEGACY-07 -**Success Criteria** (what must be TRUE): - 1. `paperforge status` reports `formal_notes` count (label is `formal_notes`, not `library_records`) — output reflects post-v1.9 reality - 2. Five command skill files (`pf-sync.md`, `pf-ocr.md`, `pf-status.md`, `pf-paper.md`, `pf-deep.md`) contain zero mentions of "library-records" — verified by `grep -r "library.record"` returning no hits in skill files - 3. `paperforge sync` no longer constructs `record_path` or calls `parse_existing_library_record()` — dead code removed, sync completes without errors - 4. Setup wizard post-install instructions describe a single `paperforge sync` workflow (not old `--selection`/`--index` two-phase flow) - 5. `paperforge repair` docstring reads "Scan formal literature notes" (not "library-records") and `ld_deep.py` return dict contains only active path keys -**Plans**: 2 plans - -Plans: -- [x] 47-001-PLAN.md — Python source cleanup: status.py label/scan path, sync.py dead code + docstrings, ld_deep.py records key, repair.py + discussion.py docstrings (LEGACY-01, 02, 03, 04, 07) -- [x] 47-002-PLAN.md — Documentation cleanup: setup_wizard.py post-install text, 10 command file copies in command/ + paperforge/command_files/ (LEGACY-05, 06) - -### Phase 48: Textual TUI Removal -**Goal**: The broken Textual TUI setup wizard is removed entirely. `paperforge setup` (bare, no `--headless`) prints a help message redirecting users to `--headless` or the plugin settings tab. All TUI classes, import paths, and the `textual` optional dependency are purged. Documentation updated to reflect headless-only setup. `headless_setup()` and all shared utilities preserved intact. -**Depends on**: Nothing (standalone removal; no dependency on PATH or LEGACY phases) -**Requirements**: DEPR-01, DEPR-02, DEPR-03 -**Success Criteria** (what must be TRUE): - 1. Running `paperforge setup` (bare, without `--headless`) prints a clean help message redirecting to `paperforge setup --headless` or the Obsidian plugin settings tab — no `NameError` crash, no TUI launch attempt - 2. `setup_wizard.py` contains zero Textual-related imports or classes — `WelcomeStep`, `DirOverviewStep`, `VaultStep`, `PlatformStep`, `DeployStep`, `DoneStep`, `SetupWizardApp`, `ContentSwitcher`, `StepScreen`, and all `from textual` import paths removed; verified by `rg "from textual" setup_wizard.py` returning no hits - 3. All three documentation files (`docs/setup-guide.md`, `docs/INSTALLATION.md`, `README.md`) reference only `paperforge setup --headless` — no bare `paperforge setup` without `--headless` flag - 4. Post-install instruction text and headless completion message describe headless-only workflow; `--non-interactive` CLI option removed; `textual` removed from project optional dependencies - 5. `headless_setup()`, shared utilities (`EnvChecker`, `AGENT_CONFIGS`, `_copy_file_incremental`, `_merge_env_incremental`) preserved and fully functional — zero behavior change for the headless code path -**Plans**: 2 plans -- [x] `48-001-PLAN.md` — TUI code removal (DEPR-01, DEPR-03): remove textual imports/classes from setup_wizard.py, replace main() with help message, update cli.py help text, remove textual from pyproject.toml -- [x] `48-002-PLAN.md` — Documentation updates (DEPR-02): update setup-guide.md and INSTALLATION.md for headless-only workflow - -### Phase 49: Module Hardening -**Goal**: New modules built during v1.6-v1.8 (discussion.py, asset_state.py, main.js) have production-grade safety guards: file locking prevents concurrent write corruption, markdown special characters are escaped, timestamps use UTC, API keys pass via environment not CLI args, DOM rendering avoids XSS vectors, and empty-state outputs are safe JSON. -**Depends on**: Phase 47 (discussion.py docstring fixes share file context with LEGACY-07) -**Requirements**: HARDEN-01, HARDEN-02, HARDEN-03, HARDEN-04, HARDEN-05, HARDEN-06, HARDEN-07 -**Success Criteria** (what must be TRUE): - 1. Two concurrent `/pf-paper` calls for the same paper do not corrupt `discussion.json` or `discussion.md` — file locking prevents interleaved write operations - 2. Markdown special characters (`*`, `#`, `[`, `_`, `` ` ``) in QA question/answer fields are escaped before writing to `discussion.md` — no broken formatting when rendered in Obsidian - 3. All timestamps in `discussion.json` use UTC (`datetime.now(timezone.utc)`) — no CST/UTC+8 hardcoded offset; verified by inspecting a newly created discussion session timestamp - 4. Obsidian plugin spawns OCR subprocess with `PADDLEOCR_API_TOKEN` in environment variable (not command-line argument) — API key not visible in process list via Task Manager - 5. Plugin directory tree renders via `createEl()` DOM API (not `innerHTML` assignment) — no XSS vector from user-configured directory names containing HTML/script tags - 6. `paperforge status --json` returns `lifecycle_level_counts`, `health_aggregate`, and `maturity_distribution` as empty dicts `{}` (not `null`) when no canonical index exists — downstream JSON parsers do not crash on field access -**Plans**: 3 plans - -Plans: -- [x] `49-001-PLAN.md` — discussion.py hardening: UTC timestamps (HARDEN-03), markdown escaping (HARDEN-02), file locking (HARDEN-01) -- [x] `49-002-PLAN.md` — main.js hardening: API key via env var (HARDEN-04), createEl() not innerHTML (HARDEN-05) -- [x] `49-003-PLAN.md` — asset_state.py + status.py hardening: reorder next_step checks (HARDEN-06), empty dicts not null (HARDEN-07) -**UI hint**: yes - -### Phase 50: Repair Blind Spots -**Goal**: Repair worker three-way divergence detection covers all 6 divergence types (was missing the `ocr_status: pending` vs `meta done/failed` case). `--fix` mode handles every detected condition or produces explicit warnings for unhandled types. Silent exception swallowing replaced with logged warnings. Dead code removed. -**Depends on**: Phase 49 (repair.py shares logging patterns with discussion.py hardening) -**Requirements**: REPAIR-01, REPAIR-02, REPAIR-03, REPAIR-04 -**Success Criteria** (what must be TRUE): - 1. `paperforge repair` detects condition 4 divergence: `ocr_status: pending` in formal note frontmatter vs `done`/`failed` in `meta.json` — output includes these findings (previously silently skipped) - 2. `paperforge repair --fix` handles all 6 detected divergence types — no silently skipped conditions; any unhandled type produces an explicit `[WARNING]` line in console output - 3. `paperforge repair --fix` logs (rather than silently ignores) index write failures during fix operations — `logger.warning()` calls replace bare `except Exception: pass` blocks - 4. Dead `load_domain_config` call and unused dict comprehension removed from `repair.py:196` — no unreachable code or unused imports -**Plans**: 1 plan - -Plans: -- [x] 50-001-PLAN.md — All 4 REPAIR fixes: dead code removal, condition 4 detection, --fix mode coverage, silent exception logging - -### Phase 51: Testing Foundation -**Goal**: Establish the testing framework — version consistency checker (L0), existing unit test relocation to `tests/unit/` (L1), 5-level hierarchical pytest fixtures, and PR check CI pipeline. -**Depends on**: Nothing (first v2.0 phase; v1.11 shipped) -**Requirements**: VC-01, VC-02, UNIT-01, UNIT-02, UNIT-03, UNIT-04, UNIT-05, UNIT-06, UNIT-07, UNIT-08, CI-01 -**Success Criteria** (what must be TRUE): - 1. `scripts/check_version_sync.py` validates all 6+ version declarations (`__init__.__version__`, manifest.json, versions.json, CHANGELOG) and fails on mismatch — CI gate blocks push on version drift - 2. All existing 473+ tests pass under `tests/unit/` directory structure with zero behavior modifications — `pytest tests/unit/` succeeds - 3. `tests/conftest.py` provides 5-level fixture hierarchy (`empty_vault` -> `config_vault` -> `vault_with_export` -> `vault_with_ocr` -> `full_test_vault`) — each level usable independently by downstream test layers - 4. `ci-pr-checks.yml` runs L0 (version check on ubuntu) + L1 (unit tests on 3 OS x 3 Python matrix) with total wall-clock under 2 minutes - 5. `pyproject.toml` updated with test markers (`unit`, `cli`, `e2e`, `journey`, `chaos`, `slow`), testpaths, and new dependencies (pytest-snapshot, pytest-timeout, pytest-mock, responses, coverage) -**Plans**: TBD - -### Phase 52: Golden Datasets & CLI Contracts -**Goal**: Build the shared `fixtures/` golden dataset (Zotero JSON, PDF samples, mock OCR responses, expected snapshots) and CLI contract tests (L2) with subprocess invoker and shape-specific snapshot assertions. -**Depends on**: Phase 51 (fixture hierarchy conftest provides base structure; test runner configured) -**Requirements**: FIX-01, FIX-02, FIX-03, FIX-04, FIX-05, CLI-01, CLI-02, CLI-03 -**Success Criteria** (what must be TRUE): - 1. `fixtures/` directory contains 8+ Zotero JSON variants (valid, empty, malformed, missing keys, CJK content, multi-attachment, 3 path formats), 4 minimal valid PDFs (including CJK filenames), 5 mock OCR response fixtures (submit, poll, result, error, timeout), and expected output snapshots — all tracked in `MANIFEST.json` with `used_by`, `generated`, `desc` fields - 2. All 7 CLI commands (`status`, `sync`, `ocr`, `doctor`, `repair`, `context`, `setup`) return stable `--json` output with consistent schema — error responses use `ok`, `error_code`, `message`, `details`, `suggestions` fields - 3. `pytest-snapshot` tests pass with shape-specific assertions (normalized dynamic fields, subset matching) — snapshot updates require explicit `--snapshot-update` flag and deliberate commit - 4. Mock OCR backend using `responses` library produces deterministic, replayable PaddleOCR responses for all API states — no external HTTP calls during test execution -**Plans**: 2 plans - -Plans: -- [ ] 52-001-PLAN.md — Golden Datasets: Zotero JSON variants (10), PDF fixtures (4 generated), mock OCR responses (6+), expected snapshots, MANIFEST.json, vault_builder.py (FIX-01, FIX-02, FIX-03, FIX-04, FIX-05) -- [ ] 52-002-PLAN.md — CLI Contract Tests: conftest with cli_invoker + mock_ocr_backend, pytest-snapshot integration, contract tests for all 7 commands + error codes (CLI-01, CLI-02, CLI-03) - -### Phase 53: Plugin Tests & Temp Vault E2E -**Goal**: Build plugin-backend integration tests (L3) with Vitest + obsidian-test-mocks, and full temp vault end-to-end tests (L4) covering sync, OCR, status, doctor, and repair workflows. -**Depends on**: Phase 52 (CLI contract outputs define the interface L3 plugin tests validate against; golden datasets provide E2E input data) -**Requirements**: PLUG-01, PLUG-02, PLUG-03, E2E-01, E2E-02, E2E-03, E2E-04, E2E-05, CI-04 -**Success Criteria** (what must be TRUE): - 1. `tests/plugin/` runs on Vitest + obsidian-test-mocks + jsdom — `resolvePythonExecutable`, `getPluginVersion`, and `checkRuntimeVersion` have passing tests - 2. Plugin error classification covers all 5 error patterns (Python missing, import failed, version mismatch, pip install failure, timeout) — `buildRuntimeInstallCommand` and `parseRuntimeStatus` dispatch tests pass - 3. Temp vault fixture (`tmp_path`-based) produces a disposable Vault with config, directories, mock Zotero data, and mock OCR state — usable by all E2E tests - 4. Full E2E pipeline test: BBT JSON -> formal notes -> canonical index -> Base views completes in temp vault without external dependencies - 5. OCR E2E test: mock PaddleOCR backend via `responses` HTTP interception processes `do_ocr: true` paper through pending -> processing -> done states - 6. Multi-domain sync test verifies multiple Zotero collections sync correctly, producing domain-separated formal notes and index entries - 7. Node 20 CI runner executes all plugin Vitest tests in `ci.yml` — L3 gate passes on PR to main -**Plans**: 2 plans - -Plans: -- [ ] 53-001-PLAN.md — Plugin source extraction & Vitest tests (L3): extract src/runtime.js, src/errors.js, src/commands.js; set up Vitest + obsidian-test-mocks + jsdom; write & pass plugin tests; add Node 20 CI runner -- [ ] 53-002-PLAN.md — Temp vault E2E tests (L4): E2E conftest with temp vault fixture; sync pipeline, multi-domain sync, OCR mock E2E, status/doctor/repair E2E tests -**UI hint**: yes - -### Phase 54: User Journey & Chaos Tests -**Goal**: Document and implement user journey tests (L5) against verifiable UX contracts, plus destructive/abnormal scenario tests (L6) with safety contracts, Docker isolation, and weekly CI schedule. -**Depends on**: Phase 53 (E2E vault infrastructure reused by journey tests; mock systems shared with chaos tests) -**Requirements**: JNY-01, JNY-02, JNY-03, CHAOS-01, CHAOS-02, CHAOS-03, CHAOS-04, CI-05 -**Success Criteria** (what must be TRUE): - 1. `docs/ux-contract.md` defines concrete, verifiable step sequences for installation, sync, OCR, and dashboard workflows — each step has a single measurable outcome - 2. New user onboarding journey test (`install -> sync -> OCR -> analyze -> deep-read`) completes in temp vault with journey fixture pack at each stage - 3. Daily workflow journey test (`existing user adds paper -> syncs -> OCRs -> reads`) completes in pre-configured temp vault with existing papers - 4. `CHAOS_MATRIX.md` documents all destructive scenarios with triggers, expected behavior, and safety contracts — no undocumented failure modes - 5. Corrupted input tests (malformed JSON, corrupt PDF, broken meta.json, missing frontmatter) produce graceful error messages — no unhandled crashes - 6. Network failure tests (OCR API timeout, HTTP 401, 500, DNS unreachable) use mock backend and produce actionable error messages - 7. Filesystem error tests (permission denied, locked files, missing directories) use isolation assertion (`assert "tmp" in str(vault)`) — no real vault damage - 8. `ci-chaos.yml` runs on weekly schedule + manual trigger with Docker isolation — chaos tests excluded from regular CI gate -**Plans**: 3 plans - -Plans: -- [x] 54-001-PLAN.md — UX Contract + Journey Tests: docs/ux-contract.md, journey fixture pack, onboarding + daily workflow tests (JNY-01, JNY-02, JNY-03) -- [x] 54-002-PLAN.md — Chaos Matrix + Chaos Tests: CHAOS_MATRIX.md, corrupted input, network failure, filesystem error tests with isolation guards (CHAOS-01, CHAOS-02, CHAOS-03, CHAOS-04) -- [x] 54-003-PLAN.md — CI Chaos Workflow: ci-chaos.yml with weekly schedule + manual trigger (CI-05) - -### Phase 55: CI Optimization & Consistency Audit -**Goal**: Harden CI with plasma matrix strategy, full L0-L4 merge gate, path-filtered triggers, and cross-layer consistency audit that validates L1 mocks against L4 ground truth. -**Depends on**: Phase 54 (all test layers L0-L6 exist in CI; optimization decisions informed by actual run data) -**Requirements**: CI-02, CI-03 -**Success Criteria** (what must be TRUE): - 1. `ci.yml` full gate runs L0 through L4 on merge to main — `re-actors/alls-green` provides single-status check for branch protection - 2. Plasma CI matrix: L1 on 3 OS x 3 Python (fast); L2 on 2 Python x 1 OS; L3-L5 on single config — total CI budget under configured concurrent runner limit - 3. Path-filtered CI triggers prevent unnecessary jobs: changes to `paperforge/ocr.py` trigger L1+L2+L4; changes to `paperforge/plugin/main.js` trigger L3 only; version files trigger L0 only - 4. Consistency audit test validates L1 mock expectations against L4 real pipeline output — `pytest tests/audit/` detects mock drift before it reaches production -**Plans**: 2 plans - -Plans: -- [ ] 55-001-PLAN.md — Consistency Audit Tests: cross-layer consistency validation detecting L1 mock drift against L4 golden dataset ground truth -- [ ] 55-002-PLAN.md — Plasma Matrix CI Pipeline: ci.yml rewrite with L0-L5 merge gate, plasma matrix, path-filtered triggers, alls-green aggregator - ---- - -## Progress - -| Phase | Milestone | Plans Complete | Status | Completed | -|-------|-----------|----------------|--------|-----------| -| 31. Bug Fixes | v1.8 | — | Complete | 2026-05-07 | -| 32. Deep-Reading Mode Detection | v1.8 | — | Complete | 2026-05-07 | -| 33. Deep-Reading Dashboard Rendering | v1.8 | — | Complete | 2026-05-07 | -| 34. Jump to Deep Reading Button | v1.8 | 1/1 | Complete | 2026-05-06 | -| 35. AI Discussion Recorder | v1.8 | 1/1 | Complete | 2026-05-06 | -| 36. Integration Verification | v1.8 | — | Complete | 2026-05-07 | -| 37. Frontmatter Rationalization | v1.9 | 1/1 | Complete | 2026-05-07 | -| 38. Workspace Stabilization | v1.9 | 1/1 | Complete | 2026-05-07 | -| 39. Base View Fix | v1.9 | — | Complete | 2026-05-07 | -| 40. Library-Record Deprecation | v1.9 | — | Complete | 2026-05-07 | -| 41. Plugin Dashboard Sync | v1.9 | — | Complete | 2026-05-07 | -| 42. Core Pipeline Fix | v1.10 | — | Complete | 2026-05-07 | -| 43. Repair & Directory Defaults | v1.10 | — | Complete | 2026-05-07 | -| 44. Documentation Update | v1.10 | — | Complete | 2026-05-07 | -| 45. Validation & Release Gate | v1.10 | — | Complete | 2026-05-07 | -| 46. Index Path Resolution | v1.11 | 2/2 | Complete | 2026-05-07 | -| 47. Library-Records Deprecation Cleanup | v1.11 | 2/2 | Complete | 2026-05-07 | -| 48. Textual TUI Removal | v1.11 | 3/2 | Complete | 2026-05-07 | -| 49. Module Hardening | v1.11 | 3/3 | Complete | 2026-05-07 | -| 50. Repair Blind Spots | v1.11 | 1/1 | Complete | 2026-05-07 | -| 51. Testing Foundation | v2.0 | 0/0 | Not started | - | -| 52. Golden Datasets & CLI Contracts | v2.0 | 2/2 | Planning | 2026-05-08 | -| 53. Plugin Tests & Temp Vault E2E | v2.0 | 0/2 | Planning | - | -| 54. User Journey & Chaos Tests | v2.0 | 3/3 | Complete | 2026-05-08 | -| 55. CI Optimization & Consistency Audit | v2.0 | 0/2 | Planning | - | - ---- -*Roadmap updated: 2026-05-08 — v2.0 milestone phases created* +# Roadmap: PaperForge v2.1 — Contract-Driven Architecture & Engineering Hardening + +## Overview + +v2.1 transforms PaperForge from feature-stacked monoliths into a contract-driven system. Each phase builds on the last: quick consistency fixes unblock the work (Phase 56), stable JSON contracts define the machine-readable API surface (Phase 57), sync.py decomposes into testable adapters and a service layer (Phase 58), the state machine formalizes with enums and a field registry (Phase 59), and setup_wizard modularizes into focused classes with per-step JSON output (Phase 60). Every requirement maps to exactly one phase. + +## Phases + +- [ ] **Phase 56: Stop the Bleeding** — Quick consistency fixes: version sync, PyYAML, README/install unification, plugin version pinning +- [ ] **Phase 57: Contract Layer** — PFResult/PFError dataclasses, ErrorCode enum, CLI --json contracts (status/doctor/sync/ocr/dashboard) [Planned, 4 plans] +- [ ] **Phase 58: Service Extraction** — Decompose sync.py into adapters (bbt/zotero_paths/obsidian_frontmatter) and SyncService +- [ ] **Phase 59: State Machine & Field Registry** — PdfStatus/OcrStatus/Lifecycle enums, ALLOWED_TRANSITIONS, field_registry.yaml, doctor field checks +- [ ] **Phase 60: Setup Modularization** — SetupPlan/Checker/RuntimeInstaller/VaultInitializer/AgentInstaller/ConfigWriter classes, setup --headless --json + +## Phase Details + +### Phase 56: Stop the Bleeding +**Goal**: Quick consistency fixes that unblock all subsequent phases — version declarations synchronized, dependency declarations resolved, install documentation unified, and plugin runtime version pinned. +**Depends on**: Nothing (v2.0 complete) +**Requirements**: BLEED-01, BLEED-02, BLEED-03, BLEED-04 +**Success Criteria** (what must be TRUE): + 1. `scripts/check_version_sync.py` passes as a CI gate — all 6+ version declarations (__init__.py, manifest.json, versions.json, pyproject.toml, plugin manifest, docs) return consistent + 2. PyYAML dependency is resolved — either added to pyproject.toml as explicit dependency, or the yaml module existence check is removed from `paperforge doctor` (single source of truth — no "optional dependency" ambiguity) + 3. README, INSTALLATION, and setup-guide present a single unified primary install path — no conflicting recommendations across the three documents + 4. Plugin runtime `pip install paperforge` pins to the plugin manifest version — no version drift possible between the Obsidian plugin and the Python package +**Plans**: 2 plans + +Plans: +- [ ] 056-01-PLAN.md — Version sync CI gate script + PyYAML doctor hardening (BLEED-01, BLEED-02) +- [ ] 056-02-PLAN.md — Install doc unification + plugin version pinning (BLEED-03, BLEED-04) + +### Phase 57: Contract Layer +**Goal**: Stable JSON contracts (PFResult/PFError dataclasses, ErrorCode enum) that CLI commands produce and the plugin consumes — defining the machine-readable API surface consumed by SYNC, STAT, and SETP phases. +**Depends on**: Phase 56 +**Requirements**: CTRT-01, CTRT-02, CTRT-03, CTRT-04, CTRT-05, CTRT-06, CTRT-07, CTRT-08 +**Success Criteria** (what must be TRUE): + 1. `PFResult` and `PFError` dataclasses are defined in `paperforge/core/result.py` with `to_json()` serialization that survives round-trip (serialize → deserialize → equal) + 2. All error codes are centralized in `ErrorCode` enum in `paperforge/core/errors.py` — no scattered string-based error codes remain in any production module + 3. `paperforge status --json`, `doctor --json`, `sync --json`, and `ocr --diagnose --json` all return PFResult-format output with consistent `{ok, command, version, data, error}` shape + 4. `paperforge dashboard --json` returns a stable UI contract with stats (papers, pdf_health, ocr_health, domain_counts) and actionable permissions (can_sync, can_ocr, can_copy_context) + 5. Plugin reads dashboard data via `paperforge dashboard --json` CLI contract; fallback to direct `formal-library.json` reading remains available during the transition period (removed after 2 release cycles of stable PFResult) +**Plans**: 4 plans +**UI hint**: yes + +Plans: +- [ ] 057-01-PLAN.md — Core contract types: ErrorCode enum + PFResult/PFError dataclasses + round-trip tests (CTRT-01, CTRT-02) +- [ ] 057-02-PLAN.md — Status & doctor --json PFResult wrapping + contract tests (CTRT-03, CTRT-04) +- [ ] 057-03-PLAN.md — Sync & ocr --diagnose --json PFResult wrapping + contract tests (CTRT-05, CTRT-06) +- [ ] 057-04-PLAN.md — Dashboard command + plugin contract integration + contract tests (CTRT-07, CTRT-08) + +### Phase 58: Service Extraction +**Goal**: Monolithic sync.py decomposed into focused adapters and a SyncService class — each module independently testable with its own unit tests, sync.py reduced to a thin CLI dispatch layer. +**Depends on**: Phase 57 +**Requirements**: SYNC-01, SYNC-02, SYNC-03, SYNC-04, SYNC-05 +**Success Criteria** (what must be TRUE): + 1. `paperforge/adapters/bbt.py` contains all BBT JSON parsing functions (`load_export_rows`, `_normalize_attachment_path`, `_identify_main_pdf`, `extract_authors`, `resolve_item_collection_paths`) — importable and testable in isolation from sync.py + 2. `paperforge/adapters/zotero_paths.py` contains all path resolution functions (`obsidian_wikilink_for_pdf`, `absolutize_vault_path`, `obsidian_wikilink_for_path`) — no path resolution logic remains in sync.py + 3. `paperforge/adapters/obsidian_frontmatter.py` contains frontmatter read/write/update operations using YAML parser (replacing regex-based parsing where possible) — importable and testable in isolation + 4. `paperforge/services/sync_service.py` exists as a `SyncService` class that wraps the decomposed adapter modules; `worker/sync.py` becomes a thin dispatch layer with no business logic beyond orchestration + 5. All extracted modules have passing unit tests covering BBT JSON variants, path formats (storage:/, absolute Windows, CJK filenames, spaces), and frontmatter edge cases — existing sync behavior preserved (no regressions) +**Plans**: TBD + +### Phase 59: State Machine & Field Registry +**Goal**: Formalized state machine with explicit enums and allowed transitions, plus a field registry that `paperforge doctor` can validate against — making state changes predictable and field drift detectable. +**Depends on**: Phase 57 +**Requirements**: STAT-01, STAT-02, STAT-03, STAT-04, STAT-05 +**Success Criteria** (what must be TRUE): + 1. `PdfStatus`, `OcrStatus`, and `Lifecycle` enums are defined in `paperforge/core/state.py` with explicit string values — every existing state string in the codebase maps to an enum member + 2. `ALLOWED_TRANSITIONS` table defines legal state migrations; workers validate transitions before writing state changes — illegal transitions are rejected with clear, actionable error messages + 3. `paperforge/schema/field_registry.yaml` defines every field in the system (formal note frontmatter, index entries, paper-meta.json) with public/required/type/owner/description metadata — complete coverage of all field-carrying structures + 4. `paperforge doctor` checks field completeness against the registry — missing required fields produce actionable warnings with migration suggestions; unknown/invalid fields produce warnings to flag drift + 5. Edge case: fields present in data but absent from registry trigger a "drift detected" diagnostic; fields in registry but absent from data trigger "missing required field" with severity levels +**Plans**: TBD + +### Phase 60: Setup Modularization +**Goal**: Monolithic setup_wizard.py decomposed into six focused classes with explicit dependencies — enabling `paperforge setup --headless --json` to return per-step status that the Obsidian plugin can render as progress. +**Depends on**: Phase 57 +**Requirements**: SETP-01, SETP-02, SETP-03, SETP-04, SETP-05, SETP-06, SETP-07 +**Success Criteria** (what must be TRUE): + 1. `SetupPlan`, `SetupChecker`, `RuntimeInstaller`, `VaultInitializer`, `AgentInstaller`, and `ConfigWriter` classes each exist with a single responsibility and explicit interface — no class exceeds its boundary + 2. `paperforge setup --headless --json` returns per-step status — each step has independent `{ok, error, message}` fields; the plugin UI can render step-by-step progress from the JSON output + 3. `ConfigWriter` writes `paperforge.json` atomically using tempfile + os.replace — no partial or corrupted config files possible on crash or interrupt + 4. `RuntimeInstaller` handles `pip install` with explicit version pinning (from plugin manifest), progress output via callback, and classified errors using the ErrorCode enum from Phase 57 + 5. `SetupChecker` validates all preconditions (Python executable, pip availability, dependency health) before any installation step begins — failures produce classified, user-readable diagnostics +**Plans**: TBD +**UI hint**: yes + +## Progress + +**Execution Order:** 56 → 57 → 58 → 59 → 60 + +| Phase | Plans Complete | Status | Completed | +|-------|----------------|--------|-----------| +| 56. Stop the Bleeding | 0/2 | Ready to execute | — | +| 57. Contract Layer | 0/TBD | Not started | — | +| 58. Service Extraction | 0/TBD | Not started | — | +| 59. State Machine & Field Registry | 0/TBD | Not started | — | +| 60. Setup Modularization | 0/TBD | Not started | — | diff --git a/.planning/STATE.md b/.planning/STATE.md index a6394c6a..34f33ac7 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -1,19 +1,3 @@ ---- -gsd_state_version: 1.0 -milestone: v2.0 -milestone_name: Testing Infrastructure — 6-Layer Quality Gates -status: verifying -stopped_at: "Completed Phase 54: User Journey & Chaos Tests" -last_updated: "2026-05-08T16:53:18.641Z" -last_activity: 2026-05-08 -progress: - total_phases: 14 - completed_phases: 6 - total_plans: 19 - completed_plans: 19 - percent: 25 ---- - # Project State ## Project Reference @@ -21,39 +5,34 @@ progress: See: .planning/PROJECT.md (updated 2026-05-08) **Core value:** Researchers always know what papers they have, what state those papers are in, and whether each paper is reliably usable by AI with traceable fulltext, figures, notes, and source links. - -**Current focus:** Phase 53 — Plugin Tests & Temp Vault E2E +**Current focus:** Phase 56 — Stop the Bleeding ## Current Position -Phase: 53 of 55 (Plugin Tests & Temp Vault E2E) -Plan: 2 of 2 in current phase -Status: Phase complete — ready for verification -Last activity: 2026-05-08 +Phase: 57 of 60 (Contract Layer) +Plan: 0 of 4 in current phase +Status: Ready to execute +Last activity: 2026-05-09 — Phase 57 plans created (4 plans, 12 tasks, 2 waves) -Progress: [##### ] 25% +Progress: [░░░░░░░░░░░░░░░░░░░░] 0% ## Performance Metrics **Velocity:** - -- Total plans completed: 2 -- Average duration: ~45min -- Total execution time: ~90min +- Total plans completed: 0 +- Average duration: — min +- Total execution time: 0.0 hours **By Phase:** | Phase | Plans | Total | Avg/Plan | |-------|-------|-------|----------| -| 53 | 2 | ~90min | ~45min | +| — | — | — | — | **Recent Trend:** - -- Last 5 plans: 2 completed (53-001, 53-002) -- Trend: Testing/infrastructure phase +- No plans executed yet. *Updated after each plan completion* -| Phase 54 P001-003 | 89 | 16 tasks | 12 files | ## Accumulated Context @@ -62,11 +41,10 @@ Progress: [##### ] 25% Decisions are logged in PROJECT.md Key Decisions table. Recent decisions affecting current work: -- All CI decisions from research (plasma matrix, path-filtered jobs, snapshot strategy) applied to phase design -- FIX requirements distributed: fixture hierarchy (conftest) in Phase 51, fixture data files (golden datasets) in Phase 52 -- CI requirements distributed: PR check in Phase 51, Node runner in Phase 53, chaos workflow in Phase 54, optimization in Phase 55 -- **53-001**: Used dependency injection pattern for src/ modules to work around vitest v2.1.x CJS/ESM mock limitations -- **53-002**: Moved test files into `paperforge/plugin/tests/` for vitest compatibility; e2e_cli_invoker returns (invoker, vault_path) tuple +- [v2.1]: Incremental extraction from sync.py — adapters first, then SyncService, keeping sync.py as thin shell. No full rewrite. +- [v2.1]: Dual JSON output during contract transition — old format + new PFResult side-by-side until 2 release cycles of stability. +- [v2.1]: Plugin keeps fallback to direct index reading during PFResult transition — removed after 2 stable release cycles. +- [v2.1]: Tests must not be modified to pass if code is broken — tests verify contracts, not implementation convenience. ### Pending Todos @@ -78,6 +56,6 @@ None yet. ## Session Continuity -Last session: 2026-05-08T16:53:18.638Z -Stopped at: Completed Phase 54: User Journey & Chaos Tests +Last session: 2026-05-09 +Stopped at: Roadmap created — files written (ROADMAP.md, STATE.md, REQUIREMENTS.md traceability) Resume file: None diff --git a/.planning/phases/057-contract-layer/057-01-PLAN.md b/.planning/phases/057-contract-layer/057-01-PLAN.md new file mode 100644 index 00000000..046f83c6 --- /dev/null +++ b/.planning/phases/057-contract-layer/057-01-PLAN.md @@ -0,0 +1,231 @@ +--- +phase: 57-contract-layer +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: + - paperforge/core/__init__.py + - paperforge/core/errors.py + - paperforge/core/result.py + - tests/cli/test_contract_roundtrip.py +autonomous: true +requirements: + - CTRT-01 + - CTRT-02 + +must_haves: + truths: + - "ErrorCode enum exists with all 8 required members (PYTHON_NOT_FOUND, VERSION_MISMATCH, BBT_EXPORT_NOT_FOUND, OCR_TOKEN_MISSING, SYNC_FAILED, VALIDATION_ERROR, INTERNAL_ERROR, UNKNOWN)" + - "PFResult dataclass serializes to JSON with {ok, command, version, data, error} shape" + - "PFError dataclass serializes to JSON with {code, message, details} shape" + - "JSON round-trip: serialize PFResult -> JSON string -> deserialize -> equal to original" + artifacts: + - path: "paperforge/core/__init__.py" + provides: "Core package init" + min_lines: 1 + - path: "paperforge/core/errors.py" + provides: "ErrorCode enum with all error codes" + exports: ["ErrorCode"] + - path: "paperforge/core/result.py" + provides: "PFResult and PFError dataclasses with to_dict/to_json/from_json" + exports: ["PFResult", "PFError"] + - path: "tests/cli/test_contract_roundtrip.py" + provides: "Round-trip unit tests for PFResult/PFError serialization" + min_lines: 40 + key_links: + - from: "paperforge/core/result.py" + to: "paperforge/core/errors.py" + via: "import ErrorCode" + pattern: "from paperforge.core.errors import ErrorCode" + - from: "tests/cli/test_contract_roundtrip.py" + to: "paperforge/core/result.py" + via: "import PFResult" + pattern: "from paperforge.core.result import PFResult" +--- + + +Create the core contract types (PFResult, PFError dataclasses and ErrorCode enum) that all subsequent CLI --json wrapping and the dashboard command depend on. This is the foundational layer for the contract-driven architecture. + +Purpose: Define the machine-readable API surface that every CLI command will produce and the plugin will consume. +Output: paperforge/core/errors.py (ErrorCode enum), paperforge/core/result.py (PFResult/PFError), round-trip unit tests + + + +@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md +@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md + + + +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/057-contract-layer/057-CONTEXT.md + + +From paperforge/__init__.py: +```python +__version__ = "1.4.17rc3" +``` + +From existing tests/cli/test_contract_helpers.py (assert_json_shape, normalize_snapshot, assert_valid_json available for reuse): +```python +def assert_json_shape(data: dict, required_keys: set[str], optional_keys: set[str] | None = None) +def assert_valid_json(output: str) -> dict +def normalize_snapshot(output: str, vault_path: str | None = None) -> str +``` + + + + + + + Task 1: Create ErrorCode enum in paperforge/core/errors.py + + paperforge/core/__init__.py + paperforge/core/errors.py + + + Create the paperforge/core/ directory with paperforge/core/__init__.py containing a minimal package docstring. + + Create paperforge/core/errors.py with: + + ErrorCode(str, Enum) -- string enum using SCREAMING_SNAKE_CASE naming with values equal to names. The 8 members per user decision (D-01, D-02, D-03): + + - PYTHON_NOT_FOUND: Python interpreter not found by the plugin + - VERSION_MISMATCH: Version mismatch between plugin manifest and installed package + - BBT_EXPORT_NOT_FOUND: Better BibTeX JSON export file missing or invalid + - OCR_TOKEN_MISSING: PaddleOCR API token not configured in .env + - SYNC_FAILED: Sync operation failed (selection-sync or index-refresh) + - VALIDATION_ERROR: Validation error -- use details.field for field-level granularity + - INTERNAL_ERROR: Unexpected internal error -- use details for trace/context info + - UNKNOWN: Catch-all fallback for unmapped/unexpected errors + + Each member MUST have a docstring describing when it is used. + + Add helper: @classmethod def from_exception(cls, exc: Exception) -> ErrorCode that maps common exception types (FileNotFoundError -> BBT_EXPORT_NOT_FOUND, ValueError -> VALIDATION_ERROR, etc.) to appropriate codes. Unknown exceptions map to UNKNOWN. + + Per user decision (CONTEXT.md): ErrorCode enum ONLY -- no JSON schema generation, no auto-generation from field_registry.yaml (deferred to Phase 59). + + + python -c "from paperforge.core.errors import ErrorCode; e = ErrorCode.UNKNOWN; assert isinstance(e, ErrorCode); assert e.value == 'UNKNOWN'; print('OK enum:', e.value)" + + ErrorCode enum exists in paperforge/core/errors.py with 8 members, all SCREAMING_SNAKE_CASE, each with docstring, plus from_exception helper. + + + + Task 2: Create PFResult and PFError dataclasses in paperforge/core/result.py + + paperforge/core/result.py + + + Create paperforge/core/result.py with two dataclasses using `from __future__ import annotations`. + + PFError dataclass: + - code: str -- Error code string (e.g., "SYNC_FAILED", "VALIDATION_ERROR") + - message: str -- Human-readable error description + - details: dict = field(default_factory=dict) -- Optional structured payload (e.g., {"field": "pdf_path", "reason": "not found"}) + - to_dict() -> dict: Returns {"code": code, "message": message, "details": details} + - WARNING: details may contain non-JSON-serializable types -- coerce with str() on serialization. + + PFResult dataclass: + - ok: bool -- True when operation succeeded (even with partial results + warnings in data); False only when error is set + - command: str -- CLI command name (e.g., "status", "sync") + - version: str -- Version string from __version__. Use lazy import inside methods (not at module level) to avoid circular imports. + - data: Any = None -- Payload (dict for commands with structured output, list for collections, None for errors) + - error: PFError | None = None -- Set only when ok is False + + __post_init__ validation: if ok is True and error is not None -> raise ValueError("ok=True requires error=None"). if ok is False and error is None -> raise ValueError("ok=False requires an error"). + + Methods: + - to_dict() -> dict: Returns {"ok": ok, "command": command, "version": version, "data": data, "error": error.to_dict() if error else None} + - to_json(indent: int = 2) -> str: JSON-dumps to_dict() with ensure_ascii=False, indent=indent, default=str for non-serializable values + - @staticmethod from_json(json_str: str) -> PFResult: Parse JSON string back to PFResult. If the "error" field is not None, construct PFError from the dict. + + Module-level __all__ = ["PFResult", "PFError"]. + + Per user decisions: + - Top-level fields MUST be {ok, command, version, data, error} following Google JSON Style Guide + - Version: semver string from __version__ (e.g., "1.4.17rc3") + - ok=false only when error is set; ok=true even with partial results (add warnings sub-field in data) + - Error shape: {"code":"ERR_CODE","message":"Human text","details":{}} + + + python -c " +from paperforge.core.result import PFResult, PFError +# Test OK result +r = PFResult(ok=True, command='status', version='1.4.17rc3', data={'papers': 5}) +d = r.to_dict() +assert d['ok'] == True +assert d['command'] == 'status' +assert d['error'] is None +# Test error result +er = PFResult(ok=False, command='sync', version='1.4.17rc3', error=PFError(code='SYNC_FAILED', message='Sync failed', details={'key': 'val'})) +d2 = er.to_dict() +assert d2['ok'] == False +assert d2['error']['code'] == 'SYNC_FAILED' +# Test round-trip +j = r.to_json() +r2 = PFResult.from_json(j) +assert r2.ok == r.ok +assert r2.command == r.command +# Test validation +try: + PFResult(ok=True, command='test', version='1.0', error=PFError(code='X', message='y')) + assert False, 'Should have raised' +except ValueError: + pass +print('OK: round-trip + validation valid') +" + + PFResult and PFError dataclasses in paperforge/core/result.py with to_dict(), to_json(), from_json(), and __post_init__ validation. Round-trip (serialize->deserialize->equal) verified. + + + + Task 3: Write round-trip unit tests in tests/cli/test_contract_roundtrip.py + + tests/cli/test_contract_roundtrip.py + + + Create tests/cli/test_contract_roundtrip.py with pytest tests covering: + + 1. TestPFResultShape: Verify PFResult.to_dict() has exactly {ok, command, version, data, error} keys. Test both success and error states. + + 2. TestPFErrorShape: Verify PFError.to_dict() has exactly {code, message, details} keys. Test with empty dict, with populated details. + + 3. TestPFResultRoundTrip: Full round-trip tests: + - Serialize PFResult (ok=True, with data dict) -> JSON string -> deserialize -> assert equal + - Serialize PFResult (ok=False, with PFError) -> JSON string -> deserialize -> assert equal + - Serialize PFResult with list data -> JSON -> deserialize -> assert equal + + 4. TestPFResultValidation: Verify __post_init__ raises ValueError for: + - ok=True with error set + - ok=False without error + + Use existing fixtures/conftest patterns (from tests/cli/conftest.py). No vault needed for these pure unit tests -- they test the dataclasses directly. + + Keep tests compact: 4 test classes, no more than 5 assertions each. + + + python -m pytest tests/cli/test_contract_roundtrip.py -v --tb=short + + All round-trip tests pass: shape verification, serialization round-trip, validation error cases. + + + + + +- python -c "from paperforge.core.errors import ErrorCode; print(list(ErrorCode))" shows all 8 members +- python -c "from paperforge.core.result import PFResult, PFError; print(PFResult(ok=True, command='status', version='1.0', data={}).to_json())" produces valid JSON +- pytest tests/cli/test_contract_roundtrip.py -v passes all tests + + + +- [ ] paperforge/core/errors.py exists with ErrorCode enum (8 members, SCREAMING_SNAKE_CASE, docstrings, from_exception) +- [ ] paperforge/core/result.py exists with PFResult and PFError dataclasses (to_dict, to_json, from_json, __post_init__ validation) +- [ ] Round-trip unit tests pass without requiring a vault +- [ ] All downstream phases (57-02, 57-03, 57-04) can import from paperforge.core + + + +After completion, create .planning/phases/057-contract-layer/057-01-SUMMARY.md + diff --git a/.planning/phases/057-contract-layer/057-02-PLAN.md b/.planning/phases/057-contract-layer/057-02-PLAN.md new file mode 100644 index 00000000..21df3ae9 --- /dev/null +++ b/.planning/phases/057-contract-layer/057-02-PLAN.md @@ -0,0 +1,281 @@ +--- +phase: 57-contract-layer +plan: 02 +type: execute +wave: 2 +depends_on: + - 57-01 +files_modified: + - paperforge/worker/status.py + - paperforge/cli.py + - paperforge/commands/status.py + - tests/cli/test_json_contracts.py +autonomous: true +requirements: + - CTRT-03 + - CTRT-04 + +must_haves: + truths: + - "paperforge status --json returns PFResult envelope with data containing current flat dict shape" + - "paperforge doctor --json returns PFResult envelope with structured checks in data" + - "All PFResult output from status and doctor has valid {ok, command, version, data, error} shape" + artifacts: + - path: "paperforge/worker/status.py" + provides: "Modified run_status (PFResult wrapping) + run_doctor with --json support" + contains: "from paperforge.core.result import" + - path: "paperforge/cli.py" + provides: "doctor subparser with --json flag" + contains: "add_argument.*--json" + - path: "tests/cli/test_json_contracts.py" + provides: "Updated PFResult shape tests for status --json, new tests for doctor --json" + min_lines: 140 + key_links: + - from: "paperforge/worker/status.py" + to: "paperforge/core/result.py" + via: "PFResult wrapper in JSON output path" + - from: "paperforge/worker/status.py" + to: "paperforge/core/errors.py" + via: "ErrorCode entries for doctor checks" + - from: "tests/cli/test_json_contracts.py" + to: "paperforge/worker/status.py" + via: "CLI subprocess invoker exercises both commands" +--- + + +Wrap the existing `paperforge status --json` output in a PFResult envelope (CTRT-03) and add `--json` output to `paperforge doctor` (CTRT-04). The existing flat dict output is preserved inside the `data` field (backward compatible). Doctor gets new structured JSON output with check results. + +Purpose: All status and doctor commands produce machine-readable PFResult-format JSON with consistent shape. +Output: Modified status.py (PFResult wrapping for run_status, JSON for run_doctor), CLI updates, contract tests + + + +@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md +@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md + + + +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/057-contract-layer/057-CONTEXT.md + + +From paperforge/core/result.py (after Plan 57-01): +```python +@dataclass +class PFResult: + ok: bool + command: str + version: str + data: Any = None + error: PFError | None = None + def to_dict(self) -> dict: ... + def to_json(self, indent: int = 2) -> str: ... + @staticmethod def from_json(json_str: str) -> PFResult: ... + +@dataclass +class PFError: + code: str + message: str + details: dict +``` + +From paperforge/core/errors.py (after Plan 57-01): +```python +class ErrorCode(str, Enum): + PYTHON_NOT_FOUND = "PYTHON_NOT_FOUND" + VERSION_MISMATCH = "VERSION_MISMATCH" + ... +``` + +Existing status --json output (paperforge/worker/status.py:973-998): +```python +data = { + "version": __import__("paperforge").__version__, + "vault": str(vault), + "system_dir": cfg["system_dir"], + "resources_dir": cfg["resources_dir"], + "exports": len(export_files), + "domains": len(config.get("domains", [])), + "total_papers": record_count, + "formal_notes": note_count, + "bases": base_count, + "ocr": {"total": ..., "pending": ..., "done": ..., "failed": ...}, + "path_errors": ..., + "env_configured": ..., + "lifecycle_level_counts": ..., + "health_aggregate": ..., + "maturity_distribution": ..., +} +print(_json.dumps(data, indent=2, ensure_ascii=False)) +``` + +Existing doctor command dispatch (paperforge/cli.py:449-453): +```python +if args.command == "doctor": + from paperforge.worker.status import run_doctor + return run_doctor(vault) +``` + +Existing run_doctor signature (worker/status.py:359): +```python +def run_doctor(vault: Path, verbose: bool = False) -> int: +``` + + + + + + + Task 1: Wrap status --json output in PFResult envelope + + paperforge/worker/status.py + + + In paperforge/worker/status.py, modify the `run_status` function's JSON output block (around line 973): + + 1. Import PFResult at the top of the function (lazy import, inside the `if json_output:` block): `from paperforge.core.result import PFResult` + + 2. Build the same `data` dict as currently, preserving its exact shape (backward compatibility of the data payload). + + 3. Wrap in PFResult: + ```python + result = PFResult( + ok=True, + command="status", + version=data["version"], # from the existing data dict + data=data, + ) + print(result.to_json()) + ``` + + 4. Remove the `import json as _json` at the function top if it becomes unused (check -- _json may still be used elsewhere in the function). If _json is only used for the print, replace with PFResult. Otherwise keep _json and suppress the warning. + + 5. Handle error case: if any unrecoverable error occurs during status computation (e.g., vault doesn't exist), wrap in PFResult with ok=False and appropriate ErrorCode. Use try/except around the existing function body, catching broad exceptions and wrapping in PFResult(ok=False, command="status", error=PFError(code=ErrorCode.INTERNAL_ERROR.value, message=str(e), details={"vault": str(vault)})). + + IMPORTANT: Do NOT change the `data` dict structure. The existing flat dict is the backward-compatible payload. Only wrap it in PFResult. + + + python -m paperforge status --json --vault C:\Users\Lin\AppData\Local\Temp\opencode 2>&1 || true + + + status --json output is wrapped in PFResult: {"ok": true, "command": "status", "version": "1.4.17rc3", "data": {existing flat dict}, "error": null} + Errors produce PFResult with ok=false. + + + + + Task 2: Add --json output to doctor command + + paperforge/cli.py + paperforge/worker/status.py + paperforge/commands/status.py + + + Two modifications: + + A. Add --json flag to doctor subparser in paperforge/cli.py: + ```python + doctor_p = sub.add_parser("doctor", help="Validate PaperForge setup and configuration") + doctor_p.add_argument("--json", action="store_true", dest="json_output", help="Output JSON") + ``` + + B. Modify run_doctor function in paperforge/worker/status.py: + 1. Add `json_output: bool = False` parameter to `def run_doctor(vault: Path, verbose: bool = False, json_output: bool = False) -> int:` + 2. At the end of the function (after verdict is computed but before print statements), add: + ```python + if json_output: + from paperforge.core.result import PFResult, PFError + from paperforge.core.errors import ErrorCode + from paperforge import __version__ + + if has_fail: + result = PFResult( + ok=False, + command="doctor", + version=__version__, + data={"checks": checks_data, "verdict": "FAIL"}, + error=PFError(code=ErrorCode.INTERNAL_ERROR.value, message=f"{verdict}: {next_action}", details={"fail_count": fail_count, "categories": list(fail_categories)}), + ) + else: + result = PFResult( + ok=True, + command="doctor", + version=__version__, + data={"checks": checks_data, "verdict": verdict, "verdict_message": next_action}, + ) + print(result.to_json()) + return 1 if has_fail else 0 + ``` + 3. Build `checks_data` as a list of dicts: `[{"category": cat, "status": status, "message": msg, "fix": fix} for cat, status, msg, fix in checks]` + 4. Compute `fail_count = sum(1 for _, s, _, _ in checks if s == "fail")` near the existing `has_fail` computation. + 5. Keep the existing text output path unchanged (everything before the `if json_output:` block). + + C. Update cli.py doctor dispatch to pass json_output: + ```python + if args.command == "doctor": + from paperforge.worker.status import run_doctor + return run_doctor(vault, verbose=getattr(args, "verbose", False), json_output=getattr(args, "json_output", False)) + ``` + + + python -m paperforge doctor --json --vault C:\Users\Lin\AppData\Local\Temp\opencode 2>&1 || true + + doctor --json produces PFResult with structured checks in data field. Text output path unchanged. CLI passes json_output to run_doctor. + + + + Task 3: Update CLI contract tests for PFResult-shaped status and doctor + + tests/cli/test_json_contracts.py + + + Update tests/cli/test_json_contracts.py: + + 1. In TestStatusJson, add a new method `test_status_json_pfresult_shape` that verifies the output is wrapped in PFResult: + - Call cli_invoker(["status", "--json"]) + - assert_valid_json(result.stdout) + - Parse into dict + - Assert top-level keys are {"ok", "command", "version", "data", "error"} using assert_json_shape + - Assert "ok" is True + - Assert "command" == "status" + - Assert "version" is a non-empty string + - Assert "data" is a dict with at least {"version", "vault"} keys + - Assert "error" is None + + 2. Add TestDoctorJson class (mirroring TestStatusJson pattern) with: + - REQUIRED_KEYS = {"ok", "command", "version", "data", "error"} + - test_doctor_json_pfresult_shape: verifies PFResult envelope + - test_doctor_json_data_has_checks: verifies data.checks is a list + - test_doctor_json_no_vault: doctor --json on nonexistent vault exits non-zero + + 3. Keep all existing tests (do NOT remove test_status_json_value_semantics etc. — they must continue to pass on the wrapped output, since the data field preserves the old shape) + + Do NOT modify test_status_json_value_semantics and test_status_json_snapshot_regression — these tests read the PFResult.data sub-dict and must still pass. + + + python -m pytest tests/cli/test_json_contracts.py -v --tb=short -k "pfresult or doctor" 2>&1 || true + + New PFResult shape tests pass. Existing contract tests (old flat dict shape inside data field) continue to pass. + + + + + +- paperforge status --json output starts with {"ok": true, "command": "status", ...} +- paperforge doctor --json output starts with {"ok": true/false, "command": "doctor", ...} +- All existing tests still pass (backward compatible data payload preserved) +- New PFResult shape tests pass + + + +- [ ] status --json wrapped in PFResult with backward-compatible data payload +- [ ] doctor --json produces PFResult with structured checks in data +- [ ] doctor subparser in cli.py has --json flag +- [ ] run_doctor accepts json_output parameter +- [ ] New PFResult contract tests pass alongside existing contract tests + + + +After completion, create .planning/phases/057-contract-layer/057-02-SUMMARY.md + \ No newline at end of file diff --git a/.planning/phases/057-contract-layer/057-03-PLAN.md b/.planning/phases/057-contract-layer/057-03-PLAN.md new file mode 100644 index 00000000..9fd8cd65 --- /dev/null +++ b/.planning/phases/057-contract-layer/057-03-PLAN.md @@ -0,0 +1,253 @@ +--- +phase: 57-contract-layer +plan: 03 +type: execute +wave: 2 +depends_on: + - 57-01 +files_modified: + - paperforge/commands/sync.py + - paperforge/commands/ocr.py + - paperforge/ocr_diagnostics.py + - paperforge/worker/sync.py + - tests/cli/test_json_contracts.py +autonomous: true +requirements: + - CTRT-05 + - CTRT-06 + +must_haves: + truths: + - "paperforge sync --json returns PFResult with sync result data" + - "paperforge ocr --diagnose --json returns PFResult with diagnostic data" + - "Both outputs have valid {ok, command, version, data, error} PFResult shape" + artifacts: + - path: "paperforge/commands/sync.py" + provides: "--json flag on sync command with PFResult output" + contains: "PFResult" + - path: "paperforge/commands/ocr.py" + provides: "--json flag on ocr --diagnose with PFResult output" + contains: "json_output" + - path: "paperforge/ocr_diagnostics.py" + provides: "Optional enhanced ocr_doctor with json-friendly output" + - path: "tests/cli/test_json_contracts.py" + provides: "PFResult shape tests for sync --json and ocr --diagnose --json" + key_links: + - from: "paperforge/commands/sync.py" + to: "paperforge/core/result.py" + via: "import PFResult" + - from: "paperforge/commands/ocr.py" + to: "paperforge/core/result.py" + via: "import PFResult" +--- + + +Add --json flag to sync and ocr --diagnose commands, wrapping output in PFResult envelope (CTRT-05, CTRT-06). Sync reports operation status (selection-sync + index-refresh). OCR diagnose wraps existing diagnostic dict. + +Purpose: All core CLI commands produce consistent PFResult JSON output. +Output: Modified sync.py, ocr.py with --json flags; contract tests + + + +@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md +@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md + + + +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/057-contract-layer/057-CONTEXT.md + + +paperforge/commands/sync.py run(args): +- Calls run_selection_sync(vault, verbose) and run_index_refresh(vault, verbose, rebuild_index) +- Both return int exit codes +- Current output: text logs via print/logger, no structured JSON + +paperforge/commands/ocr.py run(args): +- Calls _diagnose(vault, live) for --diagnose +- _diagnose calls ocr_doctor(config, live) which returns dict with {level, passed, error, fix, raw_response} +- Current output: human-readable text via print, no --json flag + +paperforge/ocr_diagnostics.py ocr_doctor(config, live): +- Returns dict: {"level": int, "passed": bool, "error": str, "fix": str, "raw_response": str (optional)} +- Already a structured result — ideal for PFResult data payload + + + + + + + Task 1: Add --json to sync command with PFResult output + + paperforge/commands/sync.py + + + In paperforge/commands/sync.py: + + 1. Add `--json` argument at the top of the module (after domain arg): + ```python + # The --json flag is passed via args — sync command currently gets its args from cli.py dispatch. + ``` + Actually, the sync parser is in cli.py. Add to the sync subparser in paperforge/cli.py: + ```python + p_sync.add_argument("--json", action="store_true", dest="json_output", help="Output JSON") + ``` + + 2. In sync.py `run()` function, after both selection_sync and index_refresh complete, add: + ```python + if getattr(args, "json_output", False): + from paperforge.core.result import PFResult, PFError + from paperforge.core.errors import ErrorCode + from paperforge import __version__ + + data = { + "selection_sync": {"exit_code": selection_code} if selection_code is not None else None, + "index_refresh": {"exit_code": index_code} if index_code is not None else None, + "vault": str(vault), + } + + if exit_code == 0: + result = PFResult(ok=True, command="sync", version=__version__, data=data) + else: + result = PFResult( + ok=False, command="sync", version=__version__, data=data, + error=PFError(code=ErrorCode.SYNC_FAILED.value, message="Sync completed with errors", details={"selection_code": selection_code, "index_code": index_code}), + ) + print(result.to_json()) + return exit_code + ``` + + 3. Track the individual exit codes: store `selection_code` and `index_code` variables instead of reusing `code`. + ```python + selection_code = None + index_code = None + if not index_only: + run_selection_sync = _get_run_selection_sync() + selection_code = run_selection_sync(vault, verbose=getattr(args, "verbose", False)) + if selection_code != 0: + exit_code = selection_code + if not selection_only: + run_index_refresh = _get_run_index_refresh() + index_code = run_index_refresh(vault, verbose=getattr(args, "verbose", False), rebuild_index=rebuild_index) + if index_code != 0 and exit_code == 0: + exit_code = index_code + ``` + + 4. IMPORTANT: The --json block must come AFTER both workers complete but BEFORE the `return exit_code`. The text output path (existing behavior) must remain unchanged. + + Capture worker std output if possible, or just report exit codes. Do NOT change worker return types. + + + python -m paperforge sync --json --vault C:\Users\Lin\AppData\Local\Temp\opencode 2>&1 || true + + sync --json produces PFResult with selection_sync and index_refresh exit codes in data. Text output path unchanged. + + + + Task 2: Add --json to ocr --diagnose with PFResult output + + paperforge/commands/ocr.py + paperforge/cli.py + + + Two modifications: + + A. In paperforge/cli.py, add --json to the ocr parser: + ```python + p_ocr.add_argument("--json", action="store_true", dest="json_output", help="Output JSON") + ``` + (The --json flag applies to both ocr run and ocr diagnose, but only diagnose uses it in this phase.) + + B. In paperforge/commands/ocr.py, modify _diagnose function: + 1. Add `json_output: bool = False` parameter + 2. Inside _diagnose, after getting the `result` dict from ocr_doctor, add: + ```python + if json_output: + from paperforge.core.result import PFResult, PFError + from paperforge.core.errors import ErrorCode + from paperforge import __version__ + + passed = result.get("passed", False) + if passed: + pf_result = PFResult( + ok=True, command="ocr-diagnose", version=__version__, + data={"level": result.get("level"), "passed": True, "message": result.get("message", "All checks passed")}, + ) + else: + pf_result = PFResult( + ok=False, command="ocr-diagnose", version=__version__, + data={"level": result.get("level"), "passed": False}, + error=PFError( + code=ErrorCode.OCR_TOKEN_MISSING.value if result.get("level") == 1 else ErrorCode.INTERNAL_ERROR.value, + message=result.get("error", "Unknown failure"), + details={"fix": result.get("fix", ""), "raw_response": result.get("raw_response", "")}, + ), + ) + print(pf_result.to_json()) + return 0 if passed else 1 + ``` + 3. Keep the existing text output path unchanged (before the `if json_output:` block). + + C. Update the call site in ocr.py `run()`: + ```python + diagnose_only = getattr(args, "diagnose", False) or ocr_action == "doctor" + json_output = getattr(args, "json_output", False) + if diagnose_only: + return _diagnose(vault, live=live, json_output=json_output) + ``` + + + python -m paperforge ocr --diagnose --json --vault C:\Users\Lin\AppData\Local\Temp\opencode 2>&1 || true + + ocr --diagnose --json produces PFResult with diagnostic level/passed/message in data. Text path unchanged. + + + + Task 3: Add CLI contract tests for sync --json and ocr --diagnose --json + + tests/cli/test_json_contracts.py + + + In tests/cli/test_json_contracts.py, add: + + 1. TestSyncJson class: + - REQUIRED_KEYS = {"ok", "command", "version", "data", "error"} + - test_sync_json_pfresult_shape: invokes sync --json on empty vault, verifies PFResult shape + - test_sync_json_data_has_keys: verifies data has selection_sync and index_refresh keys + - test_sync_json_no_vault: nonexistent vault exits non-zero + + 2. TestOcrDiagnoseJson class: + - REQUIRED_KEYS = {"ok", "command", "version", "data", "error"} + - test_ocr_diagnose_json_pfresult_shape: invokes ocr --diagnose --json, verifies PFResult shape + - test_ocr_diagnose_json_data: verifies data has level and passed keys + - test_ocr_diagnose_json_no_vault: nonexistent vault exits non-zero + + Reuse the existing assert_json_shape and assert_valid_json helpers from test_contract_helpers.py. + Keep tests compact: each test class with 3 methods. + + + python -m pytest tests/cli/test_json_contracts.py -v --tb=short -k "sync or ocr" 2>&1 || true + + New PFResult contract tests for sync and ocr pass alongside existing tests. + + + + + +- paperforge sync --json produces valid PFResult JSON +- paperforge ocr --diagnose --json produces valid PFResult JSON +- All existing sync/ocr text output paths unchanged +- All tests pass + + + +- [ ] sync --json added with PFResult output (data contains selection_sync and index_refresh codes) +- [ ] ocr --diagnose --json added with PFResult output (data contains level/passed/message) +- [ ] Both commands' text output paths preserved +- [ ] Contract tests pass for both commands + + + +After completion, create .planning/phases/057-contract-layer/057-03-SUMMARY.md + diff --git a/.planning/phases/057-contract-layer/057-04-PLAN.md b/.planning/phases/057-contract-layer/057-04-PLAN.md new file mode 100644 index 00000000..0fe30812 --- /dev/null +++ b/.planning/phases/057-contract-layer/057-04-PLAN.md @@ -0,0 +1,408 @@ +--- +phase: 57-contract-layer +plan: 04 +type: execute +wave: 2 +depends_on: + - 57-01 +files_modified: + - paperforge/cli.py + - paperforge/commands/__init__.py + - paperforge/commands/dashboard.py + - paperforge/worker/asset_index.py + - paperforge/plugin/main.js + - tests/cli/test_json_contracts.py +autonomous: true +requirements: + - CTRT-07 + - CTRT-08 + +must_haves: + truths: + - "paperforge dashboard --json returns PFResult with stats (papers, pdf_health, ocr_health, domain_counts)" + - "paperforge dashboard --json returns PFResult with permissions (can_sync, can_ocr, can_copy_context)" + - "Plugin reads dashboard data via paperforge dashboard --json CLI contract" + - "Plugin fallback to direct formal-library.json reading still works" + artifacts: + - path: "paperforge/commands/dashboard.py" + provides: "Dashboard CLI command with --json output" + exports: ["run"] + - path: "paperforge/commands/__init__.py" + provides: "dashboard entry in command registry" + - path: "paperforge/plugin/main.js" + provides: "Updated plugin _fetchStats to read dashboard --json" + key_links: + - from: "paperforge/commands/dashboard.py" + to: "paperforge/core/result.py" + via: "PFResult wrapping" + - from: "paperforge/commands/dashboard.py" + to: "paperforge/worker/asset_index.py" + via: "read_index and summarize_index" + - from: "paperforge/plugin/main.js" + to: "paperforge/commands/dashboard.py" + via: "execFile python -m paperforge dashboard --json" +--- + + +Implement `paperforge dashboard --json` command with stats (papers, pdf_health, ocr_health, domain_counts) and permissions (can_sync, can_ocr, can_copy_context). Update plugin to read dashboard data via CLI contract with fallback to direct index reading. + +Purpose: Expose machine-readable dashboard data in PFResult format for the Obsidian plugin and other consumers. +Output: New commands/dashboard.py module, CLI registration, plugin _fetchStats update, contract tests + + + +@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md +@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md + + + +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/057-contract-layer/057-CONTEXT.md + + +Per CONTEXT.md Dashboard Contract: +- Stats: papers (total count), pdf_health (healthy/broken/missing), ocr_health (pending/done/failed), domain_counts (dict of domain -> count) +- Permissions: can_sync (BBT exports exist), can_ocr (PaddleOCR configured), can_copy_context (system_dir writable) +- Version: from PFResult envelope (not duplicated in data) + +Existing plugin _fetchStats (main.js:338-430): +- Reads formal-library.json directly +- Falls back to `python -m paperforge status --json` +- Computes stats from index items (lifecycle, health, OCR counts) +- Needs to be updated to try dashboard --json first, then fallback + +Existing paperforge/worker/asset_index.py provides: +- read_index(vault) -> dict with {"items": [...], "paperforge_version": "..."} +- summarize_index(vault) -> dict with paper_count, lifecycle_level_counts, health_aggregate, maturity_distribution + +Existing config checks (from paperforge/worker/status.py for permissions): +- BBT exports: paths["exports"].glob("*.json") +- OCR token: os.environ.get("PADDLEOCR_API_TOKEN") +- System dir writable: os.access(paths["system"], os.W_OK) + + + + + + + Task 1: Create dashboard command with --json PFResult output + + paperforge/commands/dashboard.py + paperforge/commands/__init__.py + paperforge/cli.py + + + Step A: Create paperforge/commands/dashboard.py: + + ```python + """Dashboard command -- expose stats and permissions for the plugin UI.""" + from __future__ import annotations + + import argparse + import os + import logging + from pathlib import Path + + logger = logging.getLogger(__name__) + + + def _get_index_summary(vault: Path) -> dict | None: + """Read canonical index summary. Returns None if unavailable.""" + try: + from paperforge.worker.asset_index import summarize_index, read_index + summary = summarize_index(vault) + index_data = read_index(vault) + domain_counts: dict[str, int] = {} + if index_data and "items" in index_data: + for item in index_data["items"]: + domain = item.get("domain", "unknown") + domain_counts[domain] = domain_counts.get(domain, 0) + 1 + if summary: + summary["domain_counts"] = domain_counts + return summary + except Exception: + return None + + + def _check_permissions(vault: Path, cfg: dict) -> dict: + """Check system permissions for dashboard.""" + from paperforge.config import paperforge_paths + paths = paperforge_paths(vault, cfg) + exports_dir = paths.get("exports", vault / cfg["system_dir"] / "PaperForge" / "exports") + system_dir = paths.get("system", vault / cfg["system_dir"]) + + bbt_exports = sorted(exports_dir.glob("*.json")) if exports_dir.exists() else [] + can_sync = len(bbt_exports) > 0 + + ocr_token = os.environ.get("PADDLEOCR_API_TOKEN") or os.environ.get("PADDLEOCR_API_KEY") or "" + can_ocr = bool(ocr_token.strip()) + + can_copy_context = os.access(str(system_dir), os.W_OK) if system_dir.exists() else False + + return { + "can_sync": can_sync, + "can_ocr": can_ocr, + "can_copy_context": can_copy_context, + } + + + def run(args: argparse.Namespace) -> int: + """Run dashboard command.""" + vault = getattr(args, "vault_path", None) + if vault is None: + from paperforge.config import resolve_vault + vault = resolve_vault(cli_vault=getattr(args, "vault", None)) + + cfg = getattr(args, "cfg", None) + if cfg is None: + from paperforge.config import load_vault_config + cfg = load_vault_config(vault) + + from paperforge.config import paperforge_paths + paths = paperforge_paths(vault, cfg) + + json_output = getattr(args, "json_output", False) + + # Build dashboard data + summary = _get_index_summary(vault) + permissions = _check_permissions(vault, cfg) + + # Count papers from index or fallback to literature dir + papers = summary.get("paper_count", 0) if summary else 0 + + domain_counts = summary.get("domain_counts", {}) if summary else {} + + # PDF health from health_aggregate + health = summary.get("health_aggregate", {}) if summary else {} + pdf_health = health.get("pdf_health", {"healthy": 0, "unhealthy": 0}) + ocr_health_agg = health.get("ocr_health", {"healthy": 0, "unhealthy": 0}) + + # OCR breakdown from direct scan + ocr_total = 0 + ocr_pending = 0 + ocr_done = 0 + ocr_failed = 0 + if vault and paths.get("literature"): + import re + do_ocr_keys: set[str] = set() + for note_path in paths["literature"].rglob("*.md"): + if note_path.name in ("fulltext.md", "deep-reading.md", "discussion.md"): + continue + try: + text = note_path.read_text(encoding="utf-8") + if re.search(r"^do_ocr:\s*true\s*$", text, re.MULTILINE): + m = re.search(r"^zotero_key:\s*(\S+)", text, re.MULTILINE) + if m: + do_ocr_keys.add(m.group(1)) + except Exception: + continue + for key in do_ocr_keys: + ocr_total += 1 + meta_path = paths.get("ocr", vault / cfg["system_dir"] / "PaperForge" / "ocr") / key / "meta.json" + if not meta_path.exists(): + ocr_pending += 1 + continue + try: + from paperforge.worker._utils import read_json + meta = read_json(meta_path) + status = str(meta.get("ocr_status", "")).strip().lower() + if status == "done": + ocr_done += 1 + else: + ocr_failed += 1 + except Exception: + ocr_failed += 1 + + data = { + "papers": papers, + "pdf_health": { + "healthy": pdf_health.get("healthy", 0), + "broken": pdf_health.get("unhealthy", 0), + "missing": 0, + }, + "ocr_health": { + "total": ocr_total, + "pending": ocr_pending, + "done": ocr_done, + "failed": ocr_failed, + }, + "domain_counts": domain_counts, + "permissions": permissions, + } + + if json_output: + from paperforge.core.result import PFResult + from paperforge import __version__ + result = PFResult(ok=True, command="dashboard", version=__version__, data=data) + print(result.to_json()) + else: + print(f"Papers: {papers}") + print(f"PDF Health: {data['pdf_health']['healthy']} healthy, {data['pdf_health']['broken']} broken, {data['pdf_health']['missing']} missing") + print(f"OCR Health: {ocr_done}/{ocr_total} done (pending: {ocr_pending}, failed: {ocr_failed})") + print(f"Domains: {len(domain_counts)}") + print(f"Permissions: sync={'Yes' if permissions['can_sync'] else 'No'}, ocr={'Yes' if permissions['can_ocr'] else 'No'}, copy_context={'Yes' if permissions['can_copy_context'] else 'No'}") + + return 0 + ``` + + Step B: Register in paperforge/commands/__init__.py: + ```python + _COMMAND_REGISTRY: dict[str, str] = { + ... + "dashboard": "paperforge.commands.dashboard", + } + ``` + + Step C: Add subparser in paperforge/cli.py: + ```python + p_dashboard = sub.add_parser("dashboard", help="Show dashboard with stats and permissions") + p_dashboard.add_argument("--json", action="store_true", dest="json_output", help="Output JSON") + ``` + + Step D: Add dispatch in cli.py main(): + ```python + if args.command == "dashboard": + from paperforge.commands import dashboard + return dashboard.run(args) + ``` + Place this alongside the other command dispatch blocks (after the doctor block, before update). + + Per user decision (CONTEXT.md): Dashboard contract has stats (papers, pdf_health as healthy/broken/missing, ocr_health as pending/done/failed, domain_counts) and permissions (can_sync, can_ocr, can_copy_context). Version from PFResult envelope only. + + + python -m paperforge dashboard --json --vault C:\Users\Lin\AppData\Local\Temp\opencode 2>&1 || true + + dashboard --json produces PFResult with stats (papers, pdf_health, ocr_health, domain_counts) and permissions (can_sync, can_ocr, can_copy_context). Text output also works. + + + + Task 2: Update plugin to read dashboard via CLI contract + + paperforge/plugin/main.js + + + Modify the plugin's _fetchStats method (main.js:338-430) to support the PFResult dashboard contract: + + 1. Keep the existing direct index reading path (formal-library.json) as the PRIMARY path. + + 2. Add a new hybrid read strategy. After the existing try/catch for index reading fails (fallback path, line 404-428), add a SECOND fallback that calls `paperforge dashboard --json`: + ```javascript + // In the catch block (line 404), after the existing status --json fallback: + // Before the existing status --json fallback, try dashboard --json first + ``` + + Actually, modify the fallback order: + ``` + Primary: formal-library.json (existing) + Fallback 1: paperforge dashboard --json (new) + Fallback 2: paperforge status --json (existing) + ``` + + The _fetchStats function currently looks like: + ``` + try: + read formal-library.json directly -> compute stats -> _renderStats(d) + catch (err): + fallback to status --json -> _renderStats(d) + ``` + + Change to: + ``` + try: + read formal-library.json directly -> compute stats -> _renderStats(d) + catch (err): + try: + execFile python -m paperforge dashboard --json + d = JSON.parse(stdout) + // Check PFResult shape: use d.data if PFResult, or d directly if old format + const dashboardData = d.data || d; + this._cachedStats = { + version: d.version || d.version, + total_papers: dashboardData.papers || 0, + formal_notes: 0, + exports: 0, + bases: 0, + ocr: dashboardData.ocr_health || { total: 0, pending: 0, processing: 0, done: 0, failed: 0 }, + path_errors: 0, + lifecycle_level_counts: {}, + health_aggregate: { + pdf_health: { healthy: dashboardData.pdf_health?.healthy || 0, unhealthy: (dashboardData.pdf_health?.broken || 0) + (dashboardData.pdf_health?.missing || 0) }, + ocr_health: { healthy: dashboardData.ocr_health?.done || 0, unhealthy: (dashboardData.ocr_health?.pending || 0) + (dashboardData.ocr_health?.failed || 0) }, + note_health: { healthy: 0, unhealthy: 0 }, + asset_health: { healthy: 0, unhealthy: 0 }, + }, + }; + this._renderStats(this._cachedStats); + this._renderOcr(this._cachedStats); + catch (err2): + // Keep existing status --json fallback + execFile python -m paperforge status --json ... + ``` + + The key conversion map: + - dashboard.papers -> total_papers + - dashboard.pdf_health.healthy + broken/missing -> health_aggregate.pdf_health + - dashboard.ocr_health -> ocr stats (for _renderOcr) + - dashboard.permissions -> store for potential UI use (not rendered yet, but available) + + 3. Ensure the _renderStats function (line 500) can handle both formats: + - PFResult shaped: {ok, command, version, data: {papers, pdf_health, ...}} -- extract data sub-object + - Old status format: {total_papers, version, ocr, ...} -- use directly + The current _renderStats reads d.total_papers, d.version, d.ocr, etc. After the dashboard data is mapped to _cachedStats with the old shape keys (total_papers, ocr, etc.), _renderStats works unchanged. + + Per user decision: Plugin keeps fallback to direct index reading during PFResult transition -- removed after 2 stable release cycles. + + + node -e "console.log('Plugin logic verification: see test file')" + + Plugin _fetchStats falls back to dashboard --json when formal-library.json is unavailable. Old format reading still works. Status --json remains as final fallback. + + + + Task 3: Add CLI contract tests for dashboard --json + + tests/cli/test_json_contracts.py + + + In tests/cli/test_json_contracts.py, add TestDashboardJson class: + + - REQUIRED_KEYS = {"ok", "command", "version", "data", "error"} + - DATA_KEYS = {"papers", "pdf_health", "ocr_health", "domain_counts", "permissions"} + - PERM_KEYS = {"can_sync", "can_ocr", "can_copy_context"} + + Tests: + 1. test_dashboard_json_pfresult_shape: invokes dashboard --json, verifies PFResult envelope + 2. test_dashboard_json_data_keys: verifies data contains all required stat keys + 3. test_dashboard_json_permissions_shape: verifies permissions has can_sync/can_ocr/can_copy_context + 4. test_dashboard_json_no_vault: nonexistent vault exits non-zero + + Reuse existing assert_json_shape and assert_valid_json helpers. + + + python -m pytest tests/cli/test_json_contracts.py -v --tb=short -k "dashboard" 2>&1 || true + + Dashboard PFResult contract tests pass. All 4 dashboard test methods verify shape and content. + + + + + +- python -m paperforge dashboard --json produces valid PFResult JSON +- python -m pytest tests/cli/test_json_contracts.py -k "dashboard" passes +- Plugin can parse dashboard --json output via execFile +- Text output path works: python -m paperforge dashboard + + + +- [ ] dashboard command registered in cli.py and commands/__init__.py +- [ ] dashboard --json returns PFResult with stats (papers, pdf_health, ocr_health, domain_counts) +- [ ] dashboard --json returns PFResult with permissions (can_sync, can_ocr, can_copy_context) +- [ ] Plugin fallback reads dashboard --json when formal-library.json is unavailable +- [ ] Plugin fallback to status --json remains as final option +- [ ] Contract tests pass for dashboard + + + +After completion, create .planning/phases/057-contract-layer/057-04-SUMMARY.md + diff --git a/.planning/phases/057-contract-layer/057-CONTEXT.md b/.planning/phases/057-contract-layer/057-CONTEXT.md new file mode 100644 index 00000000..9239cd28 --- /dev/null +++ b/.planning/phases/057-contract-layer/057-CONTEXT.md @@ -0,0 +1,99 @@ +# Phase 57: Contract Layer - Context + +**Gathered:** 2026-05-09 +**Status:** Ready for planning + + +## Phase Boundary + +Stable JSON contracts (PFResult/PFError dataclasses, ErrorCode enum) that CLI commands produce and the plugin consumes — defining the machine-readable API surface consumed by SYNC, STAT, and SETP phases. Specifically: + +1. `paperforge/core/result.py` — PFResult and PFError dataclasses with JSON round-trip serialization +2. `paperforge/core/errors.py` — ErrorCode enum centralizing all error codes +3. CLI commands (`status --json`, `doctor --json`, `sync --json`, `ocr --diagnose --json`) wrapped in PFResult format +4. `dashboard --json` contract with stats and permissions for plugin consumption +5. Plugin reads dashboard via CLI; fallback to index reading during transition + + + +## Implementation Decisions + +### PFResult Contract Shape +- Top-level fields: `{ok, command, version, data, error}` following Google JSON Style Guide +- Version field: semver string from `__version__` (e.g., `"1.4.17rc3"`) +- Error shape: `{"code":"ERR_CODE","message":"Human text","details":{}}` — machine code + human message + optional payload +- `ok` field: `false` only when `error` is set; `true` when `data` has partial results + `warnings` sub-field + +### ErrorCode Enum +- Naming convention: `SCREAMING_SNAKE_CASE` (Python enum standard) +- Initial codes: `PYTHON_NOT_FOUND`, `VERSION_MISMATCH`, `BBT_EXPORT_NOT_FOUND`, `OCR_TOKEN_MISSING`, `SYNC_FAILED`, `VALIDATION_ERROR`, `INTERNAL_ERROR` +- `ErrorCode.UNKNOWN` fallback catch-all required +- Validation errors: single `VALIDATION_ERROR` with `details.field` for field-level granularity + +### Dashboard Contract +- Stats: `papers` (total count), `pdf_health` (healthy/broken/missing), `ocr_health` (pending/done/failed), `domain_counts` (dict of domain -> count) +- Permissions: `can_sync` (BBT exports exist), `can_ocr` (PaddleOCR configured), `can_copy_context` (system_dir writable) +- Version: from PFResult envelope (not duplicated in data) +- Transition: Fallback to direct `formal-library.json` reading allowed; removed after 2 release cycles of stable PFResult + +### Implementation Structure +- PFResult/PFError in `paperforge/core/result.py` +- ErrorCode in `paperforge/core/errors.py` +- CLI migration: wrap existing JSON output in PFResult envelope first (data field retains old shape), then refactor per-command +- Testing: round-trip unit tests for PFResult serialization + CLI contract tests asserting shape + +### the agent's Discretion +- Exact ErrorCode member names and descriptions (beyond the agreed categories) +- ErrorCode string values for serialization +- Dashboard field naming details in data payload +- CLI argument parsing for `--json` flag (follow existing patterns) + + + + +## Existing Code Insights + +### Reusable Assets +- `paperforge/worker/status.py:973-998` — existing `--json` output as flat dict (wrapping target) +- `paperforge/commands/status.py` — thin CLI dispatch layer +- `paperforge/commands/ocr.py` — `_diagnose()` and `run()` patterns +- `paperforge/worker/sync.py` — sync worker (JSON output pattern TBD) +- `paperforge/worker/status.py:206` — existing dependency check list with `"import":"yaml"` entries + +### Established Patterns +- CLI commands in `paperforge/commands/` with `run(args) -> int` signature +- Workers in `paperforge/worker/` with `run_status(vault, verbose, json_output)` pattern +- `json_output` flag passed to workers for format control +- `import json as _json` in worker code for JSON serialization +- Plugin reads from `formal-library.json` directly (transition away) + +### Integration Points +- All `--json` output paths need to be wrapped in PFResult +- Plugin `main.js` reads dashboard data — needs PFResult-aware parsing +- `paperforge/commands/` dispatch layer — where JSON output formatting should be centralized +- `paperforge/core/` — new module location for contract types +- CLI tests at `tests/cli/` — contract verification tests + + + + +## Specific Ideas + +### Prior Decision from REQUIREMENTS.md +- Dual JSON output during contract transition: old format + new PFResult side-by-side until 2 release cycles of stability +- Plugin keeps fallback to direct index reading during PFResult transition — removed after 2 stable release cycles + +### Phase 56 Dependency +- Phase 57 depends on Phase 56 (Stop the Bleeding) — version sync must be stable before contract layer +- Phase 56 created `scripts/check_version_sync.py` — version is now read from `__version__` consistently + + + + +## Deferred Ideas + +- ErrorCode auto-generation from field_registry.yaml (Phase 57 will) — deferred to Phase 59 +- Full JSON schema generation for PFResult (e.g., JSON Schema for external consumers) — out of scope for this phase +- Plugin PFResult parsing utilities (separate phase: plugin contract consumption) + +