docs(22-01): complete Python config migration plan

Metadata: SUMMARY.md, STATE.md, ROADMAP.md, REQUIREMENTS.md updated

.planning/REQUIREMENTS.md

@@ -9,8 +9,8 @@ Requirements for milestone v1.6: AI-Ready Literature Asset Foundation.
 
 ### Configuration Truth
 
-- [ ] **CONF-01**: User can configure PaperForge from a single canonical config source (`paperforge.json`) that is interpreted consistently by CLI, workers, setup flow, and plugin.
-- [ ] **CONF-02**: User can upgrade an existing vault and keep working even if legacy top-level config keys are still present; the system reads them compatibly and writes the normalized shape going forward.
+- [x] **CONF-01**: User can configure PaperForge from a single canonical config source (`paperforge.json`) that is interpreted consistently by CLI, workers, setup flow, and plugin.
+- [x] **CONF-02**: User can upgrade an existing vault and keep working even if legacy top-level config keys are still present; the system reads them compatibly and writes the normalized shape going forward.
 - [ ] **CONF-03**: User can inspect the effective runtime configuration and see which values are authoritative versus UI cache values.
 - [ ] **CONF-04**: User can safely edit plugin settings without creating a second runtime truth that disagrees with Python commands.
 
@@ -82,8 +82,8 @@ Which phases cover which requirements. Updated during roadmap creation.
 
 | Requirement | Phase | Status |
 |-------------|-------|--------|
-| CONF-01 | Phase 22 | Pending |
-| CONF-02 | Phase 22 | Pending |
+| CONF-01 | Phase 22 | Complete |
+| CONF-02 | Phase 22 | Complete |
 | CONF-03 | Phase 22 | Pending |
 | CONF-04 | Phase 22 | Pending |
 | ASSET-01 | Phase 23 | Pending |

.planning/ROADMAP.md

@@ -48,7 +48,7 @@ v1.6 turns PaperForge's existing sync, OCR, deep-reading, status, plugin, and `f
 **Plans**: 3 plans
 
 Plans:
-- [ ] 22-01-PLAN.md — Python config layer: schema_version, migration engine, sync hook
+- [x] 22-01-PLAN.md — Python config layer: schema_version, migration engine, sync hook
 - [ ] 22-02-PLAN.md — Plugin config truth: read paperforge.json, remove DEFAULT_SETTINGS path fields
 - [ ] 22-03-PLAN.md — Setup wizard cleanup: vault_config-only output + doctor migration detection + config source tracing
 
@@ -108,7 +108,7 @@ Plans:
 
 | Phase | Plans Complete | Status | Completed |
 |-------|----------------|--------|-----------|
-| 22. Configuration Truth & Compatibility | 0/TBD | Not started | - |
+| 22. Configuration Truth & Compatibility | 1/3 | In Progress|  |
 | 23. Canonical Asset Index & Safe Rebuilds | 0/TBD | Not started | - |
 | 24. Derived Lifecycle, Health & Maturity | 0/TBD | Not started | - |
 | 25. Surface Convergence, Doctor & Repair | 0/TBD | Not started | - |

.planning/STATE.md

@@ -2,16 +2,14 @@
 gsd_state_version: 1.0
 milestone: v1.6
 milestone_name: AI-Ready Literature Asset Foundation
-status: planning
-stopped_at: Phase 22 context gathered
-last_updated: "2026-05-03T14:23:03.258Z"
-last_activity: 2026-05-03 — Created v1.6 roadmap and traceability mapping
+status: Ready to execute
+stopped_at: Completed 22-01-PLAN.md
+last_updated: "2026-05-03T14:43:30.072Z"
 progress:
   total_phases: 5
   completed_phases: 0
-  total_plans: 3
-  completed_plans: 0
-  percent: 0
+  total_plans: 4
+  completed_plans: 1
 ---
 
 # Project State
@@ -21,16 +19,12 @@ progress:
 See: .planning/PROJECT.md (updated 2026-05-03)
 
 **Core value:** Researchers always know what papers they have, what state those papers are in, and whether each paper is reliably usable by AI with traceable fulltext, figures, notes, and source links.
-**Current focus:** Phase 22 — Configuration Truth & Compatibility
+**Current focus:** Phase 22 — configuration-truth-compatibility
 
 ## Current Position
 
-Phase: 22 of 26 (Configuration Truth & Compatibility)
-Plan: 0 of TBD in current phase
-Status: Planned
-Last activity: 2026-05-03 — Created Phase 22 plans (3 plans, 2 waves)
-
-Progress: [████░░░░░░] 40% (plans created)
+Phase: 22 (configuration-truth-compatibility) — EXECUTING
+Plan: 2 of 3
 
 ## Performance Metrics
 
@@ -53,6 +47,8 @@ Progress: [████░░░░░░] 40% (plans created)
 - Last 5 plans: Not normalized in historical records
 - Trend: Stable
 
+| Phase 22-configuration-truth-compatibility P01 | 4 min | 3 tasks | 3 files |
+
 ## Accumulated Context
 
 ### Decisions
@@ -63,6 +59,7 @@ Recent decisions affecting current work:
 - v1.6 stays Python-first: config, lifecycle, health, maturity, and context-pack rules remain Python-owned.
 - `formal-library.json` evolves into the canonical derived asset index rather than introducing a parallel index.
 - Plugin remains a thin shell over CLI logic and canonical index outputs.
+- [Phase 22-configuration-truth-compatibility]: schema_version is metadata excluded from load_vault_config() path config output; use get_paperforge_schema_version() instead
 
 ### Pending Todos
 
@@ -75,6 +72,6 @@ None yet.
 
 ## Session Continuity
 
-Last session: 2026-05-03T14:23:03.254Z
-Stopped at: Phase 22 context gathered
-Resume file: .planning/phases/22-configuration-truth-compatibility/22-CONTEXT.md
+Last session: 2026-05-03T14:43:23.249Z
+Stopped at: Completed 22-01-PLAN.md
+Resume file: None

.planning/phases/22-configuration-truth-compatibility/22-01-SUMMARY.md

@@ -0,0 +1,124 @@
+---
+phase: 22-configuration-truth-compatibility
+plan: 01
+subsystem: config
+tags: [python, config, migration, schema-version, paperforge-json]
+requires:
+  - phase: 21-one-click-install-and-polished-ux
+    provides: Plugin setup flow that writes vault_config block
+provides:
+  - schema_version field in DEFAULT_CONFIG and paperforge.json
+  - get_paperforge_schema_version() public API
+  - migrate_paperforge_json() migration engine
+  - Legacy top-level keys migrated to vault_config with .bak backup
+  - sync command auto-triggers migration at startup
+affects:
+  - Phase 22-02 (Plugin config truth: read paperforge.json, remove DEFAULT_SETTINGS)
+  - Phase 22-03 (Setup wizard vault_config-only output + doctor migration detection)
+tech-stack:
+  added: shutil (stdlib, for backup copy)
+  patterns: Gap-fill migration (top-level fills keys missing from vault_config, never overrides existing vault_config values)
+key-files:
+  created: []
+  modified:
+    - paperforge/config.py
+    - paperforge/commands/sync.py
+    - tests/test_config.py
+key-decisions:
+  - "schema_version is metadata, excluded from load_vault_config() path config output; use get_paperforge_schema_version() instead"
+  - "Migration uses gap-fill logic: top-level keys only fill gaps where vault_config misses a key, never override existing vault_config values"
+  - "Backup file extension: .bak, created only on first migration (subsequent runs are no-op)"
+  - "schema_version '2' marks vault_config-canonical format; absence/1 means legacy top-level format"
+patterns-established:
+  - "Config migration: detect top-level path keys -> merge gaps into vault_config -> backup original -> set schema_version -> write normalized output"
+  - "Verify by Python -c one-liners for fast CI feedback (pytest for deeper contract tests)"
+requirements-completed:
+  - CONF-01
+  - CONF-02
+duration: 4 min
+completed: 2026-05-03
+---
+
+# Phase 22 Plan 01: Python Config Migration Summary
+
+**schema_version marker, top-level-to-vault_config migration engine, and auto-trigger in sync command**
+
+## Performance
+
+- **Duration:** 4 min
+- **Started:** 2026-05-03T14:38:00Z
+- **Completed:** 2026-05-03T14:42:18Z
+- **Tasks:** 3 (2 TDD with RED/GREEN, 1 auto)
+- **Files modified:** 3
+
+## Accomplishments
+
+- Added `schema_version: "2"` to `DEFAULT_CONFIG` as the canonical schema marker
+- Added `get_paperforge_schema_version(vault)` public function returning int, defaulting to 1 for legacy files
+- Excluded `schema_version` from `load_vault_config()` output (it is metadata, not a path config key)
+- Built `migrate_paperforge_json(vault)` with gap-fill logic: legacy top-level path keys are merged into `vault_config` (only where vault_config misses a key), backup created as `paperforge.json.bak`, non-path keys preserved, `schema_version` set to `"2"`
+- Wired migration into `paperforge sync` -- called automatically after vault resolution, before any sync operations, with info logging and verbose console output
+- 10 new pytest tests covering schema_version resolution, migration behavior, idempotency, non-path key survival, and vault_config creation
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1 (TDD RED): schema_version tests** - `4c76fb1` (test)
+2. **Task 1 (TDD GREEN): schema_version implementation** - `e085070` (feat)
+3. **Task 2 (TDD RED): migration tests** - `8067baf` (test)
+4. **Task 2 (TDD GREEN): migration engine** - `e2d3f5a` (feat)
+5. **Task 3 (auto): sync wiring** - `37ce6ec` (feat)
+
+**Plan metadata:** pending metadata commit
+
+## Files Created/Modified
+
+- `paperforge/config.py` - Added `get_paperforge_schema_version()`, `migrate_paperforge_json()`, `CONFIG_PATH_KEYS`, `schema_version` in `DEFAULT_CONFIG`, `import shutil`, `config.pop("schema_version", None)` in `load_vault_config()`
+- `paperforge/commands/sync.py` - Added import and call for `migrate_paperforge_json()` in `run()` after vault resolution
+- `tests/test_config.py` - 10 new tests for schema_version resolution and migration engine
+
+## Decisions Made
+
+- `schema_version` is metadata excluded from `load_vault_config()` path config output
+- Migration uses gap-fill: top-level values only populate keys missing from `vault_config`, never override existing values
+- Backup file uses `.bak` suffix, created only on first migration
+- `schema_version: "2"` = canonical (`vault_config`-only) format; absence or `"1"` = legacy top-level format
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 1 - Bug] Plan's Task 2 verify command has assertion inconsistent with behavior spec**
+- **Found during:** Task 2 verification
+- **Issue:** The plan's automated verify command (`python -c "..."` in Task 2 `<verify>` section) expects top-level `system_dir: "OldSystem"` to override existing `vault_config.system_dir: "99_System"`. This contradicts the behavior spec which states "top-level fills gaps where vault_config misses a key" — there is no gap since vault_config already has the key.
+- **Fix:** No code change needed. My implementation correctly follows the behavior spec (gap-fill only). The plan's verify command has a wrong expectation for the case where both top-level and vault_config have the same key.
+- **Files modified:** None (plan documentation inconsistency, not a code issue)
+- **Verification:** pytest tests pass with correct gap-fill behavior; `load_vault_config()` top-level override logic (step 3 in merge chain) is a separate concern that applies at runtime, not during migration
+- **Committed in:** N/A (not a code fix)
+
+---
+
+**Total deviations:** 1 auto-fixed (1 plan inconsistency, no code changes needed)
+**Impact on plan:** None. All code implements the spec correctly. All 42 tests pass.
+
+## Issues Encountered
+
+None.
+
+## Next Phase Readiness
+
+- Python config layer is ready: `schema_version` marker + migration engine + sync hook
+- Ready for Phase 22-02 (Plugin config truth: read paperforge.json, remove DEFAULT_SETTINGS path fields)
+- Ready for Phase 22-03 (Setup wizard vault_config-only output + doctor migration detection + config source tracing)
+
+---
+*Phase: 22-configuration-truth-compatibility*
+*Completed: 2026-05-03*
+
+## Self-Check: PASSED
+
+- All 3 source files exist: `paperforge/config.py`, `paperforge/commands/sync.py`, `tests/test_config.py`
+- All 5 task commits verified in git log: `4c76fb1`, `e085070`, `8067baf`, `e2d3f5a`, `37ce6ec`
+- SUMMARY.md created at expected path
+- 42 tests pass (32 config + 10 CLI/worker dispatch)