docs(23-01): complete canonical-asset-index-safe-rebuilds plan

.planning/REQUIREMENTS.md

@@ -16,10 +16,10 @@ Requirements for milestone v1.6: AI-Ready Literature Asset Foundation.
 
 ### Canonical Asset Index
 
-- [ ] **ASSET-01**: User can rebuild a canonical literature asset index from existing library-records, OCR outputs, and formal notes without manual repair of the index file itself.
-- [ ] **ASSET-02**: User can rely on the canonical index to represent one paper as one unified asset record with stable identifiers, paths, provenance, and schema version.
+- [x] **ASSET-01**: User can rebuild a canonical literature asset index from existing library-records, OCR outputs, and formal notes without manual repair of the index file itself.
+- [x] **ASSET-02**: User can rely on the canonical index to represent one paper as one unified asset record with stable identifiers, paths, provenance, and schema version.
 - [ ] **ASSET-03**: User can refresh the canonical index incrementally after sync, OCR, deep-reading, or repair operations without corrupting existing data.
-- [ ] **ASSET-04**: User can recover safely from interrupted writes because canonical index updates are atomic and Windows-safe.
+- [x] **ASSET-04**: User can recover safely from interrupted writes because canonical index updates are atomic and Windows-safe.
 
 ### Lifecycle And Health
 
@@ -45,7 +45,7 @@ Requirements for milestone v1.6: AI-Ready Literature Asset Foundation.
 ### Brownfield Rollout
 
 - [ ] **MIG-01**: User can upgrade an existing PaperForge vault to v1.6 and detect stale or incompatible assets before they silently break dashboard or workflow behavior.
-- [ ] **MIG-02**: User can rebuild generated artifacts safely during migration without losing hand-authored notes or user intent fields.
+- [x] **MIG-02**: User can rebuild generated artifacts safely during migration without losing hand-authored notes or user intent fields.
 - [ ] **MIG-03**: User can run doctor and repair commands that explicitly identify migration issues in old configs, old index formats, old Base templates, or partial OCR assets.
 - [ ] **MIG-04**: User can recover from a failed migration with a documented, reversible rebuild path.
 
@@ -86,10 +86,10 @@ Which phases cover which requirements. Updated during roadmap creation.
 | CONF-02 | Phase 22 | Complete |
 | CONF-03 | Phase 22 | Complete |
 | CONF-04 | Phase 22 | Complete |
-| ASSET-01 | Phase 23 | Pending |
-| ASSET-02 | Phase 23 | Pending |
+| ASSET-01 | Phase 23 | Complete |
+| ASSET-02 | Phase 23 | Complete |
 | ASSET-03 | Phase 23 | Pending |
-| ASSET-04 | Phase 23 | Pending |
+| ASSET-04 | Phase 23 | Complete |
 | STATE-01 | Phase 24 | Pending |
 | STATE-02 | Phase 24 | Pending |
 | STATE-03 | Phase 24 | Pending |
@@ -103,7 +103,7 @@ Which phases cover which requirements. Updated during roadmap creation.
 | AIC-03 | Phase 26 | Pending |
 | AIC-04 | Phase 26 | Pending |
 | MIG-01 | Phase 25 | Pending |
-| MIG-02 | Phase 23 | Pending |
+| MIG-02 | Phase 23 | Complete |
 | MIG-03 | Phase 25 | Pending |
 | MIG-04 | Phase 25 | Pending |
 

.planning/ROADMAP.md

@@ -64,7 +64,12 @@ Plans:
   3. After `sync`, `ocr`, `deep-reading`, or `repair`, user can refresh the index incrementally and see affected papers update without unrelated asset records being corrupted.
   4. If an index write is interrupted, the previous readable index remains intact and user can rerun the rebuild safely on Windows.
   5. During migration or rebuild, generated artifacts can be regenerated without losing hand-authored notes or user intent fields.
-**Plans**: TBD
+**Plans**: 3 plans
+
+Plans:
+- [x] 23-01-PLAN.md — Core asset_index module: envelope format, atomic writes, filelock, extract build_index from sync.py
+- [ ] 23-02-PLAN.md — Legacy migration, incremental refresh by key, workspace path fields, --rebuild-index flag
+- [ ] 23-03-PLAN.md — Integration wiring: OCR, deep-reading, repair call incremental refresh + integration tests
 
 ### Phase 24: Derived Lifecycle, Health & Maturity
 **Goal**: Users can understand each paper's lifecycle state, health findings, maturity level, and next best action from source-derived evidence instead of hand-edited status flags.
@@ -109,7 +114,7 @@ Plans:
 | Phase | Plans Complete | Status | Completed |
 |-------|----------------|--------|-----------|
 | 22. Configuration Truth & Compatibility | 3/3 | Complete   | 2026-05-03 |
-| 23. Canonical Asset Index & Safe Rebuilds | 0/TBD | Not started | - |
+| 23. Canonical Asset Index & Safe Rebuilds | 0/3 | Planned   | - |
 | 24. Derived Lifecycle, Health & Maturity | 0/TBD | Not started | - |
 | 25. Surface Convergence, Doctor & Repair | 0/TBD | Not started | - |
 | 26. Traceable AI Context Packs | 0/TBD | Not started | - |

.planning/STATE.md

@@ -2,14 +2,14 @@
 gsd_state_version: 1.0
 milestone: v1.6
 milestone_name: AI-Ready Literature Asset Foundation
-status: Phase complete — ready for verification
-stopped_at: Completed 22-03-PLAN.md
-last_updated: "2026-05-03T15:13:06.843Z"
+status: Ready to execute
+stopped_at: Completed 23-01-PLAN.md
+last_updated: "2026-05-03T16:37:51.602Z"
 progress:
   total_phases: 5
   completed_phases: 1
-  total_plans: 4
-  completed_plans: 3
+  total_plans: 7
+  completed_plans: 4
 ---
 
 # Project State
@@ -19,12 +19,12 @@ progress:
 See: .planning/PROJECT.md (updated 2026-05-03)
 
 **Core value:** Researchers always know what papers they have, what state those papers are in, and whether each paper is reliably usable by AI with traceable fulltext, figures, notes, and source links.
-**Current focus:** Phase 22 — configuration-truth-compatibility
+**Current focus:** Phase 23 — canonical-asset-index-safe-rebuilds
 
 ## Current Position
 
-Phase: 22 (configuration-truth-compatibility) — EXECUTING
-Plan: 3 of 3
+Phase: 23 (canonical-asset-index-safe-rebuilds) — EXECUTING
+Plan: 2 of 3
 
 ## Performance Metrics
 
@@ -50,6 +50,7 @@ Plan: 3 of 3
 | Phase 22-configuration-truth-compatibility P01 | 4 min | 3 tasks | 3 files |
 | Phase 22-configuration-truth-compatibility P02 | 6 min | 3 tasks | 1 files |
 | Phase 22-configuration-truth-compatibility P03 | 8 min | 3 tasks | 3 files |
+| Phase 23-canonical-asset-index-safe-rebuilds P01 | 7 min | 3 tasks | 3 files |
 
 ## Accumulated Context
 
@@ -64,6 +65,8 @@ Recent decisions affecting current work:
 - [Phase 22-configuration-truth-compatibility]: schema_version is metadata excluded from load_vault_config() path config output; use get_paperforge_schema_version() instead
 - [Phase 22-configuration-truth-compatibility]: Added paddleocr_api_key and zotero_data_dir to DEFAULT_SETTINGS to prevent data loss from saveSettings() key filtering — Plan omitted these keys from DEFAULT_SETTINGS, but saveSettings() now filters persisted keys to only DEFAULT_SETTINGS entries - would have permanently deleted user API keys and Zotero paths
 - [Phase 22-configuration-truth-compatibility]: Clean dict replace replaces existing_config.update() to avoid accumulating stale top-level keys in setup wizard paperforge.json output
+- [Phase 23-canonical-asset-index-safe-rebuilds]: Lazy imports inside build_index avoid circular import between sync.py and asset_index.py
+- [Phase 23-canonical-asset-index-safe-rebuilds]: Orphaned-record cleanup stays in sync.py; only the core build loop moves to asset_index
 
 ### Pending Todos
 
@@ -76,6 +79,6 @@ None yet.
 
 ## Session Continuity
 
-Last session: 2026-05-03T15:13:04.067Z
-Stopped at: Completed 22-03-PLAN.md
+Last session: 2026-05-03T16:37:42.753Z
+Stopped at: Completed 23-01-PLAN.md
 Resume file: None

.planning/phases/23-canonical-asset-index-safe-rebuilds/23-01-SUMMARY.md

@@ -0,0 +1,136 @@
+---
+phase: 23-canonical-asset-index-safe-rebuilds
+plan: 01
+subsystem: worker
+tags: filelock, atomic-write, asset-index, canonical-index, sync-refactor
+
+# Dependency graph
+requires:
+  - phase: 22-configuration-truth-compatibility
+    provides: paperforge_paths resolver, schema_version handling
+provides:
+  - asset_index.py module with build_index, get_index_path, atomic_write_index
+  - Versioned envelope format for formal-library.json (schema_version, generated_at, paper_count)
+  - Atomic writes via tempfile.NamedTemporaryFile + os.replace
+  - Cross-process locking via filelock.FileLock (10s timeout)
+  - Delegation: run_index_refresh calls asset_index.build_index
+  - Tests (14) covering envelope, atomic write, locking, and empty-exports edge case
+affects:
+  - 23-02 canonical asset index incremental refresh
+  - 23-03 canonical asset index workspace paths
+
+# Tech tracking
+tech-stack:
+  added:
+    - filelock>=3.13.0 (cross-process file locking)
+  patterns:
+    - Index writes use tempfile + os.replace for atomicity
+    - Index writes use filelock for cross-process safety
+    - Lazy imports inside build_index to avoid circular deps
+
+key-files:
+  created:
+    - paperforge/worker/asset_index.py (index module)
+    - tests/test_asset_index.py (14 tests)
+  modified:
+    - paperforge/worker/sync.py (run_index_refresh delegates to build_index)
+    - pyproject.toml (filelock dependency — committed in prior session)
+
+key-decisions:
+  - "Index envelope uses CURRENT_SCHEMA_VERSION = '2' matching config.py schema_version"
+  - "Lazy imports inside build_index avoid circular import between sync.py and asset_index.py"
+  - "Orphaned-record cleanup stays in sync.py; only the core build loop moves to asset_index"
+  - "atomic_write_index uses path.with_suffix('.json.lock') as lock file, creating lock alongside index"
+
+patterns-established:
+  - "New modules that import from sync.py use lazy imports to avoid circular deps"
+  - "Atomic writes: tempfile in same directory + os.replace + filelock"
+  - "Index envelope format: {schema_version, generated_at, paper_count, items}"
+
+requirements-completed:
+  - ASSET-01
+  - ASSET-02
+  - ASSET-04
+  - MIG-02
+
+# Metrics
+duration: 7 min
+completed: 2026-05-04
+---
+
+# Phase 23 Plan 01: Canonical Asset Index & Safe Rebuilds Summary
+
+**Extracted index generation from sync.py into asset_index.py with versioned envelope format, atomic writes (tempfile + os.replace + filelock), and 14 passing tests**
+
+## Performance
+
+- **Duration:** 7 min (Tasks 2-3 in current session; Task 1 committed in prior session)
+- **Started:** 2026-05-04T00:29:08+08:00 (Task 1 from prior session)
+- **Completed:** 2026-05-04T00:36:44Z
+- **Tasks:** 3 (1 prior, 2 current)
+- **Files created:** 2
+- **Files modified:** 1
+
+## Accomplishments
+
+- Created `paperforge/worker/asset_index.py` — the single canonical home for asset index generation
+  - `get_index_path(vault)` — resolves the index file location
+  - `build_envelope(items)` — wraps items in versioned envelope (schema_version="2", generated_at, paper_count)
+  - `atomic_write_index(path, data)` — writes atomically via tempfile + os.replace + filelock (10s timeout)
+  - `build_index(vault, verbose)` — full rebuild extracted from sync.py's run_index_refresh loop
+- Modified `sync.py` `run_index_refresh()` to delegate to `asset_index.build_index()` — orphaned-record cleanup stays in sync.py
+- Created `tests/test_asset_index.py` with 14 tests covering envelope format, atomic writes, lock timeout, empty-exports edge cases
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1: Add filelock dependency** — `a2c6c8b` (chore) — *committed in prior session*
+2. **Task 2: Create asset_index.py module** — `87437cf` (feat)
+3. **Task 3: Write tests for asset_index.py** — `ccd7903` (test)
+
+**Plan metadata:** *(committed below)*
+
+## Files Created/Modified
+
+- `paperforge/worker/asset_index.py` (269 lines) — Canonical index module with envelope, atomic writes, locking, and build_index
+- `paperforge/worker/sync.py` (modified) — `run_index_refresh()` delegates to `asset_index.build_index()`
+- `tests/test_asset_index.py` (261 lines) — 14 tests for core functionality
+- `pyproject.toml` — `filelock>=3.13.0` added (Task 1, committed in prior session)
+
+## Decisions Made
+
+- **Lazy imports in build_index:** Functions imported from `sync.py` (collection_fields, frontmatter_note, etc.) are imported inside `build_index()` via lazy imports to avoid circular dependency — `sync.py` imports `asset_index` at module level.
+- **Orphaned cleanup stays in sync.py:** The orphaned-record cleanup logic (lines 1689-1733 of the modified file) remains in `run_index_refresh()` because it's tightly coupled to the `exports` dict and domain resolution that happens before the build_index call.
+- **Lock file naming:** `atomic_write_index` uses `path.with_suffix('.json.lock')` as the lock file path, creating it as a sibling to the target index file.
+
+## Deviations from Plan
+
+None - plan executed exactly as written.
+
+## Issues Encountered
+
+- `datetime.utcnow()` in `build_envelope()` triggers a Python 3.14 deprecation warning (prefer `datetime.now(datetime.UTC)`). This is cosmetic — functionality is correct. Deferred to a future cleanup pass as it's not part of the current task scope.
+
+## User Setup Required
+
+None - no external service configuration required.
+
+## Next Phase Readiness
+
+- Ready for Plan 23-02 (incremental refresh with key-based update)
+- `asset_index.py` now exists as a stable module to extend with `refresh_index_entry()`
+- All 14 tests pass, verifying envelope format, atomic writes, and lock behavior
+
+---
+
+*Phase: 23-canonical-asset-index-safe-rebuilds*
+*Completed: 2026-05-04*
+
+## Self-Check: PASSED
+
+- [x] `paperforge/worker/asset_index.py` exists
+- [x] `tests/test_asset_index.py` exists
+- [x] Commit `a2c6c8b` — Task 1: filelock dependency
+- [x] Commit `87437cf` — Task 2: asset_index.py module
+- [x] Commit `ccd7903` — Task 3: tests