---
phase: 23-canonical-asset-index-safe-rebuilds
plan: 02
type: execute
wave: 2
depends_on:
  - 23-01
files_modified:
  - paperforge/worker/asset_index.py
  - paperforge/worker/sync.py
  - paperforge/cli.py
  - tests/test_asset_index.py
autonomous: true
requirements:
  - ASSET-01
  - ASSET-02
  - ASSET-03
  - ASSET-04

must_haves:
  truths:
    - "Legacy bare-list formal-library.json is detected on read, backed up as .bak, and transparently migrated to envelope format"
    - "refresh_index_entry(vault, key) reads existing index, updates the single matching entry, writes back atomically"
    - "run_index_refresh detects schema_version mismatch and auto-triggers full rebuild instead of incremental"
    - "Workspace path fields (paper_root, main_note_path, fulltext_path, deep_reading_path, ai_path) appear in each index entry"
    - "Path fields are mirrored in the formal note frontmatter for Base view compatibility"
    - "--rebuild-index flag on `paperforge sync` forces full rebuild"
  artifacts:
    - path: "paperforge/worker/asset_index.py"
      provides: "refresh_index_entry, legacy migration, workspace paths"
      exports: "refresh_index_entry, read_index"
    - path: "paperforge/worker/sync.py"
      provides: "--rebuild-index flag handling, schema_version check"
    - path: "paperforge/cli.py"
      provides: "--rebuild-index CLI argument"
    - path: "tests/test_asset_index.py"
      provides: "Tests for incremental refresh, migration, workspace paths"
      min_lines: 150
  key_links:
    - from: "asset_index.py :: build_index()"
      to: "index envelope :: schema_version"
      via: "auto-detect legacy: no schema_version key -> migrate before rebuild"
    - from: "asset_index.py :: refresh_index_entry(vault, key)"
      to: "index file :: items[]"
      via: "match zotero_key, rebuild single entry in-place, atomic write"
    - from: "sync.py :: run_index_refresh()"
      to: "asset_index.py :: refresh_index_entry()"
      via: "incremental path when selected_keys is provided"
    - from: "cli.py :: sync command"
      to: "sync.py :: run_index_refresh(rebuild=True)"
      via: "--rebuild-index flag"

---

<objective>
Add legacy format migration, incremental refresh by key, workspace path fields, and the --rebuild-index CLI flag to the asset index system.

Purpose: Users with existing bare-list formal-library.json files get transparent migration. After sync/ocr/deep-reading, only changed entries are refreshed (faster, safer). Workspace-style path fields (matching Phase 22 decisions) are added to each entry and mirrored in note frontmatter for Base view consumption.

Output: Extended `asset_index.py` with `refresh_index_entry()`, `read_index()`, legacy migration, workspace path fields + cli.py `--rebuild-index` flag + tests.
</output>
</objective>

<execution_context>
@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md
</execution_context>

<context>
@.planning/STATE.md
@.planning/ROADMAP.md (Phase 23)
@.planning/phases/23-canonical-asset-index-safe-rebuilds/23-CONTEXT.md (D-04, D-08..D-14, D-18..D-19)
@.planning/phases/23-canonical-asset-index-safe-rebuilds/23-01-PLAN.md (existing asset_index.py module)
@paperforge/worker/asset_index.py (just created in Plan 01)
@paperforge/worker/sync.py (lines 1669-1792 -- current run_index_refresh)
@paperforge/worker/_utils.py (lines 241-265 -- pipeline_paths paths)
@paperforge/cli.py (sync command registration)
@paperforge/config.py (lines 105-112 -- get_paperforge_schema_version)
</context>

<tasks>

<task type="auto">
  <name>Task 1: Legacy format detection, backup, and auto-migration in build_index</name>
  <files>paperforge/worker/asset_index.py</files>

  <read_first>
  @paperforge/worker/asset_index.py (full file — Plan 01 output)
  @.planning/phases/23-canonical-asset-index-safe-rebuilds/23-CONTEXT.md (D-04, D-18, D-19)
  </read_first>

  <action>
  Add the following functions to `paperforge/worker/asset_index.py`:

  **Function: `read_index(vault: Path) -> dict | list | None`**
  - Returns `None` if the index file does not exist
  - Reads and parses the JSON at `get_index_path(vault)`
  - Returns the raw parsed data (could be dict with envelope, or list for old bare-list format)

  **Function: `is_legacy_format(data) -> bool`**
  - Returns `True` if `data` is a `list` (old bare-list format before v1.6)
  - Returns `False` if `data` is a `dict` with `"schema_version"` key

  **Function: `migrate_legacy_index(vault: Path) -> bool`**
  - Called by `build_index()` before the rebuild loop
  - Reads the existing index via `read_index()`
  - If the format is legacy (list):
    1. Back up the legacy file: copy to `formal-library.json.bak` (same directory, same name + `.bak`)
    2. Log: `f"Legacy format detected at {path}, backed up to {path}.bak"`
    3. Return True (migration pending — will be overwritten by build_index)
  - If already envelope format, return False (no-op)
  - If file doesn't exist, return False (fresh build)
  - If file doesn't exist or is already envelope: no action, return False

  **Modify `build_index()`:**
  - At the start of `build_index()` (after `ensure_base_views` but before the export loop), call:
    ```python
    migrated = migrate_legacy_index(vault)
    if migrated:
        print("Legacy index format detected and backed up. Rebuilding with envelope...")
    ```
  - Otherwise the rebuild loop naturally writes the new envelope format

  **Edge cases:**
  - No existing index file: start fresh (no backup needed, no migration)
  - Already envelope format: no action
  - Backup file already exists: overwrite with current content (idempotent — last backup is most recent state)
  - Corrupt JSON: treat as missing file (start fresh, log warning)
  </action>

  <verify>
  <automated>python -c "from paperforge.worker.asset_index import read_index, is_legacy_format, migrate_legacy_index; print('OK')"</automated>
  </verify>

  <acceptance_criteria>
  - `read_index`, `is_legacy_format`, `migrate_legacy_index` are exported from asset_index
  - `is_legacy_format([])` returns True (list = legacy)
  - `is_legacy_format({"schema_version": "2", "items": []})` returns False
  - `migrate_legacy_index` creates `.bak` copy when legacy format detected (verified via manual test)
  - `build_index()` calls `migrate_legacy_index()` at start (grep for call in build_index)
  </acceptance_criteria>

  <done>Legacy format detection, backup, and auto-migration added to asset_index.py</done>
</task>

<task type="auto">
  <name>Task 2: Implement incremental refresh (refresh_index_entry) and schema_version check</name>
  <files>paperforge/worker/asset_index.py</files>

  <read_first>
  @paperforge/worker/asset_index.py (full file — includes Task 1 additions)
  @paperforge/worker/sync.py (lines 1686-1744 — entry construction logic, shared with build_index)
  @.planning/phases/23-canonical-asset-index-safe-rebuilds/23-CONTEXT.md (D-08..D-11)
  </read_first>

  <action>
  Add the following function to `paperforge/worker/asset_index.py`:

  **Function: `refresh_index_entry(vault: Path, key: str) -> bool`**
  - Reads the existing index via `read_index()`
  - If the index is in legacy format, calls `build_index(vault)` for full rebuild (returns False to indicate full rebuild happened)
  - If the index is envelope format:
    1. Reads the `items` list
    2. Finds the entry where `item["zotero_key"] == key`
    3. If found: rebuilds that single entry (using the same entry-construction logic as `build_index()` — shared helper), replaces it in the items list, preserves all other entries unchanged
    4. If not found: could be a new paper; rebuilds the entry and appends it
    5. Calls `atomic_write_index(path, build_envelope(items))` to write back
    6. Returns True (incremental refresh done)
  - **Shared helper approach**: Extract the entry-construction code from `build_index()` into a helper function:
    ```python
    def _build_entry(item: dict, vault: Path, paths: dict, **kwargs) -> dict:
        """Construct a single index entry from a BBT export item."""
    ```
    This is used by both `build_index()` (loop) and `refresh_index_entry()` (single).
  - The helper `_build_entry` should accept the same parameters and return the same dict shape as the current entry construction in `build_index()`
  - The `key` parameter is the Zotero citation key (8-char alphanumeric)

  **Add to `build_index()`:**
  - At the start, add a `schema_version` check:
    ```python
    # Check if existing index has matching schema_version
    existing = read_index(vault)
    if isinstance(existing, dict) and existing.get("schema_version") != CURRENT_SCHEMA_VERSION:
        print(f"Schema version mismatch: index has {existing.get('schema_version')}, need {CURRENT_SCHEMA_VERSION}. Rebuilding...")
        # Continue with full rebuild (no early return)
    ```

  **Edge cases:**
  - Index file does not exist yet: cannot refresh — delegate to `build_index()`
  - `key` not found in items: rebuild entry from scratch and append (handles newly-synced papers)
  - `key` found but export data missing (deleted from Zotero): log warning and skip
  - Race condition: lock inside `atomic_write_index` protects concurrent writes
  - Empty items list and key not found: add as single entry
  </action>

  <verify>
  <automated>python -c "from paperforge.worker.asset_index import refresh_index_entry, _build_entry; print('OK')"</automated>
  </verify>

  <acceptance_criteria>
  - `refresh_index_entry(vault, key)` is exported from asset_index
  - `_build_entry(item, vault, paths)` is a callable helper that returns a dict with same shape as current entry construction
  - `refresh_index_entry` handles legacy format by falling back to `build_index`
  - `refresh_index_entry` preserves non-matching entries in the items list (only the targeted key changes)
  - Schema mismatch check in `build_index` compares existing schema_version vs CURRENT_SCHEMA_VERSION
  </acceptance_criteria>

  <done>refresh_index_entry and schema_version check implemented</done>
</task>

<task type="auto">
  <name>Task 3: Add workspace path fields to index entries, mirror in frontmatter, wire --rebuild-index flag</name>
  <files>
    paperforge/worker/asset_index.py
    paperforge/worker/sync.py
    paperforge/cli.py
  </files>

  <read_first>
  @paperforge/worker/asset_index.py (full file — includes Tasks 1-2)
  @paperforge/worker/sync.py (lines 1711-1740 — current entry dict construction, frontmatter_note function)
  @paperforge/cli.py (sync command definition)
  @.planning/phases/23-canonical-asset-index-safe-rebuilds/23-CONTEXT.md (D-12..D-14)
  @.planning/phases/22-configuration-truth-compatibility/22-CONTEXT.md (D-01..D-05 — paper workspace layout)
  </read_first>

  <action>
  **A. Add workspace path fields to `_build_entry()` in asset_index.py:**

  After the existing entry fields (zotero_key, domain, title, etc., deep_reading_md_path), add these workspace path fields (per D-12):

  ```python
  # Workspace paths (Phase 22 paper workspace structure)
  "paper_root": f"Literature/{domain}/{key} - {title_slug}/",
  "main_note_path": f"Literature/{domain}/{key} - {title_slug}/{key} - {title_slug}.md",
  "fulltext_path": f"Literature/{domain}/{key} - {title_slug}/fulltext.md",
  "deep_reading_path": f"Literature/{domain}/{key} - {title_slug}/deep-reading.md",
  "ai_path": f"Literature/{domain}/{key} - {title_slug}/ai/",
  ```

  - All paths are relative to vault root, using forward slashes (consistent with existing `note_path` format)
  - The `title_slug` comes from `slugify_filename(item["title"])` (already computed in _build_entry)

  **B. Keep existing system paths** (D-13): Do NOT remove `ocr_path`, `meta_path`, `note_path`, `deep_reading_md_path` — these remain for backward compatibility.

  **C. Mirror path fields in note frontmatter** (D-14):

  In the `_build_entry()` function, the existing code writes the formal note via `frontmatter_note()`. The path fields need to be passed through to the note frontmatter as YAML properties:

  - Locate where `frontmatter_note(entry, existing_text)` is called (currently in the `build_index` loop and also in `refresh_index_entry`)
  - The `frontmatter_note()` function already generates YAML frontmatter from the entry dict
  - Adding the new path fields (`paper_root`, `main_note_path`, etc.) to the entry dict means they automatically become part of the frontmatter that Base views read
  - Verify this by examining `frontmatter_note()` in sync.py — it should iterate entry keys for frontmatter fields

  **D. Add --rebuild-index flag to cli.py sync command** (D-10):

  In `paperforge/cli.py`, find the sync command definition and add:
  ```python
  @click.option("--rebuild-index", is_flag=True, help="Force full rebuild of the canonical asset index")
  ```

  Then in the sync command handler, pass `rebuild_index` to `run_index_refresh()`.

  In `sync.py` `run_index_refresh()`, accept the `rebuild_index: bool = False` parameter:
  - If True: skip incremental path, always call `asset_index.build_index(vault, verbose)`
  - If False: use existing behavior (currently always full rebuild; will add incremental check later)

  **E. Auto-rebuild on schema_version mismatch** (D-10):

  Already handled in Task 2 within `build_index()` — schema_version mismatch detection triggers full rebuild.

  **Edge cases:**
  - `title_slug` may differ from existing directory name if the title changed; this is consistent with current behavior where stale dirs are cleaned up
  - Path fields should be consistent across full rebuild and incremental refresh (both use `_build_entry`)
  - Frontmatter mirroring: the fields MUST appear in the YAML frontmatter of the formal note so that Base views (which read frontmatter) can use them
  </action>

  <verify>
  <automated>python -c "from paperforge.worker.asset_index import _build_entry; e = _build_entry({'key':'abc','title':'Test','domain':'test'}, None, None); assert 'paper_root' in e; assert 'main_note_path' in e; assert 'fulltext_path' in e; assert 'deep_reading_path' in e; assert 'ai_path' in e; print('OK')"</automated>
  </verify>

  <acceptance_criteria>
  - `_build_entry` result dict contains `paper_root`, `main_note_path`, `fulltext_path`, `deep_reading_path`, `ai_path` keys
  - All path values use forward slashes and are relative to vault root
  - Existing system paths (`ocr_path`, `meta_path`, `note_path`, `deep_reading_md_path`) are still present (removed in error if missing)
  - `frontmatter_note` in sync.py includes the new path fields (check if frontmatter_note uses all entry keys)
  - `cli.py` sync command has `--rebuild-index` option
  - `sync.py` `run_index_refresh()` accepts `rebuild_index` parameter
  </acceptance_criteria>

  <done>Workspace path fields added, frontmatter mirrored, --rebuild-index flag wired</done>
</task>

</tasks>

<verification>
1. `python -c "from paperforge.worker.asset_index import read_index, is_legacy_format, migrate_legacy_index, refresh_index_entry, _build_entry; print('OK')"` — all new exports available
2. `python -c "e = {'key':'abc','title':'Test','domain':'test'}; from paperforge.worker.asset_index import _build_entry as b; r = b(e, None, None); assert all(k in r for k in ['paper_root','main_note_path','fulltext_path','deep_reading_path','ai_path'])"` — path fields present
3. `grep 'rebuild-index' paperforge/cli.py` — CLI flag defined
4. `grep 'rebuild_index' paperforge/worker/sync.py` — parameter accepted
5. `python -c "from paperforge.worker.asset_index import is_legacy_format; assert is_legacy_format([]) == True; assert is_legacy_format({'schema_version':'2'}) == False"` — legacy detection works
6. `grep -n '\.bak' paperforge/worker/asset_index.py` — migration creates backup
</verification>

<success_criteria>
- Legacy bare-list formal-library.json is detected and backed up as .bak before transparent migration
- refresh_index_entry(vault, key) updates single entry atomically
- Workspace path fields (paper_root, main_note_path, fulltext_path, deep_reading_path, ai_path) present in each index entry
- Path fields mirrored in note frontmatter for Base compatibility
- CLI --rebuild-index flag forces full rebuild
- Schema_version mismatch auto-triggers full rebuild
</success_criteria>

<output>
After completion, create `.planning/phases/23-canonical-asset-index-safe-rebuilds/23-02-SUMMARY.md`
</output>
