---
phase: 15-deep-reading-queue-merge
plan: 01
type: execute
wave: 1
depends_on:
  - "14-shared-utilities-extraction"   # _utils.py must exist (already on disk)
files_modified:
  - "paperforge/worker/_utils.py"
  - "paperforge/worker/deep_reading.py"
  - "paperforge/skills/literature-qa/scripts/ld_deep.py"
autonomous: true
requirements:
  - CH-03

must_haves:
  truths:
    - "paperforge/worker/_utils.py exports scan_library_records(vault) returning list[dict] with all D-03 fields"
    - "paperforge deep-reading CLI produces identical categorized output (ready/waiting/blocked) as before refactor"
    - "ld_deep.py::scan_deep_reading_queue() returns identical results (same keys, same filtering, same sort order)"
    - "~50 lines of duplicate scanning logic removed from ld_deep.py"
    - "All existing smoke tests pass with zero failures"
  artifacts:
    - path: "paperforge/worker/_utils.py"
      provides: "scan_library_records() + _resolve_formal_note_path()"
      min_lines: 30
      contains: "def scan_library_records"
    - path: "paperforge/worker/deep_reading.py"
      provides: "Refactored run_deep_reading() using scan_library_records()"
      has_import: "scan_library_records"
    - path: "paperforge/skills/literature-qa/scripts/ld_deep.py"
      provides: "Refactored scan_deep_reading_queue() as thin wrapper"
      has_import: "scan_library_records"
  key_links:
    - from: "paperforge/worker/_utils.py::scan_library_records"
      to: "paperforge.config::paperforge_paths"
      via: "function-level import from paperforge.config"
      pattern: "from paperforge.config import paperforge_paths"
    - from: "paperforge/worker/_utils.py::scan_library_records"
      to: "paperforge/worker/_utils.py::_resolve_formal_note_path"
      via: "direct call within function"
      pattern: "_resolve_formal_note_path"
    - from: "paperforge/worker/deep_reading.py::run_deep_reading"
      to: "paperforge/worker/_utils.py::scan_library_records"
      via: "from paperforge.worker._utils import scan_library_records"
      pattern: "scan_library_records(vault)"
    - from: "paperforge/skills/literature-qa/scripts/ld_deep.py::scan_deep_reading_queue"
      to: "paperforge/worker/_utils.py::scan_library_records"
      via: "from paperforge.worker._utils import scan_library_records"
      pattern: "scan_library_records(vault)"

---

<objective>
**Phase 15: Deep-Reading Queue Merge -- Plan 01: Canonical scan_library_records()**

Purpose: Eliminate the divergent queue-scanning logic between worker/deep_reading.py and skills/ld_deep.py by creating a single canonical `scan_library_records()` in _utils.py that both callers use. Zero behavioral change to queue filtering, status output, or sync behavior.

Output:
1. `scan_library_records(vault: Path) -> list[dict]` in _utils.py (pure data acquisition, no side effects)
2. `_resolve_formal_note_path()` moved from deep_reading.py to _utils.py (re-export preserved)
3. `deep_reading.py::run_deep_reading()` refactored to call scan_library_records() + retain status sync & reporting
4. `ld_deep.py::scan_deep_reading_queue()` becomes a thin wrapper calling scan_library_records() + filter + sort
5. ~50 lines of duplicate scanning logic removed from ld_deep.py
</objective>

<execution_context>
@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md
@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md
</execution_context>

<context>
@D:\L\Med\Research\99_System\LiteraturePipeline\github-release\.planning\phases\15-deep-reading-queue-merge\15-CONTEXT.md
@D:\L\Med\Research\99_System\LiteraturePipeline\github-release\.planning\phases\14-shared-utilities-extraction\14-CONTEXT.md
@D:\L\Med\Research\99_System\LiteraturePipeline\github-release\.planning\ROADMAP.md

<interfaces>
<!-- Key types and contracts the executor needs -->

From paperforge/worker/_utils.py (existing):
```python
def read_json(path: Path) -> dict: ...      # JSON file reader
STANDARD_VIEW_NAMES = frozenset([...])      # Section headers constant
# Module-level logger: logger = logging.getLogger(__name__)
# Function-level import pattern: from paperforge.config import load_vault_config  (inside load_journal_db)
```

From paperforge/worker/deep_reading.py::run_deep_reading() (current signature):
```python
def run_deep_reading(vault: Path, verbose: bool = False) -> int
```
Current flow: iterate export JSON -> check record exists -> read frontmatter -> check note content -> sync status -> build pending_queue with {zotero_key, domain, title, ocr_status, is_analyze, is_do_ocr} -> categorize ready/waiting/blocked -> write report -> print summary

From paperforge/skills/literature-qa/scripts/ld_deep.py::scan_deep_reading_queue() (current signature):
```python
def scan_deep_reading_queue(vault: Path) -> list[dict]
```
Current return keys: zotero_key, title, domain, deep_reading_status, ocr_status
Current logic: iterate records_root -> read frontmatter -> filter analyze=true + dr!="done" -> lookup ocr_status -> sort by (ocr_done_first, domain, key) -> return

From paperforge/worker/deep_reading.py::_resolve_formal_note_path() (to be moved to _utils.py):
```python
def _resolve_formal_note_path(vault: Path, zotero_key: str, domain: str) -> Path | None
```
Uses: pipeline_paths(vault)['literature']  (same value as paperforge_paths(vault)['literature'])
</interfaces>
</context>

<tasks>

<task type="auto" tdd="false">
  <name>Task 1: Add scan_library_records() and _resolve_formal_note_path() to _utils.py</name>
  <files>
      - "paperforge/worker/_utils.py"
  </files>
  <action>
    Add two functions to `paperforge/worker/_utils.py` in a new section `# --- Deep-Reading Queue ---` at the end of the file (after line 113, the `_extract_year` function).

    **1. _resolve_formal_note_path(vault, zotero_key, domain) -> Path | None**
    Move this function from deep_reading.py (lines 117-131) into _utils.py.
    - The original uses `pipeline_paths(vault)['literature']` -- replace with function-level `from paperforge.config import paperforge_paths; lit_root = paperforge_paths(vault)['literature']` (same path, follows _utils.py leaf module constraint per D-01)
    - Import `re` at the top of the function (already available, `re` is imported at module level in _utils.py)
    - Import `Path` (already imported at module level)
    - Keep the exact same logic: iterate domain_dir.rglob('*.md'), search for `zotero_key:` in frontmatter, return the first match or None
    - Exception handling: keep the existing UnicodeDecodeError fallback pattern

    **2. scan_library_records(vault: Path) -> list[dict]** (signature per D-02)
    Per D-02, D-03, D-04, D-09:

    ```python
    def scan_library_records(vault: Path) -> list[dict]:
        """Scan library-records for analyze=true entries.

        Pure data acquisition -- no side effects, no categorization.
        Returns all library records with analyze=true regardless of deep_reading_status.
        Caller filters and categorizes as needed.

        Return value fields (per D-03):
          - zotero_key: str
          - domain: str
          - title: str
          - analyze: bool (always True in results, included for symmetrical interface)
          - do_ocr: bool
          - deep_reading_status: str ("pending" | "done" -- from frontmatter, not validated)
          - ocr_status: str ("pending" | "processing" | "done" | "failed" -- from meta.json)
          - note_path: Path | None (resolved via _resolve_formal_note_path)
        """
    ```

    Implementation:
    - Use function-level `from paperforge.config import paperforge_paths` (same pattern as load_journal_db)
    - Get `records_root = paperforge_paths(vault)['library_records']` and `ocr_root = paperforge_paths(vault)['ocr']`
    - If records_root doesn't exist, return empty list immediately
    - Iterate `sorted(records_root.iterdir())` for domain directories (skip non-dirs)
    - Within each domain, iterate `sorted(domain_dir.glob('*.md'))` for records
    - Use try/except on record_path.read_text() to skip unreadable files (per D-09 agent's discretion on lenient error handling)
    - Extract frontmatter fields using `re.search(re.MULTILINE)` -- same regex patterns as CURRENT ld_deep.py version for consistency (they're already battle-tested):
      - `zotero_key_match = re.search(r'^zotero_key:\s*(.+)$', text, re.MULTILINE)` -- strip and unquote
      - `analyze_match = re.search(r'^analyze:\s*(true|false)$', text, re.MULTILINE)` -- skip if not "true"
      - `title_match = re.search(r'^title:\s*"?(.+?)"?$', text, re.MULTILINE)` -- strip and unquote
      - `do_ocr_match = re.search(r'^do_ocr:\s*(true|false)$', text, re.MULTILINE)` -- boolean
      - `status_match = re.search(r'^deep_reading_status:\s*"?(.*?)"?$', text, re.MULTILINE)` -- strip, default "pending"
    - OCR status lookup (per D-09): read `ocr_root / zotero_key / 'meta.json'` using `read_json()`. If exists and readable, extract `meta.get('ocr_status', 'pending')`. Otherwise default to 'pending'. Wrap in try/except for graceful degradation.
    - Resolve note_path by calling `_resolve_formal_note_path(vault, zotero_key, domain)`
    - Append dict with all D-03 fields

    **Sorting**: Per D-03 agent's discretion, scan_library_records() does NOT sort its output. Callers sort as needed.

    **Section header**: Add `# --- Deep-Reading Queue ---` above the two functions at the end of the file.
  </action>
  <verify>
    <automated>python -c "from paperforge.worker._utils import scan_library_records, _resolve_formal_note_path; print('OK')"</automated>
  </verify>
  <done>
    - _utils.py has new `# --- Deep-Reading Queue ---` section
    - `scan_library_records(vault: Path) -> list[dict]` exported from _utils.py
    - `_resolve_formal_note_path(vault, key, domain) -> Path | None` exported from _utils.py
    - No circular imports (verified by import test above)
    - All D-03 fields present in return dict structure
  </done>
</task>

<task type="auto" tdd="false">
  <name>Task 2: Refactor deep_reading.py and ld_deep.py to call scan_library_records()</name>
  <files>
      - "paperforge/worker/deep_reading.py"
      - "paperforge/skills/literature-qa/scripts/ld_deep.py"
  </files>
  <action>
    **IMPORTANT**: Read both files before modifying to see their exact current state.

    ---

    **Part A: Refactor deep_reading.py**

    1. **Add imports**: Add `scan_library_records` and `_resolve_formal_note_path` to the existing `from paperforge.worker._utils import ...` block (lines 28-42). Both must be added.

    2. **Replace `_resolve_formal_note_path` definition** (lines 117-131): Remove the function body and replace with a re-export comment:
       ```python
       # Re-exported from _utils.py for backward compatibility
       ```
       This matches the Phase 14 re-export pattern established in all 7 workers.

    3. **Refactor `run_deep_reading()`** (lines 133-232):

       The refactored function keeps its existing signature `def run_deep_reading(vault: Path, verbose: bool = False) -> int:` and all existing behavior EXCEPT the inline scanning loop.

       **What changes:**
       - Remove the inline scan loop (lines 149-191, from `for export_path in sorted(...)` through the pending_queue.append block)
       - Replace with a call to `scan_library_records(vault)` at the top of the function (after line 146 `domain_lookup = ...`), then convert the returned records into the pending_queue
       - Keep the sync logic that checks `has_deep_reading_content()` against the actual note (per D-05)

       **New flow:**
       ```python
       paths = pipeline_paths(vault)
       config = load_domain_config(paths)
       ensure_base_views(vault, paths, config)
       domain_lookup = {entry['export_file']: entry['domain'] for entry in config['domains']}
       synced = 0
       pending_queue: list[dict] = []

       records = scan_library_records(vault)

       for record in records:
           key = record['zotero_key']
           domain = record['domain']

           # Status sync: check actual note content vs frontmatter status
           note_path = record['note_path']
           has_content = False
           if note_path and note_path.exists():
               note_text = note_path.read_text(encoding='utf-8')
               has_content = has_deep_reading_content(note_text)
           correct_status = 'done' if has_content else 'pending'

           if record['deep_reading_status'] != correct_status:
               record_dir = paths['library_records'] / domain
               record_path = record_dir / f'{key}.md'
               record_text = record_path.read_text(encoding='utf-8')
               new_text = re.sub(
                   '^deep_reading_status:\\s*"?.*?"?$',
                   f'deep_reading_status: {yaml_quote(correct_status)}',
                   record_text, flags=re.MULTILINE, count=1
               )
               record_path.write_text(new_text, encoding='utf-8')
               synced += 1

           if correct_status == 'pending':
               pending_queue.append({
                   'zotero_key': key,
                   'domain': domain,
                   'title': record['title'],
                   'ocr_status': record['ocr_status'],
                   'is_analyze': True,
                   'is_do_ocr': record['do_ocr'],
               })
       ```

       **NO changes to** the rest of the function (lines 192-232): the categorization into ready/waiting/blocked, the markdown report generation, the print() summary, and the return 0 all remain identical.

       **Why `record['do_ocr']` replaces `is_do_ocr`**: This field now comes from scan_library_records() (which reads it from library-record frontmatter) instead of being extracted inline. The value is the same boolean.

    **Part B: Refactor ld_deep.py**

    Per D-07, D-08:

    1. **Add module-level import**: At the top of the file (after line 11, alongside existing `from pathlib import Path` imports), add:
       ```python
       from paperforge.worker._utils import scan_library_records
       ```
       This is module-level direct import per D-08. No circular dependency risk (_utils.py is a leaf module).

    2. **Replace `scan_deep_reading_queue()` body** (lines 1161-1216): The function signature stays the same for backward compatibility, but its body becomes a thin wrapper:

       ```python
       def scan_deep_reading_queue(vault: Path) -> list[dict]:
           """Scan library-records for analyze=true + deep_reading_status!=done entries.

           Thin wrapper around scan_library_records() in _utils.py.
           Re-exported from _utils.py for backward compatibility.
           """
           all_records = scan_library_records(vault)
           # Filter: only records that still need deep reading
           queue = [r for r in all_records if r['deep_reading_status'] != 'done']
           # Sort: OCR completed first, then by domain, then by key
           queue.sort(key=lambda row: (
               0 if row['ocr_status'] == 'done' else 1,
               row['domain'],
               row['zotero_key'],
           ))
           return queue
       ```

       Note: The wrapper keeps the same sort order as the original ld_deep.py implementation. The return dict includes ALL D-03 fields (including `analyze`, `do_ocr`, `note_path`) which is backward-compatible for callers that check for specific keys.

    3. **Add re-export comment** at the original definition site (replacing the old docstring body is sufficient):
       The function now IS the wrapper, so the definition site IS the wrapper. No separate re-export needed -- the function itself is the backward-compatible entry point.

    4. **Clean up**: The `_read_json()` function (line 949) and `_paperforge_paths()` function (line 23) can be left in place for now. They may be unused by other functions -- dead code removal is deferred to Phase 17 (per CONTEXT.md "Out of scope"). Do NOT remove them in this phase.
  </action>
  <verify>
    <automated>pytest tests/test_smoke.py -x -v 2>&1 | findstr /C:"passed" /C:"failed" /C:"ERROR"</automated>
  </verify>
  <done>
    - deep_reading.py imports scan_library_records and _resolve_formal_note_path from _utils
    - deep_reading.py re-exports _resolve_formal_note_path with comment
    - run_deep_reading() calls scan_library_records() and retains sync + report logic
    - ld_deep.py module-level imports scan_library_records from _utils
    - scan_deep_reading_queue() is a thin wrapper around scan_library_records()
    - ~50 lines removed from ld_deep.py (duplicate scanning loop)
    - All existing smoke tests pass
  </done>
</task>

</tasks>

<verification>
1. Run the smoke test suite to confirm all tests pass:
   `pytest tests/test_smoke.py -x -v`

2. Verify Python import:
   `python -c "from paperforge.worker._utils import scan_library_records, _resolve_formal_note_path; print('OK: _utils exports verified')"`

3. Verify deep_reading.py uses the new function:
   `python -c "import paperforge.worker.deep_reading; print('OK: deep_reading imports without circular deps')"`

4. Verify ld_deep.py module-level import works:
   `python -c "from paperforge.skills.literature_qa.scripts.ld_deep import scan_deep_reading_queue; print('OK: ld_deep imports scan_library_records via re-export')"`
</verification>

<success_criteria>
- [ ] `scan_library_records()` exists in _utils.py and exports correctly (verified by import test)
- [ ] deep_reading.py imports from _utils and run_deep_reading() works (verified by import test)
- [ ] ld_deep.py imports from _utils module-level and scan_deep_reading_queue() works (verified by import test)
- [ ] All 205 existing smoke tests pass (verified by pytest)
- [ ] ~50 lines of duplicate scanning logic removed from ld_deep.py
- [ ] Zero behavioral change to `paperforge deep-reading` CLI output
</success_criteria>

<output>
After successful implementation and verification, create `.planning/phases/15-deep-reading-queue-merge/15-01-SUMMARY.md`
</output>
