---
phase: 16-retry-progress-bars
plan: 02
type: execute
wave: 2
depends_on:
  - "16-01"
files_modified:
  - paperforge/cli.py
  - paperforge/commands/ocr.py
  - paperforge/worker/ocr.py
autonomous: true
requirements:
  - REL-01
  - REL-02
  - REL-03
  - REL-04
  - OBS-04

must_haves:
  truths:
    - "User can run `paperforge --no-progress ocr` to suppress all progress bars"
    - "Transient PaddleOCR upload failures (HTTP 429/503, ConnectionError, Timeout) trigger automatic retry with exponential backoff"
    - "Transient PaddleOCR poll failures trigger automatic retry with exponential backoff"
    - "meta.json after a retry event contains retry_count (int), last_error (str|null), last_attempt_at (ISO-8601 str|null)"
    - "Zombie 'queued'/'running' jobs older than 30 minutes are reset to 'pending' on ocr start"
    - "A single upload failure does not abort the batch — failed items are logged, meta.json updated, processing continues"
    - "tqdm progress bars appear during OCR upload and poll iterations in interactive terminals"
  artifacts:
    - path: paperforge/cli.py
      provides: "--no-progress global root parser flag (same level as --verbose)"
      contains: "--no-progress"
    - path: paperforge/commands/ocr.py
      provides: "Wires no_progress from argparse → run_ocr(vault, verbose, no_progress)"
      contains: "no_progress"
    - path: paperforge/worker/ocr.py
      provides: "Full integration: retry on upload/poll, meta.json extensions, zombie reset, progress bars, batch resilience"
      contains: "configure_retry"
  key_links:
    - from: paperforge/cli.py
      to: paperforge/commands/ocr.py
      via: "args.no_progress is available to command dispatch"
      pattern: "no_progress"
    - from: paperforge/commands/ocr.py
      to: paperforge/worker/ocr.py
      via: "run_ocr(vault, verbose=verbose, no_progress=no_progress)"
      pattern: "run_ocr.*no_progress"
    - from: paperforge/worker/ocr.py
      to: paperforge/worker/_retry.py
      via: "from paperforge.worker._retry import configure_retry, retry_with_meta"
      pattern: "from.*_retry import"
    - from: paperforge/worker/ocr.py
      to: paperforge/worker/_progress.py
      via: "from paperforge.worker._progress import progress_bar"
      pattern: "from.*_progress import"
---

<objective>
Wire --no-progress through the CLI layer and integrate retry, meta.json extensions, zombie reset, progress bars, and batch resilience into run_ocr().

Purpose: Make the OCR pipeline resilient (retry on transient failures, zombie cleanup, batch resilience) and observable (progress bars, meta.json retry tracking). All infrastructure (_retry.py, _progress.py) was created in Plan 16-01.

Output: paperforge/cli.py (modified), paperforge/commands/ocr.py (modified), paperforge/worker/ocr.py (modified)
</objective>

<execution_context>
@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md
@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/16-retry-progress-bars/16-CONTEXT.md

@paperforge/cli.py
@paperforge/commands/ocr.py
@paperforge/worker/ocr.py (especially run_ocr() at l.1126-1285, ensure_ocr_meta() at l.119-143, cleanup_blocked_ocr_dirs() at l.266)

Interfaces from plan 16-01 (already on disk):
- `_retry.py::configure_retry()` — returns a tenacity `@retry` decorator with env-var config
- `_retry.py::retry_with_meta(fn, meta_path, *args, **kwargs)` — retries fn, writes retry info to meta.json on each attempt, re-raises on exhaustion
- `_progress.py::progress_bar(iterable, desc, total, disable)` — tqdm wrapper on stderr
</interfaces>

<tasks>

<task type="auto">
  <name>Task 1: Add --no-progress global flag and wire through command dispatch</name>
  <files>paperforge/cli.py, paperforge/commands/ocr.py</files>
  <action>
    Per D-03, add `--no-progress` as a **global root parser flag** at the same level as `--verbose` in `cli.py::build_parser()`:

    In `paperforge/cli.py`:
    1. After the `--verbose` argument (l.128-132), add:
       ```python
       parser.add_argument(
           "--no-progress",
           action="store_true",
           help="Disable progress bars (tqdm) for all commands",
       )
       ```
       Note: argparse converts `--no-progress` to attribute `no_progress` (dash → underscore).

    2. No changes needed in `main()` for dispatch — `args.no_progress` is already available on the namespace and will be passed by command modules.

    In `paperforge/commands/ocr.py`:
    1. Modify `run()` to accept and pass through `no_progress`:
       - Read `no_progress = getattr(args, "no_progress", False)` in the `run()` function
       - Pass it to `run_ocr()`: add `no_progress=no_progress` to the call at l.77
    2. Update the `run_ocr` call signature expectation — currently `run_ocr(vault, verbose=getattr(args, "verbose", False))`, change to `run_ocr(vault, verbose=getattr(args, "verbose", False), no_progress=getattr(args, "no_progress", False))`

    DO NOT:
    - Modify any other command modules (other commands don't use progress bars yet)
    - Add --no-progress to sub-parsers (it's root-level only per D-03)
    - Remove or modify the --verbose flag
  </action>
  <verify>
    <automated>
      python -c "import sys; sys.path.insert(0, '.'); from paperforge.cli import build_parser; p = build_parser(); a = p.parse_args(['--no-progress', 'ocr']); assert a.no_progress == True; assert a.command == 'ocr'; print('OK: --no-progress flag accepted at root level')"
    </automated>
  </verify>
  <done>
    `python -m paperforge --help` shows `--no-progress` option; `paperforge --no-progress ocr` passes `no_progress=True` to command layer; no syntax errors in cli.py or commands/ocr.py.
  </done>
</task>

<task type="auto">
  <name>Task 2: Integrate retry, meta.json extensions, zombie reset, progress bars, and batch resilience into run_ocr()</name>
  <files>paperforge/worker/ocr.py</files>
  <action>
    This is the core integration task. Modify `run_ocr()` (l.1126-1285) and its dependencies.

    **A. Update function signature (l.1126):**
    Change `def run_ocr(vault: Path, verbose: bool = False) -> int:` to:
    `def run_ocr(vault: Path, verbose: bool = False, no_progress: bool = False) -> int:`

    **B. Add imports (near top of file with other imports, ~l.24-44):**
    ```python
    from paperforge.worker._retry import configure_retry, retry_with_meta
    from paperforge.worker._progress import progress_bar
    ```

    **C. Zombie Reset (REL-03, D-08) — add at the top of run_ocr() body, after l.1130:**
    After `cleanup_blocked_ocr_dirs(paths)` and before loading control actions, add a zombie scan:
    ```
    Scan paths['ocr'] for all subdirectories containing meta.json.
    For each meta.json:
      - Read it
      - If ocr_status is 'queued' or 'running' (i.e., "processing" per D-08) AND ocr_started_at is set:
        - Parse ocr_started_at as ISO-8601 datetime
        - If it's older than the threshold (read from env var, e.g. PAPERFORGE_ZOMBIE_TIMEOUT, default "30", parsed as int minutes):
          - Set ocr_status = 'pending'
          - Increment retry_count (set default 0 if missing, then +1)
          - Write updated meta.json via write_json()
          - Log: logger.warning(f"Zombie reset {key}: was {status}, started at {started}, reset to pending")
    ```
    Threshold env var name: `PAPERFORGE_ZOMBIE_TIMEOUT_MINUTES` (default 30).

    **D. Retry + meta.json extensions (REL-01, REL-02, D-04, D-05, D-06, D-07):**

    **D1. Upload retry — wrap the upload block (~l.1256-1269):**
    The current upload code opens the PDF, POSTs, and calls raise_for_status(). On exception, it classifies the error and continues.
    
    Refactor the upload into a small inner function `_do_upload()` that captures the response:
    ```python
    def _do_upload(token, job_url, model, optional_payload, resolved_pdf):
        with open(resolved_pdf, 'rb') as f:
            resp = requests.post(job_url, headers={'Authorization': f'bearer {token}'},
                                 data={'model': model, 'optionalPayload': json.dumps(optional_payload)},
                                 files={'file': f}, timeout=120)
        resp.raise_for_status()
        return resp
    ```
    
    Replace the try/except upload block (l.1256-1269) with:
    ```python
    try:
        response = retry_with_meta(_do_upload, meta_path, token, job_url, model, optional_payload, resolved_pdf)
    except Exception as e:
        # Existing exception handler (l.1260-1269) — classify_error, set status, write meta
        ...
        continue
    ```
    
    `retry_with_meta()` will:
    - Retry upload on ConnectionError, Timeout, HTTPError 429/503 with exponential backoff
    - Write retry_count, last_error, last_attempt_at to meta.json on each retry
    - Re-raise the final exception if all retries exhausted
    - The existing `except Exception as e` handler catches the re-raised exception, classifies it, updates meta.json (which now already has retry_count/last_error), and continues to the next item (batch resilience per D-09)

    **D2. Poll retry — wrap the poll GET + raise_for_status (l.1175-1176):**
    Wrap:
    ```python
    try:
        response = retry_with_meta(_do_poll, meta_path, job_url, meta['ocr_job_id'], token)
    except Exception as e:
        # classify error, set status, write meta
        ...
        continue  # batch resilience — don't abort entire poll loop
    ```
    
    Where `_do_poll` is:
    ```python
    def _do_poll(job_url, job_id, token):
        resp = requests.get(f"{job_url}/{job_id}", headers={'Authorization': f'bearer {token}'}, timeout=60)
        resp.raise_for_status()
        return resp
    ```
    
    This catches poll failures that previously crashed the batch (l.1176 `response.raise_for_status()` with no try/except). Per D-09, poll failures must not abort the batch.

    **D3. ensure_ocr_meta() extension (REL-02):**
    In `ensure_ocr_meta()` (~l.119-143), add three new setdefault lines after l.142:
    ```python
    meta.setdefault('retry_count', 0)
    meta.setdefault('last_error', None)
    meta.setdefault('last_attempt_at', None)
    ```

    **E. Progress bars (OBS-04, D-10):**

    **E1. Upload loop progress:**
    The upload loop runs as the second `for queue_row in ocr_queue:` block (starting ~l.1225). Wrap it with `progress_bar()`:
    ```python
    upload_items = [row for row in ocr_queue if ...]  # filter to items that need upload
    for queue_row in progress_bar(upload_items, desc="Uploading", disable=no_progress):
        ...
    ```
    Actually, the upload filtering is more nuanced — the existing loop already checks `status == 'done'`, `status in {'queued', 'running'}`, etc. To avoid duplicating filter logic, use the original `ocr_queue` with `total=len(ocr_queue)` and let tqdm handle it. Or filter to non-done items. The executor should choose the approach that requires minimal structural change.

    **Best approach (recommended):** Keep the outer loop structure intact. Wrap the iterable with progress_bar:
    ```python
    for queue_row in progress_bar(ocr_queue, desc="Uploading", disable=no_progress):
    ```
    tqdm handles the iteration transparently. The desc label changes based on context.

    **E2. Poll loop progress:**
    Similarly wrap the poll loop:
    ```python
    active_items = [row for row in ocr_queue if row.get('queue_status') in ...]
    for queue_row in progress_bar(active_items, desc="Polling", disable=no_progress):
    ```
    Or simply wrap the outer loop with progress_bar. Executor's discretion.

    The key constraint (D-10): `progress_bar(..., disable=no_progress)` where `no_progress` is the value from `--no-progress` flag, passed through from `run_ocr()` parameter.

    **F. Batch resilience (REL-04, D-09):**
    - The upload retry (D1) already achieves batch resilience: if `retry_with_meta` re-raises, the `except Exception` block catches it, classifies the error, updates meta, and `continue`s.
    - The poll retry (D2) achieves poll resilience: poll failures are caught, classified, and `continue`d instead of crashing the batch.
    - No additional structural changes needed — the existing `for queue_row in ocr_queue:` loop naturally continues to the next item.

    **G. Update `run_ocr()` signature docstring if one exists.** Ensure the new parameter `no_progress: bool = False` is documented.

    IMPORTANT VERIFICATION CHECK:
    - The `run_ocr` function must accept `no_progress: bool = False` as a keyword argument
    - All three meta.json new fields (retry_count, last_error, last_attempt_at) are written via `retry_with_meta` and defaulted in `ensure_ocr_meta`
    - The poll `raise_for_status()` at l.1175-1176 is wrapped in try/except (previously uncaught!)
    - Upload failure does not `continue` without updating meta.json first
  </action>
  <verify>
    <automated>
      python -c "import sys; sys.path.insert(0, '.'); from paperforge.worker.ocr import run_ocr; import inspect; sig = inspect.signature(run_ocr); assert 'no_progress' in sig.parameters; print('OK: run_ocr accepts no_progress parameter')"
    </automated>
  </verify>
  <done>
    All five integration points complete:
    1. `run_ocr()` accepts `no_progress` parameter and passes it to `progress_bar()`
    2. Upload wrapped with `retry_with_meta()` — retries on transient failures, writes meta.json per retry
    3. Poll wrapped with `retry_with_meta()` — retries on poll failures, writes meta.json, catches exceptions for batch resilience
    4. `ensure_ocr_meta()` includes `retry_count=0`, `last_error=None`, `last_attempt_at=None` defaults
    5. Zombie reset at startup — resets 'queued'/'running' older than threshold to 'pending'
    6. Progress bars visible during upload and poll iterations (unless --no-progress or non-TTY)
    7. Batch resilience: single item failure doesn't abort the loop
  </done>
</task>

</tasks>

<verification>
1. [Automated] `python -m paperforge --help` output includes `--no-progress`
2. [Automated] `python -c "from paperforge.worker.ocr import run_ocr; run_ocr.__code__.co_varnames"` shows `no_progress` in parameter list
3. [Automated] `python -c "from paperforge.worker.ocr import ensure_ocr_meta; import inspect; source = inspect.getsource(ensure_ocr_meta); assert 'retry_count' in source; assert 'last_error' in source; assert 'last_attempt_at' in source; print('OK: meta.json schema extended')"`
4. [Automated] Python's `pytest tests/ -x --timeout=30` passes (no regressions — new imports don't break existing tests)
5. [Manual] `paperforge doctor` shows no errors with new dependencies
</verification>

<success_criteria>
- `--no-progress` appears in `paperforge --help`, accessible as `args.no_progress`
- `run_ocr()` accepts `no_progress` parameter and imports from `_retry` and `_progress`
- `ensure_ocr_meta()` returns meta dicts with `retry_count`, `last_error`, `last_attempt_at` fields
- Upload and poll operations wrapped with retry via `retry_with_meta()`
- Zombie reset runs at the start of `run_ocr()`, resetting stale 'queued'/'running' jobs
- All existing tests pass with zero failures
</success_criteria>

<output>
After completion, create `.planning/phases/16-retry-progress-bars/16-02-SUMMARY.md`
</output>
