---
phase: 16-retry-progress-bars
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - pyproject.toml
  - paperforge/worker/_retry.py
  - paperforge/worker/_progress.py
autonomous: true
requirements:
  - REL-01
  - REL-02
  - OBS-04

must_haves:
  truths:
    - "tenacity and tqdm are declared in pyproject.toml and installable via pip"
    - "from paperforge.worker._retry import configure_retry, retry_with_meta succeeds at runtime"
    - "from paperforge.worker._progress import progress_bar succeeds at runtime"
    - "configure_retry() reads PAPERFORGE_RETRY_MAX and PAPERFORGE_RETRY_BACKOFF from os.environ"
    - "progress_bar() wraps tqdm outputting to sys.stderr with auto-disable for non-TTY"
  artifacts:
    - path: pyproject.toml
      provides: "tenacity + tqdm declared in [project.dependencies]"
      contains: "tenacity"
    - path: paperforge/worker/_retry.py
      provides: "Retry decorator with env-var config, meta.json update callback"
      exports: ["configure_retry", "retry_with_meta"]
    - path: paperforge/worker/_progress.py
      provides: "tqdm wrapper with stderr output, auto-disable, disable flag"
      exports: ["progress_bar"]
  key_links:
    - from: paperforge/worker/_retry.py
      to: tenacity
      via: "import tenacity"
      pattern: "from tenacity import|import tenacity"
    - from: paperforge/worker/_progress.py
      to: tqdm
      via: "from tqdm import tqdm"
      pattern: "import tqdm|from tqdm"
---

<objective>
Create the two new leaf modules (_retry.py and _progress.py) and add their dependencies to pyproject.toml.

Purpose: Establish the infrastructure for retry logic and progress bars as independent, testable leaf modules following the _utils.py pattern. These are dependencies for the Phase 16 integration work (Plan 16-02).

Output: pyproject.toml (modified), paperforge/worker/_retry.py (new), paperforge/worker/_progress.py (new)
</objective>

<execution_context>
@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md
@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/16-retry-progress-bars/16-CONTEXT.md

@paperforge/worker/_utils.py (leaf module pattern to replicate — imports only stdlib + external deps, no intra-project imports)
@paperforge/logging_config.py (single-purpose module precedent, ~69 lines)
@paperforge/config.py (env var reading pattern: os.environ.get)

interface contracts from existing code:
- `_utils.py::write_json(path: Path, data)` — used by retry_with_meta to persist meta.json updates
- `_utils.py::read_json(path: Path)` — reads meta.json before updating
- `paperforge/worker/ocr.py` l.1140-1146 (PAPERFORGE_RETRY_MAX env var pattern via os.environ.get)
- `paperforge/logging_config.py` l.37-40 (env var reading pattern: level_name = os.environ.get("PAPERFORGE_LOG_LEVEL", "INFO"))
</context>

<interfaces>
Existing contracts that plan_02 (OCR integration) will use:

```python
# _retry.py — exported by this plan, consumed by plan_02
def configure_retry() -> tenacity.retry: ...
    # Reads PAPERFORGE_RETRY_MAX (int, default 5) and PAPERFORGE_RETRY_BACKOFF (float, default 2.0)
    # Returns a tenacity @retry decorator configured with:
    #   - exponential backoff: min=1s, max=30s, multiplier=backoff
    #   - jitter (tenacity default jitter)
    #   - stop_after_attempt(max)
    #   - retry_on: ConnectionError, Timeout, HTTPError 429/503
    #   - reraise=True
    # Used as: @configure_retry() on a helper function

def retry_with_meta(fn: Callable, meta_path: Path, *args, **kwargs) -> Any: ...
    # Calls fn(*args, **kwargs) with tenacity retry.
    # On each retry attempt, writes to meta.json:
    #   retry_count, last_error, last_attempt_at (ISO-8601)
    # On eventual exhaustion: re-raises the last exception.
    # Used as: try: resp = retry_with_meta(upload_fn, meta_path, ...); except: ...

# _progress.py — exported by this plan, consumed by plan_02
def progress_bar(iterable, desc: str = "", total: int | None = None, disable: bool = False) -> tqdm: ...
    # Wraps tqdm.tqdm(iterable, desc=desc, total=total, disable=disable, file=sys.stderr)
    # Auto-detects non-TTY fallback (CI/pipe) — tqdm's built-in mininterval
    # Returns the tqdm iterator.
```

Existing pattern from _utils.py (leaf module):
```python
from __future__ import annotations
import logging  # stdlib only
from pathlib import Path

logger = logging.getLogger(__name__)
# exports: public functions only
```
</interfaces>

<tasks>

<task type="auto">
  <name>Task 1: Add tenacity and tqdm to pyproject.toml dependencies</name>
  <files>pyproject.toml</files>
  <action>
    Add two new entries to the `[project.dependencies]` list in pyproject.toml, following the existing format:
    - `"tenacity>=8.2.0"` (latest stable as of 2026-04)
    - `"tqdm>=4.66.0"` (latest stable as of 2026-04)

    Place them after `"textual>=0.47.0"` in alphabetical order by package name (tqdm before tenacity).

    Do NOT modify any other part of pyproject.toml.
  </action>
  <verify>
    <automated>
      python -c "import tomllib; d = tomllib.load(open('pyproject.toml', 'rb')); deps = d['project']['dependencies']; assert any('tenacity' in dep for dep in deps); assert any('tqdm' in dep for dep in deps); print('OK: tenacity and tqdm in deps')"
    </automated>
  </verify>
  <done>
    pyproject.toml lists tenacity>=8.2.0 and tqdm>=4.66.0 in [project.dependencies]; format validation passes.
  </done>
</task>

<task type="auto">
  <name>Task 2: Create paperforge/worker/_retry.py leaf module</name>
  <files>paperforge/worker/_retry.py</files>
  <action>
    Create a new leaf module at `paperforge/worker/_retry.py` that follows the same patterns as `_utils.py`:
    - imports only stdlib + tenacity + requests (external deps only, no intra-project imports)
    - single-purpose: retry infrastructure for OCR operations
    - uses `logging.getLogger(__name__)` for diagnostic logging

    Exports two public functions:

    1. **`configure_retry() -> tenacity.retry`**
       - Reads `PAPERFORGE_RETRY_MAX` from os.environ, default `"5"`, parsed as int
       - Reads `PAPERFORGE_RETRY_BACKOFF` from os.environ, default `"2.0"`, parsed as float
       - Returns a `tenacity.retry` decorator configured with:
         - `stop=tenacity.stop_after_attempt(retry_max)`
         - `wait=tenacity.wait_exponential(multiplier=backoff, min=1, max=30)`
         - `retry=tenacity.retry_if_exception(_is_retryable)` where `_is_retryable` checks:
           - `requests.ConnectionError` → retryable
           - `requests.Timeout` → retryable
           - `requests.HTTPError` with `response.status_code in (429, 503)` → retryable
           - All other exceptions → NOT retryable (fail immediately)
         - `reraise=True` (always re-raise the last exception after exhaustion)
       - Logs `logger.warning` on each retry attempt (use `tenacity.before_sleep_log(logger, logging.WARNING)`)

    2. **`retry_with_meta(fn: Callable, meta_path: Path, *args, **kwargs) -> Any`**
       - Calls `fn(*args, **kwargs)` with tenacity retry configured identically to `configure_retry()` (env vars)
       - On each retry attempt (`before_sleep` callback), reads meta.json from `meta_path` and writes/updates:
         - `retry_count` (int) — current attempt number from `retry_state.attempt_number`
         - `last_error` (str | null) — `str(retry_state.outcome.exception())` if failed, else `null`
         - `last_attempt_at` (ISO-8601 str) — `datetime.now(timezone.utc).isoformat()`
       - After all retries exhausted: re-raises the last exception. The caller (ocr.py) handles it.
       - Uses `_utils.write_json(meta_path, meta)` for persistence
       - Per D-09 (batch resilience): function does NOT catch — it re-raises, letting the caller decide how to handle failure

    Internal helper:
    - `_is_retryable(exc: BaseException) -> bool` — classification logic per D-05

    Log retry events via `logger.warning("Retry attempt %d for %s: %s", attempt, fn.__name__, error)`.

    Add `from __future__ import annotations` as the first line.

    DO NOT:
    - Import anything from `paperforge.` (leaf constraint per D-01)
    - Catch exceptions in `retry_with_meta` (must re-raise for caller to handle)
    - Modify any existing files
  </action>
  <verify>
    <automated>
      python -c "import sys; sys.path.insert(0, '.'); from paperforge.worker._retry import configure_retry, retry_with_meta; print('OK: _retry imports clean')"
    </automated>
  </verify>
  <done>
    paperforge/worker/_retry.py exists; `from paperforge.worker._retry import configure_retry, retry_with_meta` succeeds; both functions respect env var defaults.
  </done>
</task>

<task type="auto">
  <name>Task 3: Create paperforge/worker/_progress.py leaf module</name>
  <files>paperforge/worker/_progress.py</files>
  <action>
    Create a new leaf module at `paperforge/worker/_progress.py` following the same leaf-module pattern:
    - imports only stdlib + tqdm
    - single export: `progress_bar()` function

    **`progress_bar(iterable, desc: str = "", total: int | None = None, disable: bool = False) -> tqdm`**
    - Returns `tqdm.tqdm(iterable, desc=desc, total=total, disable=disable, file=sys.stderr, unit="item")`
    - Configures sensible formatting defaults:
      - `file=sys.stderr` (D-10 — tqdm output on stderr, preserving stdout for piped data)
      - `mininterval=1.0` (update at most once per second — reduces flicker on fast iterations)
      - `unit="item"` (generic unit label)
      - `bar_format="{l_bar}{bar}| {n_fmt}/{total_fmt} [{elapsed}<{remaining}]"` (compact format showing progress)
    - `disable=False` → tqdm automatically detects TTY (per tqdm defaults). The `disable` parameter is for explicit control via `--no-progress`.
    - Uses `tqdm.auto.tqdm` or `from tqdm import tqdm` (standard tqdm has built-in `{bool(sys.stderr.isatty())}` behavior for `disable=None`, but we pass the `disable` flag explicitly)

    The module should be small (~30 lines) — a single function wrapping tqdm.

    DO NOT:
    - Import anything from `paperforge.` (leaf constraint per D-02)
    - Do any auto-disable logic beyond what `disable` parameter controls — `tqdm` handles TTY detection natively
  </action>
  <verify>
    <automated>
      python -c "import sys; sys.path.insert(0, '.'); from paperforge.worker._progress import progress_bar; print('OK: _progress imports clean')"
    </automated>
  </verify>
  <done>
    paperforge/worker/_progress.py exists; `from paperforge.worker._progress import progress_bar` succeeds; progress_bar returns a tqdm instance writing to stderr.
  </done>
</task>

</tasks>

<verification>
1. [Automated] `pip install -e .` succeeds (dependencies resolve)
2. [Automated] `python -c "from paperforge.worker._retry import configure_retry, retry_with_meta"` succeeds
3. [Automated] `python -c "from paperforge.worker._progress import progress_bar"` succeeds
4. [Automated] `python -c "from paperforge.worker._retry import configure_retry; configure_retry()"` — no errors, returns decorator
5. All three files committed
</verification>

<success_criteria>
- pyproject.toml includes tenacity and tqdm
- `paperforge/worker/_retry.py` exists, exports `configure_retry()` and `retry_with_meta()`, no intra-project imports
- `paperforge/worker/_progress.py` exists, exports `progress_bar()`, no intra-project imports
- Module imports verified clean at Python level
</success_criteria>

<output>
After completion, create `.planning/phases/16-retry-progress-bars/16-01-SUMMARY.md`
</output>
