---
phase: 54-dashboard-workflow-closure
plan: 002
type: execute
wave: 1
depends_on: []
files_modified:
  - tests/chaos/__init__.py
  - tests/chaos/conftest.py
  - tests/chaos/scenarios/CHAOS_MATRIX.md
  - tests/chaos/test_corrupted_inputs.py
  - tests/chaos/test_network_failures.py
  - tests/chaos/test_filesystem_errors.py
autonomous: true
requirements: [CHAOS-01, CHAOS-02, CHAOS-03, CHAOS-04]

must_haves:
  truths:
    - "CHAOS_MATRIX.md documents all destructive scenarios with triggers, expected behavior, and safety contracts"
    - "Corrupted input tests (malformed JSON, corrupt PDF, broken meta.json, missing frontmatter) produce graceful error messages — no unhandled crashes"
    - "Network failure tests (OCR API timeout, HTTP 401, 500, DNS unreachable) use mock backend and produce actionable error messages"
    - "Filesystem error tests (permission denied, locked files, missing directories, path too long) use isolation assertion — no real vault damage"
    - "All chaos tests are marked @pytest.mark.chaos and excluded from default CI gate"
    - "All chaos tests use tmp_path vault with assert guard; none ever touch a real vault"
  artifacts:
    - path: "tests/chaos/scenarios/CHAOS_MATRIX.md"
      provides: "Complete documentation of all destructive test scenarios"
      min_lines: 50
    - path: "tests/chaos/conftest.py"
      provides: "Chaos test vault fixture with isolation guard"
      contains: "chaos_vault"
    - path: "tests/chaos/test_corrupted_inputs.py"
      provides: "Corrupted input destructive tests (CHAOS-01)"
      contains: "test_malformed_json"
    - path: "tests/chaos/test_network_failures.py"
      provides: "Network failure destructive tests (CHAOS-02)"
      contains: "test_ocr_api_timeout"
    - path: "tests/chaos/test_filesystem_errors.py"
      provides: "Filesystem error destructive tests (CHAOS-03)"
      contains: "test_permission_denied"
  key_links:
    - from: "tests/chaos/conftest.py"
      to: "fixtures/vault_builder.py::VaultBuilder"
      via: "chaos_vault fixture creates disposable vaults"
      pattern: "from fixtures.vault_builder import VaultBuilder"
    - from: "tests/chaos/test_network_failures.py"
      to: "fixtures/ocr/mock_ocr_backend.py"
      via: "Uses mock_ocr_error() and mock_ocr_timeout() for network failure simulation"
      pattern: "mock_ocr_error|mock_ocr_timeout"
    - from: "tests/chaos/conftest.py::chaos_vault"
      to: "tmp_path"
      via: "pytest tmp_path ensures isolation, assert guard prevents real vault access"
      pattern: 'assert "tmp" in str(vault)'
---

<objective>
**Objective:** Document all destructive/abnormal scenarios in CHAOS_MATRIX.md and implement Level 6 chaos tests — corrupted inputs, network failures, and filesystem errors — with strict isolation guards.

**Purpose:** Chaos tests verify that PaperForge handles abnormal conditions gracefully (no unhandled crashes, actionable error messages) and that destructive scenarios are explicitly documented. The isolation guard (`assert "tmp" in str(vault)`) ensures chaos tests can NEVER accidentally run against a real user vault.

**Output:** `tests/chaos/scenarios/CHAOS_MATRIX.md` with all destructive scenarios; `tests/chaos/` with conftest + 3 test files covering corrupted inputs (CHAOS-01), network failures (CHAOS-02), and filesystem errors (CHAOS-03).
</objective>

<execution_context>
@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md
@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/REQUIREMENTS.md
@.planning/ROADMAP.md
@.planning/phases/54-dashboard-workflow-closure/54-CONTEXT.md

<interfaces>
Reusable infrastructure:

From tests/e2e/conftest.py:
```python
# e2e_cli_invoker fixture
# vault_builder fixture (session-scoped VaultBuilder)
# temp_vault fixture (tmp_path-based, "standard" level)
# full_vault fixture (tmp_path-based, "full" level)
```

From fixtures/vault_builder.py:
```python
# VaultBuilder.build("minimal") -> Path  (config + dirs + .env)
# VaultBuilder.build("standard") -> Path  (+ exports + PDFs + storage)
# VaultBuilder.build("full") -> Path  (+ OCR fixtures + formal notes + index)
```

From fixtures/ocr/mock_ocr_backend.py:
```python
# mock_ocr_success() -> RequestsMock context manager
# mock_ocr_error(status=401) -> RequestsMock  (also 400, 500)
# mock_ocr_timeout() -> RequestsMock  (poll returns 'queued' forever)
```

From fixtures/zotero/:
- 8+ JSON variants including: valid, empty, malformed, missing citationKey, missing title, etc.

From fixtures/pdf/:
- Minimal valid PDFs including CJK filenames
</interfaces>
</context>

<tasks>

<task type="auto" tdd="false">
  <name>Task 1: Write CHAOS_MATRIX.md</name>
  <files>
    - tests/chaos/scenarios/CHAOS_MATRIX.md
  </files>
  <action>
    Create `tests/chaos/scenarios/CHAOS_MATRIX.md` — a complete reference of all destructive test scenarios. Ensure the directory `tests/chaos/scenarios/` exists first by creating it.

    Structure as a markdown table with columns:
    | Scenario ID | Category | Trigger | Expected Behavior | Safety Contract | Test File |
    |-------------|----------|--------|-------------------|-----------------|-----------|

    Document at minimum these scenarios:

    **Corrupted Inputs (CHAOS-01):**
    | CI-01 | corrupted_input | Place malformed JSON in exports/ (missing closing brace, trailing comma) | `paperforge sync` prints "Error parsing JSON: {path}" with line number, exits with non-zero code, no crash | Only reads from tmp_path vault; assert "tmp" in str(vault) |
    | CI-02 | corrupted_input | Place an empty JSON file (`{}`) in exports/ | `paperforge sync` prints "No items found in {path}" or warning, exits successfully (0) — no formal notes created for empty export | Only reads from tmp_path vault |
    | CI-03 | corrupted_input | BBT JSON with items missing `citationKey` field | Sync gracefully skips the item, prints warning "Skipping item at index {N}: missing citationKey", continues processing valid items | Only reads from tmp_path vault |
    | CI-04 | corrupted_input | Corrupt PDF file (truncated, binary garbage) in Zotero storage | OCR submission fails or PDF processing prints "Error processing PDF: {path}" — no crash, non-zero exit but graceful | Only reads from tmp_path vault |
    | CI-05 | corrupted_input | Broken meta.json in OCR dir (invalid JSON, missing required fields) | `paperforge ocr` prints "Warning: meta.json corrupted for {key}" — ocr_status set to "failed" with error detail, no crash | Only reads from tmp_path vault |
    | CI-06 | corrupted_input | Formal note frontmatter missing `zotero_key` field | Status/doctor/repair handle gracefully — prints warning, skips the note in aggregate counts, no crash | Only reads from tmp_path vault |

    **Network Failures (CHAOS-02):**
    | NF-01 | network_failure | OCR API returns HTTP 401 on submit | `paperforge ocr` prints "OCR API authentication failed (401)" — ocr_status set to "failed" with error detail, no crash, suggests checking PADDLEOCR_API_TOKEN | Mock backend via responses; no real HTTP calls |
    | NF-02 | network_failure | OCR API returns HTTP 500 on submit | `paperforge ocr` prints "OCR API server error (500)" — retries with backoff (configured max retries), eventually sets ocr_status: "failed" | Mock backend via responses |
    | NF-03 | network_failure | OCR poll returns 'queued' indefinitely (timeout) | After max poll attempts, `paperforge ocr` prints "OCR job {id} did not complete after {N} polls" — ocr_status set to "failed" or "pending" with timeout note, no crash | mock_ocr_timeout() from fixtures |
    | NF-04 | network_failure | DNS unreachable / connection refused | `paperforge ocr` prints "Network error: unable to reach OCR API" — exits with error message, no traceback, no crash | Mock backend via responses or monkeypatch |

    **Filesystem Errors (CHAOS-03):**
    | FE-01 | filesystem_error | PDF attachments directory deleted after sync | `paperforge status` or `paperforge doctor` prints "PDF not found: {path}" — path_error field set, graceful degradation | assert "tmp" in str(vault); never touches real vault |
    | FE-02 | filesystem_error | System/PaperForge/ocr directory deleted | `paperforge ocr` re-creates missing dirs or prints "OCR directory missing, creating: {path}" — non-fatal | assert "tmp" in str(vault) |
    | FE-03 | filesystem_error | Formal note file deleted but entry still in canonical index | `paperforge repair` detects divergence, prints "Formal note missing for {key}" — suggests repair with `--fix` | assert "tmp" in str(vault) |
    | FE-04 | filesystem_error | Path too long (deeply nested Zotero path with CJK chars) | Path resolution falls back gracefully — prints warning about long path, uses available alternative path, no crash | assert "tmp" in str(vault) |
    | FE-05 | filesystem_error | Permission denied on exports/ directory | `paperforge sync` prints "Cannot read exports directory: {path}" — non-zero exit, actionable message, no crash | assert "tmp" in str(vault) |

    After the table, add a section:
    ## Safety Contracts (Mandatory)
    - EVERY chaos test MUST include `assert "tmp" in str(vault)` or `assert "pytest" in str(vault)` to prevent real vault access.
    - Chaos tests run ONLY in scheduled CI (ci-chaos.yml) — never in regular CI gate.
    - Chaos tests use mock backends for network operations — no external HTTP calls.
    - All tests are marked `@pytest.mark.chaos`.

    Target: ~80 lines.
  </action>
  <verify>
    <automated>python -c "p=__import__('pathlib').Path('tests/chaos/scenarios/CHAOS_MATRIX.md'); assert p.exists(); c=p.read_text('utf-8'); assert 'Scenario ID' in c; assert 'Safety Contract' in c; assert 'CI-01' in c; assert 'NF-01' in c; assert 'FE-01' in c"</automated>
  </verify>
  <done>
    CHAOS_MATRIX.md documents 15+ destructive scenarios across 3 categories with IDs, triggers, expected behavior, and safety contracts for each.
  </done>
</task>

<task type="auto" tdd="false">
  <name>Task 2: Create Chaos Test Infrastructure + Corrupted Input Tests</name>
  <files>
    - tests/chaos/__init__.py
    - tests/chaos/conftest.py
    - tests/chaos/test_corrupted_inputs.py
  </files>
  <action>
    Create the chaos test scaffold with strict isolation and implement corrupted input tests (CHAOS-01).

    **`tests/chaos/__init__.py`** — empty file.

    **`tests/chaos/conftest.py`**:

    ```python
    """Chaos test fixtures — disposable vaults with isolation guards.

    ALL chaos test vaults MUST use tmp_path and include the isolation assertion:
        assert "tmp" in str(vault)
    """
    ```

    Provide these fixtures:

    1. **`chaos_vault` fixture** (function-scoped, tmp_path):
       - Builds vault at "minimal" level via `VaultBuilder().build("minimal")`
       - Returns Path to vault root
       - **Isolation guard**: `assert "tmp" in str(vault)`
       - Cleanup managed by tmp_path

    2. **`chaos_vault_standard` fixture** (function-scoped, tmp_path):
       - Builds vault at "standard" level (has BBT exports, PDFs, Zotero storage)
       - Returns Path to vault root
       - **Isolation guard**: `assert "tmp" in str(vault)`

    3. **`chaos_cli_invoker` fixture**:
       - Returns `invoke_fn(vault_path, args) -> CompletedProcess`
       - Runs `python -m paperforge --vault {vault} {args}` via subprocess
       - Sets `PAPERFORGE_VAULT` in environment

    Also provide module-level helpers:
    - `corrupt_json(path)` — truncate last 5 chars of a JSON file
    - `corrupt_pdf(path)` — write binary garbage to a PDF file
    - `strip_frontmatter_fields(path, fields)` — remove specific frontmatter fields from a markdown file
    - `create_broken_meta_json(path)` — write invalid JSON to meta.json
    - `setup_vault_from_export(vault, fixture_name)` — helper to copy a specific fixture JSON into vault exports

    **`tests/chaos/test_corrupted_inputs.py`**:

    Mark all tests with `@pytest.mark.chaos`. Each test MUST include the isolation guard.

    ```python
    """CHAOS-01: Corrupted input tests — malformed JSON, corrupt PDF, broken meta.json, missing frontmatter."""
    import pytest
    pytestmark = pytest.mark.chaos
    ```

    Tests:
    1. `test_malformed_bbt_json(chaos_vault, chaos_cli_invoker)`:
       - Copy a valid BBT fixture JSON into vault
       - Call `corrupt_json()` on it to make it malformed
       - Run `chaos_cli_invoker(chaos_vault, ["sync"])`
       - Assert non-zero exit code
       - Assert stderr or stdout contains "Error" or "parsing" (graceful error, not crash)

    2. `test_empty_bbt_json(chaos_vault, chaos_cli_invoker)`:
       - Write `{}` or `{"items": []}` as export JSON
       - Run `chaos_cli_invoker(chaos_vault, ["sync"])`
       - Assert exit code 0 (no crash on empty data)
       - Assert output mentions no items found or empty result

    3. `test_bbt_json_missing_citation_key(chaos_vault_standard, chaos_cli_invoker)`:
       - Start with standard vault (has valid exports)
       - Corrupt one export: load JSON, remove citationKey from one item, write back
       - Run `chaos_cli_invoker(chaos_vault_standard, ["sync"])`
       - Assert exit code 0 (continues processing valid items)
       - Assert output mentions "Skipping" or missing citation key

    4. `test_corrupt_pdf(chaos_vault_standard, chaos_cli_invoker)`:
       - Start with standard vault that has PDFs
       - Call `corrupt_pdf()` on a PDF in Zotero storage
       - Run `chaos_cli_invoker(chaos_vault_standard, ["ocr", "--diagnose"])` or relevant command
       - Assert graceful error message (no crash, no traceback)

    5. `test_broken_meta_json(chaos_vault_standard, chaos_cli_invoker)`:
       - Build vault, create meta.json with invalid JSON
       - Run OCR command with mock backend
       - Assert graceful handling (no crash)

    6. `test_missing_frontmatter_field(chaos_vault_standard, chaos_cli_invoker)`:
       - Build vault at standard level, manually create a formal note missing zotero_key
       - Run `chaos_cli_invoker(chaos_vault_standard, ["status"])`
       - Assert graceful handling, no crash

    Use mock OCR backends where needed. Every test MUST have `assert "tmp" in str(chaos_vault)`.
  </action>
  <verify>
    <automated>python -c "
import sys; from pathlib import Path
sys.path.insert(0, str(Path.cwd()))
for f in ['tests/chaos/__init__.py', 'tests/chaos/conftest.py', 'tests/chaos/test_corrupted_inputs.py']:
    assert Path(f).exists(), f'Missing {f}'
c = Path('tests/chaos/conftest.py').read_text('utf-8')
assert 'chaos_vault' in c and 'chaos_cli_invoker' in c and 'tmp' in c
t = Path('tests/chaos/test_corrupted_inputs.py').read_text('utf-8')
assert 'chaos' in t and 'pytestmark' in t and 'def test_' in t
"</automated>
  </verify>
  <done>
    tests/chaos/ infrastructure created with isolation guards. Corrupted input tests cover malformed JSON, empty JSON, missing citationKey, corrupt PDF, broken meta.json, and missing frontmatter fields.
  </done>
</task>

<task type="auto" tdd="false">
  <name>Task 3: Implement Network Failure & Filesystem Error Chaos Tests</name>
  <files>
    - tests/chaos/test_network_failures.py
    - tests/chaos/test_filesystem_errors.py
  </files>
  <action>
    Implement network failure (CHAOS-02) and filesystem error (CHAOS-03) tests.

    **`tests/chaos/test_network_failures.py`**:

    ```python
    """CHAOS-02: Network failure tests — OCR API timeout, HTTP 401, 500, DNS unreachable."""
    import pytest
    import responses
    from fixtures.ocr.mock_ocr_backend import mock_ocr_error, mock_ocr_timeout

    pytestmark = pytest.mark.chaos
    ```

    Tests:
    1. `test_ocr_api_401(chaos_vault_standard, chaos_cli_invoker)`:
       - Set up a formal note with `do_ocr: true` in vault
       - Use `mock_ocr_error(status=401)` as context manager
       - Run `chaos_cli_invoker(chaos_vault_standard, ["ocr"])`
       - Assert graceful error: stderr or stdout contains "401" or "authentication" or "failed"
       - Assert return code is non-zero on error (or zero if graceful handled — check actual behavior)
       - Assert no crash / no raw traceback

    2. `test_ocr_api_500(chaos_vault_standard, chaos_cli_invoker)`:
       - Use `mock_ocr_error(status=500)` as context manager
       - Run `chaos_cli_invoker(chaos_vault_standard, ["ocr"])`
       - Assert output mentions "500" or "server error"
       - Assert graceful handling

    3. `test_ocr_api_timeout(chaos_vault_standard, chaos_cli_invoker)`:
       - Use `mock_ocr_timeout()` context manager
       - Run `chaos_cli_invoker(chaos_vault_standard, ["ocr"])`
       - Assert output mentions "timeout" or "queued" or did not complete
       - Assert graceful handling

    4. `test_ocr_dns_unreachable(chaos_vault_standard, chaos_cli_invoker)`:
       - Use requests-mock to make all requests raise `ConnectionError`
       - Or use monkeypatch to set PADDLEOCR_JOB_URL to an invalid host
       - Run OCR command
       - Assert output mentions "Network error" or "unable to reach" or connection
       - Assert no raw traceback

    **`tests/chaos/test_filesystem_errors.py`**:

    ```python
    """CHAOS-03: Filesystem error tests — permission denied, locked files, missing directories."""
    import pytest
    from pathlib import Path

    pytestmark = pytest.mark.chaos
    ```

    Tests:
    1. `test_pdf_directory_deleted(chaos_vault_standard, chaos_cli_invoker)`:
       - Delete the Zotero storage directory containing the PDF
       - Run `chaos_cli_invoker(chaos_vault_standard, ["doctor"])` or `["status"]`
       - Assert graceful handling: output mentions "not found" or path_error
       - Assert assert "tmp" in str(chaos_vault_standard)

    2. `test_ocr_directory_deleted(chaos_vault_standard, chaos_cli_invoker)`:
       - Delete the System/PaperForge/ocr directory
       - Run `chaos_cli_invoker(chaos_vault_standard, ["ocr", "--diagnose"])`
       - Assert graceful handling — re-creates dir or prints non-fatal warning

    3. `test_formal_note_deleted_out_of_band(chaos_vault_standard, chaos_cli_invoker)`:
       - Delete a formal note .md file
       - Run `chaos_cli_invoker(chaos_vault_standard, ["repair"])`
       - Assert output mentions divergence or missing note
       - Assert graceful handling, no crash

    4. `test_exports_permission_denied(chaos_vault_standard, chaos_cli_invoker)`:
       - On POSIX: chmod 000 on exports directory
       - On Windows: this may not be feasible via subprocess, so skip with `pytest.mark.skipif`
       - Run `chaos_cli_invoker(chaos_vault_standard, ["sync"])`
       - Assert graceful error message about reading exports directory

    Every test MUST include the isolation guard pattern. Use pytest.approx / in-assertions for output matching (not exact string matching — error messages may change). Prefer checking for keywords or absence of traceback patterns.

    Use `monkeypatch.setattr` or `responses` for network failure simulation. Import from `fixtures.ocr.mock_ocr_backend` — do not redefine.
  </action>
  <verify>
    <automated>python -c "
from pathlib import Path
for f in ['tests/chaos/test_network_failures.py', 'tests/chaos/test_filesystem_errors.py']:
    assert Path(f).exists(), f'Missing {f}'
    content = Path(f).read_text('utf-8')
    assert 'pytestmark' in content
    assert 'chaos' in content
    assert 'def test_' in content
"</automated>
  </verify>
  <done>
    Network failure tests cover 401, 500, timeout, and DNS unreachable using mock_ocr_backend. Filesystem error tests cover deleted directories, missing notes, and permission denied. All tests include isolation guards.
  </done>
</task>

</tasks>

<verification>
1. CHAOS_MATRIX.md documents 15+ destructive scenarios with IDs, triggers, expected behavior, safety contracts
2. `pytest tests/chaos/ -m chaos --tb=short` passes with all tests green
3. Every chaos test uses tmp_path-based vault with `assert "tmp"` isolation guard
4. Network failure tests use mock_ocr_backend (no real HTTP calls)
5. All tests marked `@pytest.mark.chaos` (excluded from default CI gate)
</verification>

<success_criteria>
- [ ] CHAOS_MATRIX.md documents all destructive scenarios with triggers, expected behavior, and safety contracts
- [ ] Corrupted input tests produce graceful error messages — no unhandled crashes
- [ ] Network failure tests produce actionable error messages — no raw tracebacks
- [ ] Filesystem error tests all include `assert "tmp" in str(vault)` — no real vault damage possible
- [ ] All chaos tests pass with `pytest tests/chaos/ -m chaos --tb=short`
</success_criteria>

<output>
After completion, create `.planning/phases/54-dashboard-workflow-closure/54-002-SUMMARY.md`
</output>
