---
phase: 55-ci-optimization-consistency-audit
plan: 001
type: execute
wave: 1
depends_on: []
files_modified:
  - tests/audit/__init__.py
  - tests/audit/conftest.py
  - tests/audit/test_consistency.py
  - pyproject.toml
autonomous: true
requirements:
  - CI-02
  - CI-03

must_haves:
  truths:
    - "Consistency audit detects mock drift — when L1 test mocks no longer match real pipeline output, the audit test fails"
    - "Audit runs against golden dataset fixtures (not against live Zotero/OCR), so it is deterministic and CI-safe"
    - "L1 snapshot contracts (formal note shape, status JSON keys) are validated against real pipeline execution"
    - "OCR mock expectations (response shapes, status transitions) are validated against golden fixture ground truth"
  artifacts:
    - path: "tests/audit/__init__.py"
      provides: "Package init for pytest discovery"
      min_lines: 1
    - path: "tests/audit/conftest.py"
      provides: "Audit test fixtures — VaultBuilder-based full vault, CLI invoker, golden dataset comparators"
      min_lines: 60
    - path: "tests/audit/test_consistency.py"
      provides: "Cross-layer consistency validation tests — run real pipeline, compare with L1 contract snapshots"
      min_lines: 80
    - path: "pyproject.toml"
      provides: "Updated with `audit` marker and `tests/audit` testpath"
  key_links:
    - from: "tests/audit/conftest.py"
      to: "fixtures/vault_builder.py"
      via: "VaultBuilder import"
      pattern: "from fixtures.vault_builder import VaultBuilder"
    - from: "tests/audit/test_consistency.py"
      to: "fixtures/snapshots/"
      via: "Expected output contract validation"
      pattern: "snapshot|contract|compare_output"
    - from: "tests/audit/test_consistency.py"
      to: "paperforge CLI"
      via: "Subprocess invocation, same pattern as L2 CLI tests"
      pattern: "subprocess|cli_invoker"
---

<objective>
**Create the cross-layer consistency audit test suite that validates L1 mock expectations against L4 golden dataset ground truth.**

**Purpose:** Detect "mock drift" before it reaches production. When L1 unit test mocks (fake Zotero JSON, fake configs, mock OCR responses) become stale relative to the real pipeline, this test catches the divergence. The audit acts as a canary — if the golden dataset snapshot contracts don't match real pipeline output, the mocks need updating.

**Output:**
- `tests/audit/__init__.py` — Package init
- `tests/audit/conftest.py` — Audit fixtures (golden vault builder, CLI invoker, expected output comparators)
- `tests/audit/test_consistency.py` — Consistency validation tests
- `pyproject.toml` — Updated with `audit` marker and testpath
</objective>

<execution_context>
@.planning/phases/55-ci-optimization-consistency-audit/55-CONTEXT.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/REQUIREMENTS.md
@.planning/ROADMAP.md

@fixtures/vault_builder.py
@fixtures/MANIFEST.json
@fixtures/snapshots/status_json/empty_vault.json
@fixtures/snapshots/formal_note_frontmatter/orthopedic_article.yaml
@fixtures/snapshots/index_json/after_sync.json

@tests/cli/conftest.py
@tests/cli/test_json_contracts.py
@tests/cli/test_contract_helpers.py
@tests/e2e/conftest.py

@pyproject.toml

<interfaces>
From fixtures/vault_builder.py:
```python
class VaultBuilder:
    def build(self, level: str = "minimal") -> Path:  # "minimal", "standard", or "full"
```

From tests/cli/conftest.py:
```python
@pytest.fixture
def cli_invoker(vault_builder):  # Returns (_invoke_fn) where _invoke(args, vault_level, input_text, env) -> CompletedProcess

@pytest.fixture
def mock_ocr_backend():  # Context manager, yields RequestsMock for HTTP interception

@pytest.fixture
def snapshot_dir():  # Returns Path to fixtures/snapshots/
```

From tests/cli/test_contract_helpers.py (import via tests/cli/test_contract_helpers.py):
```python
def normalize_snapshot(text: str) -> str:  # Normalizes paths, timestamps, versions
def assert_valid_json(output: str) -> dict:  # Asserts valid JSON, returns parsed
def assert_json_shape(data: dict, required: list, optional: list):  # Shape contract validation
```
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Create audit test fixtures — conftest.py + __init__.py</name>
  <files>
    tests/audit/__init__.py
    tests/audit/conftest.py
  </files>
  <action>
    Create `tests/audit/__init__.py` — empty file for pytest package discovery.

    Create `tests/audit/conftest.py` with the following fixtures:

    1. `vault_builder` fixture (session-scoped):
       - Import and return `VaultBuilder` from `fixtures.vault_builder`
       - Same pattern as tests/cli/conftest.py

    2. `cli_invoker` fixture (function-scoped):
       - Returns a callable `_invoke(args, vault_level="full", input_text=None, env=None)` -> subprocess.CompletedProcess
       - Builds a vault at the requested level via `vault_builder.build(level)`
       - Runs `[sys.executable, "-m", "paperforge", "--vault", str(vault)] + args` via subprocess
       - Sets `PAPERFORGE_VAULT` in env
       - Cleans up vault directories on teardown
       - NOTE: Default vault_level is "full" (unlike other conftests that default to "minimal") — audit tests need the richest golden dataset

    3. `golden_vault` fixture (function-scoped):
       - Builds a "full" vault using vault_builder
       - Runs `paperforge sync` against it (real pipeline, L4 path)
       - Returns (invoker, vault_path) tuple for tests
       - Cleans up on teardown

    4. `snapshot_contracts` fixture (function-scoped):
       - Loads all snapshot contracts from `fixtures/snapshots/`
       - Returns dict mapping contract name -> parsed content (JSON for .json, YAML for .yaml)
       - Handles: status_json/empty_vault.json, formal_note_frontmatter/orthopedic_article.yaml, index_json/after_sync.json, paths_json/default_config.json

    5. `golden_dataset_manifest` fixture:
       - Loads and returns `fixtures/MANIFEST.json` as dict
       - Provides fixture metadata (what each fixture tests, which tests use it)

    Import from fixtures/paths relative to repo root (see tests/e2e/conftest.py for the sys.path pattern).
    Use the same Windows-safe cleanup (`_force_rmtree` with `os.chmod(S_IWRITE)`) from tests/e2e/conftest.py.
    Use subprocess for CLI invocation, not in-process Python calls — this is the same L4 boundary the E2E tests use.
  </action>
  <verify>
    <automated>python -c "import sys; sys.path.insert(0, '.'); from tests.audit.conftest import *; print('conftest imports OK')"</automated>
  </verify>
  <done>
    - `tests/audit/__init__.py` exists (empty file)
    - `tests/audit/conftest.py` exists with vault_builder, cli_invoker, golden_vault, snapshot_contracts, golden_dataset_manifest fixtures
    - Fixtures imported cleanly without errors
    - VaultBuilder, VaultBuilder import works from tests/audit/ context
  </done>
</task>

<task type="auto">
  <name>Task 2: Create consistency audit tests + update pyproject.toml</name>
  <files>
    tests/audit/test_consistency.py
    pyproject.toml
  </files>
  <action>
    Create `tests/audit/test_consistency.py` with the following test classes and tests. Each test validates that L1 mock-derived contract snapshots still match what the real L4 pipeline produces:

    ---

    **Class: TestFormalNoteConsistency** (mark: audit)
    Validates that formal note generation from the real pipeline matches L1 snapshot contracts.

    1. `test_formal_note_frontmatter_shape`:
       - Use `golden_vault` fixture to get a synced full vault
       - Find formal notes via glob in `Resources/Literature/**/*.md`
       - Parse frontmatter from first formal note found
       - Assert it contains required keys: `zotero_key`, `domain`, `title`, `year`, `doi`, `has_pdf`, `pdf_path`, `analyze`, `do_ocr`, `ocr_status`
       - Compare against `formal_note_frontmatter/orthopedic_article.yaml` snapshot shape (required + optional keys)
       - FAILS if: pipeline produces different frontmatter fields than what L1 snapshot tests expect (mock drift signal)

    2. `test_formal_note_frontmatter_values`:
       - Parse formal note frontmatter from golden_vault after sync
       - Validate value types: `zotero_key` is string, `year` is string, `has_pdf` is boolean, `do_ocr` is boolean, `ocr_status` is one of ["pending", "processing", "done", "failed"]
       - Same value-level assertions that L1 unit tests perform on mock data
       - FAILS if: real pipeline produces values that violate L1 mock-based assumptions

    ---

    **Class: TestStatusJsonConsistency** (mark: audit)
    Validates that `status --json` output from the real pipeline matches L1 CLI contract snapshots.

    3. `test_status_json_shape`:
       - Run `paperforge status --json` against golden_vault
       - Parse JSON output
       - Assert required keys match `fixtures/snapshots/status_json/empty_vault.json` contract
       - Use `normalize_snapshot` (adapted from tests/cli/test_contract_helpers.py) to normalize dynamic fields (paths, timestamps, versions)
       - FAILS if: real status output shape differs from L1 CLI contract snapshot

    4. `test_status_json_counts_match_expected`:
       - Run `paperforge status --json` against golden_vault
       - Assert `formal_notes` count matches expected number based on golden dataset
       - Expected: the "full" vault level has 1 formal note (FIXT0001) + however many the golden BBT exports generate after sync
       - This is the cross-layer contract check: L1 mocks assume certain counts, L4 pipeline should match

    ---

    **Class: TestSyncPipelineConsistency** (mark: audit)
    Validates that the sync pipeline produces artifacts consistent with L1 test expectations.

    5. `test_sync_produces_index_entry`:
       - Run `paperforge sync` against golden_vault
       - Check canonical index file exists
       - Validate index entry shape matches index_json/after_sync.json snapshot contract
       - FAILS if: index entry structure diverges from L1 snapshot expectations

    6. `test_sync_roundtrip_idempotent`:
       - Run `paperforge sync` twice against golden_vault
       - First run: expect new notes created
       - Second run: expect dry-run (no new notes, zero exit code)
       - FAILS if: second sync is not idempotent (indicates pipeline drift from L1 assumptions)

    ---

    **Class: TestConsistencyAuditSelfTest** (mark: audit)
    Validates that the audit tests themselves are correctly wired.

    7. `test_audit_can_detect_drift`:
       - Modifies a test fixture file to simulate drift (e.g., temporarily add a new required key to formal note contract)
       - Verify that the audit test fails when drift exists
       - Restore fixture file
       - This is a self-test proving the audit mechanism works
       - NOTE: Use careful fixture backup/restore pattern to avoid corrupting golden data

    ---

    **Class: TestOcrMockConsistency** (mark: audit)
    Validates that OCR mock expectations (used by L1 unit tests) match golden fixture ground truth.

    8. `test_ocr_mock_response_shape`:
       - Load mock OCR responses from `fixtures/ocr/paddleocr_submit.json`, `fixtures/ocr/paddleocr_poll_done.json`, `fixtures/ocr/paddleocr_result.json`
       - Validate each response has the expected shape (keys, types) that L1 OCR state machine tests assume
       - Compare with actual OCR state transitions documented in `fixtures/MANIFEST.json`
       - FAILS if: mock response shapes don't match what the real pipeline expects

    9. `test_ocr_fulltext_shape`:
       - Load `fixtures/ocr/extracted_fulltext.md`
       - Validate it has expected structure (--- page markers, text content)
       - Validate `fixtures/ocr/figure_map.json` has expected shape (list of figures with page, path, caption)
       - FAILS if: golden OCR fixture structure contradicts L1 mock assumptions

    ---

    **Marker and testpath setup:**
    Update `pyproject.toml`:

    1. Add `"tests/audit"` to `[tool.pytest.ini_options] testpaths` list (after `"tests/chaos"`).
    2. Add a new marker to the markers list:
       ```
       "audit: Consistency audit tests — validate L1 mocks against L4 real pipeline output",
       ```
    3. Place it after `"chaos"` and before `"slow"` in the markers list.

    All test classes inherit from `object` (not pytest.Unittest). Use `@pytest.mark.audit` on every test function/class. Import `normalize_snapshot` and `assert_json_shape` from `tests/cli/test_contract_helpers.py` (adapt if necessary — they live in tests/cli/ and may need path adjustment for import discovery).

    Use `subprocess.run` with `capture_output=True`, `text=True`, `timeout=120` for CLI invocations. Use `sys.executable` for the Python interpreter, not a hardcoded path.
  </action>
  <verify>
    <automated>python -m pytest tests/audit/ -m audit -v --tb=short --timeout=120 2>&1</automated>
  </verify>
  <done>
    - `tests/audit/test_consistency.py` exists with 9+ consistency audit tests across 5 test classes
    - All tests pass against the golden dataset fixtures
    - `pyproject.toml` updated with `tests/audit` in testpaths and `audit` marker
    - Tests detect mock drift (verified by self-test)
    - `pytest tests/audit/ -m audit` runs cleanly with all tests passing
  </done>
</task>

</tasks>

<verification>
1. `pytest tests/audit/ -m audit -v --tb=short` — all tests pass
2. `pytest tests/audit/ -m audit --co` — collects correct test count
3. `ruff check tests/audit/` — no lint errors
4. `pyproject.toml` has `tests/audit` in testpaths and `audit` marker
5. Audit self-test validates drift detection works
</verification>

<success_criteria>
1. All 9+ consistency audit tests pass against golden dataset fixtures
2. Audit tests detect mock drift (self-test in TestConsistencyAuditSelfTest proves this)
3. L1 formal note frontmatter contract is validated against L4 real pipeline output
4. L1 status JSON contract is validated against L4 real `status --json` output
5. OCR mock response shapes are validated against golden fixture ground truth
6. `pytest tests/audit/` runs as a standalone test collection in CI
7. pyproject.toml is updated with audit marker and testpath
</success_criteria>

<output>
After completion, create `.planning/phases/55-ci-optimization-consistency-audit/55-001-SUMMARY.md`
</output>
