---
phase: 19-testing
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
  - tests/test_e2e_pipeline.py
autonomous: true
requirements:
  - TEST-01

must_haves:
  truths:
    - "selection-sync produces library records with correct frontmatter"
    - "index-refresh produces formal notes with correct frontmatter"
    - "Formal notes contain wikilink to PDF"
    - "OCR queue shows TSTONE001 with ocr_status=done"
    - "A paper without PDF gets ocr_status=nopdf"
    - "Pipeline produces consistent state across sync → index-refresh → queue"
  artifacts:
    - path: "tests/test_e2e_pipeline.py"
      provides: "Full E2E integration tests for the Zotero → sync → index → OCR queue → formal notes pipeline"
      min_lines: 120
  key_links:
    - from: "tests/test_e2e_pipeline.py"
      to: "tests/conftest.py"
      via: "test_vault fixture"
    - from: "tests/test_e2e_pipeline.py"
      to: "paperforge/worker/sync.py"
      via: "run_selection_sync, run_index_refresh"
    - from: "tests/test_e2e_pipeline.py"
      to: "paperforge/worker/_utils.py"
      via: "scan_library_records"
---

<objective>
Create E2E integration tests covering the full Zotero JSON → selection-sync → index-refresh → OCR queue → formal notes pipeline.

Purpose: Validate TEST-01 — the complete pipeline produces correct library records, formal notes, and state transitions when run end-to-end against the existing test_vault fixture.
Output: `tests/test_e2e_pipeline.py` with 2-3 test classes covering the full pipeline.
</objective>

<execution_context>
@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md
</execution_context>

<context>
@tests/conftest.py
@tests/test_smoke.py
@tests/test_selection_sync_pdf.py
@tests/test_repair.py
@paperforge/worker/sync.py
@paperforge/worker/_utils.py
@paperforge/worker/deep_reading.py
@tests/sandbox/exports/骨科.json
@paperforge/config.py
</context>

<interfaces>
Key functions and imports for the E2E tests:

```python
# From paperforge/worker/sync.py:
def run_selection_sync(vault: Path, verbose: bool = False) -> int:
    """Sync Zotero JSON exports → library records. Returns count of written/updated."""
def run_index_refresh(vault: Path, verbose: bool = False) -> int:
    """Sync library records → formal notes. Returns count of index rows written."""

# From paperforge/worker/_utils.py:
def scan_library_records(vault: Path) -> list[dict]:
    """Scan library-records for analyze=true entries.
    Returns list of dicts with keys: zotero_key, domain, title, analyze, do_ocr,
    deep_reading_status, ocr_status, note_path

# From paperforge/config.py:
def paperforge_paths(vault: Path) -> dict[str, Path]:
    """Returns paths dict with keys: vault, system, paperforge, exports, ocr,
    resources, literature, control, library_records, bases, worker_script, etc."""

# From tests/conftest.py:
# test_vault fixture creates a vault with:
# - paperforge.json
# - exports/骨科.json (TSTONE001 only)
# - ocr/TSTONE001/ with fulltext.md, figure-map.json, meta.json (ocr_status: done)
# - library-records/骨科/TSTONE001.md (already exists)
# - Literature/骨科/TSTONE001 - title...md (already exists)
# - Zotero/storage/TSTONE001/TSTONE001.pdf (mock PDF)
```
</interfaces>

<tasks>

<task type="auto">
  <name>Task 1: Create test_e2e_pipeline.py — full pipeline integration tests</name>
  <files>tests/test_e2e_pipeline.py</files>
  <action>
Create `tests/test_e2e_pipeline.py` with 3 test classes that use the `test_vault` fixture from conftest.py:

Test class 1 — TestSelectionSyncE2E: Validates that `run_selection_sync()` produces correct library records with all required frontmatter fields, correct wikilinks, and correct state transitions.

Key fields to validate in library records: zotero_key, domain, title, year, doi, has_pdf, pdf_path (wikilink format), recommend_analyze, ocr_status.

The `test_vault` fixture already has `exports/骨科.json` with TSTONE001 (has PDF). We need to verify the library record that comes out of selection-sync.

HOWEVER: The test_vault fixture already creates a library-record and formal note for TSTONE001. For a clean E2E test, we should either:
  a) Verify the EXISTING record has correct fields (since it was created by an earlier selection-sync), OR  
  b) Delete the existing records and run selection-sync to generate fresh ones

Option (a) is simpler and more reliable. The existing fixture record should match what selection-sync would produce from the same export JSON.

Actually, the cleanest approach: The test fixture creates the vault with pre-seeded records. We verify those records have correct structure AND state transitions. But the records are already there. For a real E2E test, we should run selection-sync on the fixture vault and verify it produces/deduplicates.

Better approach: Run selection-sync on the test_vault. Since the export JSON has TSTONE001 and the library record already exists, selection-sync should update it (not create a duplicate). Then verify the updated record matches expected values.

```python
"""End-to-end integration tests for the PaperForge Lite literature pipeline.

Covers: Zotero JSON export → selection-sync → index-refresh → OCR queue → formal notes.

Uses the existing test_vault fixture from conftest.py which already has:
- paperforge.json with correct paths
- exports/骨科.json with TSTONE001 journal article
- OCR fixture at ocr/TSTONE001/ (ocr_status: done)
- Zotero/storage/TSTONE001/TSTONE001.pdf (mock PDF)
"""

from __future__ import annotations

import re
from pathlib import Path

import pytest

from paperforge.config import paperforge_paths
from paperforge.worker._utils import scan_library_records
from paperforge.worker.sync import run_index_refresh, run_selection_sync


class TestSelectionSyncProducesLibraryRecords:
    """E2E: selection-sync produces correct library records."""

    def test_selection_sync_runs_without_error(self, test_vault: Path) -> None:
        """Verify selection-sync completes without exceptions."""
        count = run_selection_sync(test_vault)
        assert isinstance(count, int)

    def test_library_record_has_required_frontmatter(self, test_vault: Path) -> None:
        """Verify the TSTONE001 library record has all required frontmatter fields."""
        run_selection_sync(test_vault)
        paths = paperforge_paths(test_vault)
        record_path = paths["library_records"] / "骨科" / "TSTONE001.md"
        assert record_path.exists(), "library record should exist after sync"

        text = record_path.read_text(encoding="utf-8")

        # Required fields per AGENTS.md Section 7
        assert 'zotero_key: "TSTONE001"' in text
        assert 'domain: "骨科"' in text
        assert "title:" in text
        assert "year:" in text
        assert "doi:" in text
        assert "has_pdf:" in text
        assert "pdf_path:" in text
        assert "recommend_analyze:" in text
        assert "analyze:" in text
        assert "do_ocr:" in text
        assert "ocr_status:" in text
        assert "deep_reading_status:" in text

    def test_library_record_wikilink_is_valid(self, test_vault: Path) -> None:
        """Verify pdf_path is a valid Obsidian wikilink with forward slashes."""
        run_selection_sync(test_vault)
        paths = paperforge_paths(test_vault)
        record_path = paths["library_records"] / "骨科" / "TSTONE001.md"
        text = record_path.read_text(encoding="utf-8")

        pdf_link_match = re.search(r"pdf_path:\s*\"\[\[(.+?)\]\]\"", text)
        assert pdf_link_match is not None, "pdf_path should be a wikilink"
        pdf_link = pdf_link_match.group(1)

        # Must use forward slashes (per D-03)
        assert "\\" not in pdf_link, f"wikilink should use forward slashes, got: {pdf_link}"
        assert pdf_link.endswith(".pdf"), "wikilink should point to a PDF file"
        assert "TSTONE001" in pdf_link, "wikilink should reference TSTONE001 storage key"

    def test_pdf_with_attachment_has_ocr_status(self, test_vault: Path) -> None:
        """Verify TSTONE001 (has PDF) gets a non-empty ocr_status."""
        run_selection_sync(test_vault)
        paths = paperforge_paths(test_vault)
        record_path = paths["library_records"] / "骨科" / "TSTONE001.md"
        text = record_path.read_text(encoding="utf-8")

        ocr_match = re.search(r"^ocr_status:\s*\"?(\w+)\"?\s*$", text, re.MULTILINE)
        assert ocr_match is not None, "ocr_status should be present"
        ocr_status = ocr_match.group(1)
        assert ocr_status in ("done", "pending", "nopdf"), f"unexpected ocr_status: {ocr_status}"

    def test_first_author_and_journal_present(self, test_vault: Path) -> None:
        """Verify library records include first_author and journal."""
        run_selection_sync(test_vault)
        paths = paperforge_paths(test_vault)
        record_path = paths["library_records"] / "骨科" / "TSTONE001.md"
        text = record_path.read_text(encoding="utf-8")

        fa_match = re.search(r"^first_author:\s*\"?(.+?)\"?$", text, re.MULTILINE)
        assert fa_match is not None, "first_author should be present"
        assert fa_match.group(1).strip(), "first_author should not be empty"

        j_match = re.search(r"^journal:\s*\"?(.+?)\"?$", text, re.MULTILINE)
        assert j_match is not None, "journal should be present"
        assert j_match.group(1).strip(), "journal should not be empty"


class TestIndexRefreshProducesFormalNotes:
    """E2E: index-refresh produces correct formal notes."""

    def test_index_refresh_runs_without_error(self, test_vault: Path) -> None:
        """Verify index-refresh completes without exceptions."""
        count = run_index_refresh(test_vault)
        assert isinstance(count, int)

    def test_formal_note_has_required_frontmatter(self, test_vault: Path) -> None:
        """Verify formal notes have all required frontmatter fields per AGENTS.md Section 7."""
        run_index_refresh(test_vault)
        paths = paperforge_paths(test_vault)
        lit_dir = paths["literature"] / "骨科"
        note_files = list(lit_dir.glob("TSTONE001 - *.md"))
        assert len(note_files) >= 1, "formal note should exist for TSTONE001"

        text = note_files[0].read_text(encoding="utf-8")

        # Required frontmatter fields for formal notes
        assert "title:" in text
        assert "year:" in text
        assert "type:" in text
        assert "journal:" in text
        assert "category:" in text
        assert "tags:" in text
        assert "pdf_link:" in text

    def test_formal_note_pdf_link_is_wikilink(self, test_vault: Path) -> None:
        """Verify pdf_link in formal notes is a valid Obsidian wikilink."""
        run_index_refresh(test_vault)
        paths = paperforge_paths(test_vault)
        lit_dir = paths["literature"] / "骨科"
        note_files = list(lit_dir.glob("TSTONE001 - *.md"))
        text = note_files[0].read_text(encoding="utf-8")

        pdf_link_match = re.search(r"pdf_link:\s*\"\[\[(.+?)\]\]\"", text)
        assert pdf_link_match is not None, "pdf_link should be a wikilink"

        link = pdf_link_match.group(1)
        assert "\\" not in link, f"wikilink should use forward slashes, got: {link}"
        assert link.endswith(".pdf")

    def test_formal_note_has_abstract_content(self, test_vault: Path) -> None:
        """Verify formal note body contains the abstract from the export JSON."""
        run_index_refresh(test_vault)
        paths = paperforge_paths(test_vault)
        lit_dir = paths["literature"] / "骨科"
        note_files = list(lit_dir.glob("TSTONE001 - *.md"))
        text = note_files[0].read_text(encoding="utf-8")

        # Abstract from 骨科.json: "This study compares the biomechanical properties..."
        assert "biomechanical" in text.lower() or "suture anchor" in text.lower()

    def test_formal_note_filename_uses_slug(self, test_vault: Path) -> None:
        """Verify the formal note filename uses slugified title."""
        run_index_refresh(test_vault)
        paths = paperforge_paths(test_vault)
        lit_dir = paths["literature"] / "骨科"
        note_files = list(lit_dir.glob("TSTONE001 - *.md"))
        assert len(note_files) >= 1
        filename = note_files[0].name
        # Filename should start with key + " - " + slug
        assert filename.startswith("TSTONE001 - ")


class TestOcrQueueStates:
    """E2E: OCR queue and state transitions are correct after pipeline run."""

    def test_scan_library_records_returns_tstone001(self, test_vault: Path) -> None:
        """Verify TSTONE001 appears in library records scan with analyze=true."""
        run_selection_sync(test_vault)
        records = scan_library_records(test_vault)
        tstone = [r for r in records if r["zotero_key"] == "TSTONE001"]
        assert len(tstone) >= 1, "TSTONE001 should be in library records scan"
        assert tstone[0]["analyze"] is True, "TSTONE001 should have analyze=true"
        assert tstone[0]["domain"] == "骨科"

    def test_tstone001_ocr_status_done(self, test_vault: Path) -> None:
        """Verify TSTONE001 has OCR status 'done' (fixture has meta.json with ocr_status: done)."""
        run_selection_sync(test_vault)
        records = scan_library_records(test_vault)
        tstone = [r for r in records if r["zotero_key"] == "TSTONE001"]
        assert len(tstone) >= 1
        assert tstone[0]["ocr_status"] == "done", (
            f"Expected ocr_status=done, got: {tstone[0]['ocr_status']}"
        )

    def test_full_pipeline_consistency(self, test_vault: Path) -> None:
        """Run full pipeline and verify consistency across all stages."""
        # Stage 1: selection-sync produces library record
        run_selection_sync(test_vault)
        paths = paperforge_paths(test_vault)
        record_path = paths["library_records"] / "骨科" / "TSTONE001.md"
        assert record_path.exists(), "Stage 1: library record should exist"

        # Stage 2: index-refresh produces formal note
        run_index_refresh(test_vault)
        lit_dir = paths["literature"] / "骨科"
        note_files = list(lit_dir.glob("TSTONE001 - *.md"))
        assert len(note_files) >= 1, "Stage 2: formal note should exist"

        # Stage 3: Formal note frontmatter matches library record
        note_text = note_files[0].read_text(encoding="utf-8")
        record_text = record_path.read_text(encoding="utf-8")

        # Extract titles from both
        note_title = re.search(r'^title:\s*"(.+?)"', note_text, re.MULTILINE)
        record_title = re.search(r'^title:\s*"(.+?)"', record_text, re.MULTILINE)
        if note_title and record_title:
            assert note_title.group(1) == record_title.group(1), (
                "Formal note title should match library record title"
            )

        # Stage 4: OCR state is consistent
        records = scan_library_records(test_vault)
        tstone = [r for r in records if r["zotero_key"] == "TSTONE001"]
        assert len(tstone) >= 1
        assert tstone[0]["ocr_status"] == "done", "Stage 4: OCR status should be done"

        # Stage 5: Formal note links to correct PDF location
        pdf_link_match = re.search(r'pdf_link:\s*"\[\[(.+?)\]\]"', note_text, re.MULTILINE)
        assert pdf_link_match is not None, "Stage 5: pdf_link should be in formal note"
        assert "TSTONE001.pdf" in pdf_link_match.group(1), (
            "Stage 5: pdf_link should reference correct PDF"
        )
```

IMPORTANT IMPLEMENTATION DETAILS:
- Use `test_vault` fixture from conftest.py (auto-imported via pytest)
- `run_selection_sync` and `run_index_refresh` both accept `vault: Path` and return `int`
- `scan_library_records` from `_utils.py` is the canonical queue scan (merged in v1.5 Phase 15)
- `paperforge_paths` from `config.py` returns a dict with `library_records`, `literature`, `ocr` keys
- The fixture export JSON (骨科.json) has TSTONE001 with `storage:TSTONE001/TSTONE001.pdf` attachment path
- The fixture creates `ocr/TSTONE001/meta.json` with `ocr_status: "done"`
- Do NOT import or call `run_ocr` from ocr.py — that would require network access. OCR is validated via meta.json state only.
- All tests must pass with `python -m pytest tests/test_e2e_pipeline.py -x -q`
- After this task, run the full test suite to verify no regressions: `python -m pytest tests/ -x -q`
  </action>
  <verify>
    <automated>python -m pytest tests/test_e2e_pipeline.py -x -q</automated>
  </verify>
  <done>All E2E tests pass: selection-sync produces correct library records, index-refresh produces correct formal notes, OCR queue shows correct states, full pipeline consistency validated</done>
</task>

</tasks>

<verification>
```bash
python -m pytest tests/test_e2e_pipeline.py -x -q
python -m pytest tests/ -x -q  # Verify 0 regressions
```
</verification>

<success_criteria>
1. `tests/test_e2e_pipeline.py` exists with 3 test classes covering selection-sync, index-refresh, and OCR queue
2. All tests pass against the test_vault fixture
3. No existing tests broken: `pytest tests/ -x -q` passes with 205+ tests
4. Library record frontmatter completeness validated
5. Formal note frontmatter completeness validated
6. Wikilink correctness validated (forward slashes, .pdf extension)
7. OCR state transitions validated (done/pending/nopdf)
</success_criteria>

<output>
After completion, create `.planning/phases/19-testing/19-02-SUMMARY.md`
</output>
