docs: add pdf annotation layer design spec and implementation plan

2026-07-22 06:50:53 +00:00 · 2026-05-20 15:33:13 +08:00 · 2026-05-20 15:33:13 +08:00 · 3c270c6932
commit 3c270c6932
parent 63c6be3f78
2 changed files with 1286 additions and 0 deletions
--- a/docs/superpowers/plans/2026-05-20-pdf-annotation-layer.md
+++ b/docs/superpowers/plans/2026-05-20-pdf-annotation-layer.md
@ -0,0 +1,787 @@
+# PaperForge PDF Annotation Layer Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Build a read-only Zotero annotation import pipeline, persist normalized annotations in a dedicated PaperForge database, surface them through CLI JSON contracts, and render/edit them in Obsidian's native PDF viewer via overlay patching.
+
+**Architecture:** The work is split into five sequential packages. Package A adds a dedicated annotation database and schema lifecycle. Package B implements read-only Zotero SQLite probing and import/update behavior. Package C exposes annotation commands through the CLI with stable JSON envelopes. Package D extends the current plain `paperforge/plugin/main.js` plugin with subprocess-backed annotation loading and viewer overlay patching. Package E prepares the future write-back lane without implementing Web API push yet.
+
+**Tech Stack:** Python 3.10+, SQLite/WAL/FTS5, argparse CLI, Obsidian plugin CommonJS (`main.js`), Node child_process, Vitest, pytest
+
+---
+
+## File Map
+
+### New Python Annotation Files
+
+- `paperforge/annotation/__init__.py` — annotation package marker and narrow public exports
+- `paperforge/annotation/db.py` — `annotations.db` path resolution and connection helpers
+- `paperforge/annotation/schema.py` — annotation schema DDL, versioning, migrations, FTS triggers
+- `paperforge/annotation/probe.py` — read-only Zotero SQLite access and normalization helpers
+- `paperforge/annotation/importer.py` — import/upsert/reconciliation logic from Zotero rows to PaperForge rows
+- `paperforge/annotation/service.py` — CRUD/search/export operations against `annotations.db`
+
+### New / Modified CLI Files
+
+- `paperforge/commands/annotation.py` — new command family implementation
+- `paperforge/cli.py` — register `annotation` subcommands and flags
+- `paperforge/core/result.py` — reuse existing result envelope only if a small helper addition is needed
+- `paperforge/config.py` — only if Zotero data dir resolution needs a shared helper
+
+### Plugin Files
+
+- `paperforge/plugin/main.js` — add annotation subprocess bridge, viewer patch lifecycle, overlay UI wiring
+- `paperforge/plugin/src/testable.js` — extract pure helper functions for argument building, annotation payload parsing, and coordinate helpers where possible
+- `paperforge/plugin/styles.css` — overlay, popover, readonly-lock, and action styles
+- `paperforge/plugin/package.json` — add `monkey-around` only if needed for patching helper ergonomics
+
+### New / Modified Tests
+
+- `fixtures/zotero/test_annotations.sqlite` — minimal Zotero SQLite fixture for importer/integration coverage
+- `tests/unit/annotation/test_schema.py` — new schema creation/version tests
+- `tests/unit/annotation/test_probe.py` — read-only SQLite normalization tests
+- `tests/unit/annotation/test_importer.py` — import/update/delete reconciliation tests
+- `tests/unit/annotation/test_service.py` — create/patch/delete/list/export behavior tests
+- `tests/cli/test_annotation_commands.py` — CLI envelope and argument validation tests
+- `tests/integration/test_annotation_import_workflow.py` — fixture-driven import and rebuild-survival tests
+- `paperforge/plugin/tests/runtime.test.mjs` — extend with annotation runtime path/config helpers if needed
+- `paperforge/plugin/tests/commands.test.mjs` — subprocess argument construction for annotation commands
+- `paperforge/plugin/tests/errors.test.mjs` — plugin-side error handling for annotation subprocess and patch failures
+
+### Existing Reference Files To Read During Execution
+
+- `paperforge/memory/db.py` — existing SQLite connection patterns
+- `paperforge/memory/schema.py` — versioning and trigger style reference
+- `paperforge/pdf_resolver.py` — PDF path resolution rules
+- `paperforge/plugin/main.js` — current plugin architecture and runtime helpers
+- `paperforge/plugin/src/testable.js` — current pure helper extraction pattern
+- `tests/unit/memory/test_schema.py` — schema testing style
+- `tests/cli/test_json_contracts.py` — JSON contract assertions pattern
+
+---
+
+## Package A: Dedicated Annotation Database
+
+### Task A1: Add Annotation DB Path and Connection Helpers
+
+**Files:**
+- Create: `paperforge/annotation/__init__.py`
+- Create: `paperforge/annotation/db.py`
+- Test: `tests/unit/annotation/test_schema.py`
+
+- [ ] **Step 1: Write the failing tests for DB path and connection behavior**
+
+Cover:
+
+- `annotations.db` resolves under `System/PaperForge/indexes/`
+- read/write connection enables WAL and foreign keys
+- read-only connection uses SQLite URI `mode=ro`
+
+- [ ] **Step 2: Run the new targeted tests to verify failure**
+
+Run: `python -m pytest tests/unit/annotation/test_schema.py -v --tb=short`
+
+Expected: FAIL because annotation DB helpers do not exist yet.
+
+- [ ] **Step 3: Implement minimal path and connection helpers**
+
+Follow the `paperforge/memory/db.py` style:
+
+- `get_annotations_db_path(vault: Path) -> Path`
+- `get_annotations_connection(db_path: Path, read_only: bool = False) -> sqlite3.Connection`
+
+- [ ] **Step 4: Re-run the targeted tests**
+
+Run: `python -m pytest tests/unit/annotation/test_schema.py -v --tb=short`
+
+Expected: PASS for the new DB helper assertions.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add paperforge/annotation/__init__.py paperforge/annotation/db.py tests/unit/annotation/test_schema.py
+git commit -m "feat: add annotation db helpers"
+```
+
+### Task A2: Add Annotation Schema and Version Lifecycle
+
+**Files:**
+- Create: `paperforge/annotation/schema.py`
+- Modify: `tests/unit/annotation/test_schema.py`
+
+- [ ] **Step 1: Add failing schema lifecycle tests**
+
+Test:
+
+- schema bootstrap creates `meta`, `annotations`, `annotations_fts`, `sync_queue`
+- schema version is stored as `1`
+- `ensure_schema()` is idempotent
+
+- [ ] **Step 2: Run the targeted schema tests**
+
+Run: `python -m pytest tests/unit/annotation/test_schema.py -v --tb=short`
+
+Expected: FAIL because tables and version logic do not exist.
+
+- [ ] **Step 3: Implement the minimal schema module**
+
+Include:
+
+- `ANNOTATIONS_SCHEMA_VERSION = 1`
+- DDL matching the columns, indexes, and FTS requirements defined in `docs/superpowers/specs/2026-05-20-pdf-annotation-layer-design.md` Section 8
+- FTS triggers
+- `ensure_schema(conn)`
+- `get_schema_version(conn)`
+
+- [ ] **Step 4: Re-run the schema tests**
+
+Run: `python -m pytest tests/unit/annotation/test_schema.py -v --tb=short`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add paperforge/annotation/schema.py tests/unit/annotation/test_schema.py
+git commit -m "feat: add annotation schema"
+```
+
+---
+
+## Package B: Read-Only Zotero SQLite Probe and Import
+
+### Task B1: Implement Read-Only Zotero Annotation Probe
+
+**Files:**
+- Create: `paperforge/annotation/probe.py`
+- Create: `tests/unit/annotation/test_probe.py`
+- Reference: `experiments/zotero_annotation_probe.py`
+
+- [ ] **Step 1: Write failing tests for probe normalization**
+
+Cover:
+
+- annotation type integer → string mapping
+- `position` JSON parsing
+- tag aggregation
+- read-only connection open path
+- invalid schema error handling
+
+- [ ] **Step 2: Run the probe tests to verify failure**
+
+Run: `python -m pytest tests/unit/annotation/test_probe.py -v --tb=short`
+
+Expected: FAIL because the probe module does not exist.
+
+- [ ] **Step 3: Implement minimal probe behavior**
+
+Add:
+
+- `copy_db_to_temp()`
+- `open_readonly()`
+- `fetch_annotations()`
+- `fetch_annotation_tags()`
+- `normalize_zotero_annotation_row()`
+
+Use parameterized SQL only.
+
+Copy-to-temp must be the default behavior. A future `--no-copy` CLI flag is the explicit escape hatch.
+
+- [ ] **Step 4: Re-run the probe tests**
+
+Run: `python -m pytest tests/unit/annotation/test_probe.py -v --tb=short`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add paperforge/annotation/probe.py tests/unit/annotation/test_probe.py
+git commit -m "feat: add zotero annotation probe"
+```
+
+### Task B1b: Add Fixture-Driven Zotero Import Coverage
+
+**Files:**
+- Create: `fixtures/zotero/test_annotations.sqlite`
+- Create: `tests/integration/test_annotation_import_workflow.py`
+
+- [ ] **Step 1: Write the failing fixture-driven import test**
+
+Cover one end-to-end import from a real SQLite fixture containing at least:
+
+- highlight with multi-rect position
+- note annotation
+- underline annotation
+- tags
+- non-ASCII text
+
+- [ ] **Step 2: Run the focused integration test**
+
+Run: `python -m pytest tests/integration/test_annotation_import_workflow.py -v --tb=short`
+
+Expected: FAIL because the fixture and end-to-end import path are not wired yet.
+
+- [ ] **Step 3: Build the minimal SQLite fixture and test harness**
+
+Use the real Zotero schema subset needed by the importer so the test exercises actual SQL joins instead of mocks.
+
+- [ ] **Step 4: Re-run the focused integration test**
+
+Run: `python -m pytest tests/integration/test_annotation_import_workflow.py -v --tb=short`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add fixtures/zotero/test_annotations.sqlite tests/integration/test_annotation_import_workflow.py
+git commit -m "test: add fixture-driven annotation import coverage"
+```
+
+### Task B2: Implement Import/Reconciliation Logic
+
+**Files:**
+- Create: `paperforge/annotation/importer.py`
+- Create: `tests/unit/annotation/test_importer.py`
+- Modify: `paperforge/annotation/schema.py` only if an index/helper is missing
+
+- [ ] **Step 1: Write failing importer tests**
+
+Cover:
+
+- first import inserts new rows with `source='zotero_db'` and `sync_state='zotero_synced'`
+- re-import with unchanged source is a no-op
+- re-import with changed source updates payload and timestamps
+- missing source annotation soft-deletes stale row if not locally modified
+- locally modified stale row becomes `conflict` instead of blind delete
+
+- [ ] **Step 2: Run the importer tests**
+
+Run: `python -m pytest tests/unit/annotation/test_importer.py -v --tb=short`
+
+Expected: FAIL because the importer does not exist.
+
+- [ ] **Step 3: Implement minimal reconciliation logic**
+
+Add:
+
+- stable upsert by `(source, zotero_key)`
+- source version + modified time tracking
+- soft-delete rules
+- conflict marking rules
+
+- [ ] **Step 4: Re-run the importer tests**
+
+Run: `python -m pytest tests/unit/annotation/test_importer.py -v --tb=short`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add paperforge/annotation/importer.py tests/unit/annotation/test_importer.py
+git commit -m "feat: add annotation import reconciliation"
+```
+
+### Task B3: Implement Local Annotation CRUD/Search/Export Service
+
+**Files:**
+- Create: `paperforge/annotation/service.py`
+- Create: `tests/unit/annotation/test_service.py`
+
+- [ ] **Step 1: Write failing service tests**
+
+Cover:
+
+- create local annotation
+- patch local comment/color
+- soft delete local annotation
+- list by paper/page
+- export JSON
+- export Markdown
+- reject patching imported readonly rows unless an explicit future override exists
+
+- [ ] **Step 2: Run the service tests**
+
+Run: `python -m pytest tests/unit/annotation/test_service.py -v --tb=short`
+
+Expected: FAIL because the service module does not exist.
+
+- [ ] **Step 3: Implement the minimal service layer**
+
+Expose:
+
+- `create_annotation(...)`
+- `patch_annotation(...)`
+- `delete_annotation(...)`
+- `list_annotations(...)`
+- `export_annotations_json(...)`
+- `export_annotations_markdown(...)`
+
+- [ ] **Step 4: Re-run the service tests**
+
+Run: `python -m pytest tests/unit/annotation/test_service.py -v --tb=short`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add paperforge/annotation/service.py tests/unit/annotation/test_service.py
+git commit -m "feat: add annotation service"
+```
+
+---
+
+## Package C: CLI Surface and JSON Contracts
+
+### Task C1: Add `paperforge annotation` Parser Surface
+
+**Files:**
+- Modify: `paperforge/cli.py`
+- Create: `paperforge/commands/annotation.py`
+- Create: `tests/cli/test_annotation_commands.py`
+
+- [ ] **Step 1: Write failing CLI parser and envelope tests**
+
+Cover:
+
+- subcommands: `import`, `list`, `create`, `patch`, `delete`, `export`, `status`
+- required args and mutually exclusive flags
+- `--json` envelope structure
+
+- [ ] **Step 2: Run the CLI tests**
+
+Run: `python -m pytest tests/cli/test_annotation_commands.py -v --tb=short`
+
+Expected: FAIL because the parser and command module do not exist.
+
+- [ ] **Step 3: Implement minimal parser + dispatch**
+
+Add the new top-level `annotation` subparser in `paperforge/cli.py` and implement the command module that delegates to the new annotation service/importer.
+
+Treat this CLI argument and JSON envelope contract as frozen before plugin Package D2 begins.
+
+- [ ] **Step 4: Re-run the CLI tests**
+
+Run: `python -m pytest tests/cli/test_annotation_commands.py -v --tb=short`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add paperforge/cli.py paperforge/commands/annotation.py tests/cli/test_annotation_commands.py
+git commit -m "feat: add annotation cli"
+```
+
+### Task C2: Add Targeted End-to-End JSON Contract Tests
+
+**Files:**
+- Modify: `tests/cli/test_json_contracts.py`
+- Modify: `tests/test_e2e_cli.py` or create a small focused annotation E2E test if cleaner
+
+- [ ] **Step 1: Write failing JSON contract assertions for annotation commands**
+
+Cover:
+
+- `annotation status --json`
+- `annotation list ... --json`
+- `annotation create ... --json`
+- error envelope when Zotero DB is missing
+
+- [ ] **Step 2: Run the focused JSON contract tests**
+
+Run: `python -m pytest tests/cli/test_json_contracts.py tests/test_e2e_cli.py -v --tb=short`
+
+Expected: FAIL on missing contract coverage or incorrect envelope shape.
+
+- [ ] **Step 3: Implement the minimal envelope fixes**
+
+Keep the command outputs aligned with existing `PFResult`-style expectations where practical.
+
+- [ ] **Step 4: Re-run the focused JSON contract tests**
+
+Run: `python -m pytest tests/cli/test_json_contracts.py tests/test_e2e_cli.py -v --tb=short`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add tests/cli/test_json_contracts.py tests/test_e2e_cli.py paperforge/commands/annotation.py
+git commit -m "test: cover annotation json contracts"
+```
+
+### Task C3: Prove Annotation Persistence Survives Memory Rebuild
+
+**Files:**
+- Modify: `tests/integration/test_annotation_import_workflow.py`
+
+- [ ] **Step 1: Add the failing rebuild-survival test**
+
+Cover:
+
+- import annotations from the SQLite fixture
+- run `paperforge memory build`
+- run annotation list again
+- assert imported annotations still exist unchanged in `annotations.db`
+
+- [ ] **Step 2: Run the focused integration test file**
+
+Run: `python -m pytest tests/integration/test_annotation_import_workflow.py -v --tb=short`
+
+Expected: FAIL if any code path incorrectly couples `annotations.db` to memory rebuild lifecycle.
+
+- [ ] **Step 3: Fix lifecycle boundaries if needed**
+
+The expected fix should be minimal: ensure annotation DB code is fully separate from `paperforge.memory.*` rebuild paths.
+
+- [ ] **Step 4: Re-run the focused integration test file**
+
+Run: `python -m pytest tests/integration/test_annotation_import_workflow.py -v --tb=short`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add tests/integration/test_annotation_import_workflow.py
+git commit -m "test: verify annotation persistence across memory rebuild"
+```
+
+---
+
+## Package D: Obsidian Plugin Overlay and Local Editing
+
+### Task D1: Extend Plugin Testable Helpers for Annotation Runtime
+
+**Files:**
+- Modify: `paperforge/plugin/src/testable.js`
+- Modify: `paperforge/plugin/tests/commands.test.mjs`
+- Modify: `paperforge/plugin/tests/errors.test.mjs`
+
+- [ ] **Step 1: Write failing plugin helper tests**
+
+Cover:
+
+- annotation command argument builders
+- parsing of CLI JSON envelopes
+- readonly/local annotation classification
+- safe fallback when annotation subprocess fails
+
+- [ ] **Step 2: Run the focused plugin tests**
+
+Run: `cd paperforge/plugin && npx vitest run tests/commands.test.mjs tests/errors.test.mjs`
+
+Expected: FAIL because the helper functions do not exist.
+
+- [ ] **Step 3: Add pure helper functions to `src/testable.js`**
+
+Add only pure logic here, for example:
+
+- `buildAnnotationListArgs()`
+- `buildAnnotationCreateArgs()`
+- `parseAnnotationResult()`
+- `isReadonlyAnnotation()`
+
+- [ ] **Step 4: Re-run the plugin tests**
+
+Run: `cd paperforge/plugin && npx vitest run tests/commands.test.mjs tests/errors.test.mjs`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add paperforge/plugin/src/testable.js paperforge/plugin/tests/commands.test.mjs paperforge/plugin/tests/errors.test.mjs
+git commit -m "feat: add annotation plugin helpers"
+```
+
+### Task D2: Add Annotation Subprocess Bridge to the Plugin
+
+**Files:**
+- Modify: `paperforge/plugin/main.js`
+- Modify: `paperforge/plugin/tests/runtime.test.mjs`
+
+- [ ] **Step 1: Write failing runtime tests for annotation bridge behavior**
+
+Cover:
+
+- plugin resolves Python executable and calls `paperforge annotation list`
+- plugin handles JSON parse failures safely
+- plugin degrades cleanly when CLI returns error
+
+- [ ] **Step 2: Run the focused runtime tests**
+
+Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs tests/errors.test.mjs`
+
+Expected: FAIL because the bridge does not exist.
+
+- [ ] **Step 3: Implement minimal annotation bridge functions in `main.js`**
+
+Add small functions rather than a big framework:
+
+- `runAnnotationCommand(...)`
+- `fetchAnnotationsForPaper(...)`
+- `createLocalAnnotation(...)`
+- `patchLocalAnnotation(...)`
+- `deleteLocalAnnotation(...)`
+
+Fetch all annotations for the active paper once on document open and cache them in memory grouped by `page_index`. Do not spawn subprocesses on every page render event.
+
+- [ ] **Step 4: Re-run the focused runtime tests**
+
+Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs tests/errors.test.mjs`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add paperforge/plugin/main.js paperforge/plugin/tests/runtime.test.mjs paperforge/plugin/tests/errors.test.mjs
+git commit -m "feat: add annotation plugin bridge"
+```
+
+### Task D3: Add Native PDF Viewer Patch Lifecycle
+
+**Files:**
+- Modify: `paperforge/plugin/main.js`
+- Modify: `paperforge/plugin/package.json` only if `monkey-around` is needed
+- Modify: `paperforge/plugin/tests/errors.test.mjs`
+
+- [ ] **Step 1: Write failing tests for patch setup and soft-failure behavior**
+
+Cover:
+
+- plugin does not crash when PDF internals are unavailable
+- patch init exits early outside desktop / unsupported state
+- patch failure surfaces a controlled warning instead of breaking startup
+
+- [ ] **Step 2: Run the focused patch-failure tests**
+
+Run: `cd paperforge/plugin && npx vitest run tests/errors.test.mjs`
+
+Expected: FAIL because no annotation patch lifecycle exists.
+
+- [ ] **Step 3: Implement minimal patch registration**
+
+Inside `main.js`, add a narrow patch layer that:
+
+- detects active `pdf` leaves
+- resolves internal viewer prototypes/classes for the native PDF view and uses the PDF.js event bus events called out in the spec (`textlayerrendered`, `annotationlayerrendered`, `pagerendered`, `scalechanged`)
+- installs patched `load` / `loadFile` / cleanup behavior
+- bails out safely if internals are missing
+
+Keep the patching additive and local. `paperforge/plugin/main.js` is already monolithic, so avoid introducing another large in-file subsystem unless the logic cannot be expressed as narrow helper functions.
+
+- [ ] **Step 4: Re-run the patch-failure tests**
+
+Run: `cd paperforge/plugin && npx vitest run tests/errors.test.mjs`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add paperforge/plugin/main.js paperforge/plugin/package.json paperforge/plugin/tests/errors.test.mjs
+git commit -m "feat: add pdf overlay patch lifecycle"
+```
+
+### Task D4: Render Overlays for Imported Annotations
+
+**Files:**
+- Modify: `paperforge/plugin/main.js`
+- Modify: `paperforge/plugin/styles.css`
+- Modify: `paperforge/plugin/src/testable.js`
+- Modify: `paperforge/plugin/tests/commands.test.mjs`
+
+- [ ] **Step 1: Write failing tests for coordinate conversion and render payload shaping**
+
+Cover:
+
+- page grouping by `page_index`
+- rect normalization to overlay percentages
+- readonly annotation style classification
+
+- [ ] **Step 2: Run the focused plugin tests**
+
+Run: `cd paperforge/plugin && npx vitest run tests/commands.test.mjs tests/runtime.test.mjs`
+
+Expected: FAIL because conversion/render helpers are incomplete.
+
+- [ ] **Step 3: Implement minimal render helpers and overlay styles**
+
+In `main.js`:
+
+- build per-page overlay containers
+- render highlight/underline/note rects
+- attach hover/click listeners
+
+In `styles.css`:
+
+- add overlay, rect, popover, readonly-lock styles
+
+Put pure math/string helpers in `src/testable.js` when possible.
+
+- [ ] **Step 4: Re-run the focused plugin tests**
+
+Run: `cd paperforge/plugin && npx vitest run tests/commands.test.mjs tests/runtime.test.mjs`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add paperforge/plugin/main.js paperforge/plugin/styles.css paperforge/plugin/src/testable.js paperforge/plugin/tests/commands.test.mjs paperforge/plugin/tests/runtime.test.mjs
+git commit -m "feat: render annotation overlays"
+```
+
+### Task D5: Add Local Annotation Create/Edit/Delete UI Flow
+
+**Files:**
+- Modify: `paperforge/plugin/main.js`
+- Modify: `paperforge/plugin/styles.css`
+- Modify: `paperforge/plugin/tests/commands.test.mjs`
+- Modify: `paperforge/plugin/tests/errors.test.mjs`
+
+- [ ] **Step 1: Write failing tests for local annotation command flow**
+
+Cover:
+
+- selection payload becomes `annotation create` CLI args
+- popover edit action becomes `annotation patch`
+- delete action becomes `annotation delete`
+- readonly rows never expose edit actions
+
+- [ ] **Step 2: Run the focused plugin tests**
+
+Run: `cd paperforge/plugin && npx vitest run tests/commands.test.mjs tests/errors.test.mjs`
+
+Expected: FAIL because local annotation UI flow is not wired.
+
+- [ ] **Step 3: Implement the minimal local action flow**
+
+Add to `main.js`:
+
+- selection capture
+- floating create affordance or context action
+- popover with edit/delete buttons for local annotations only
+- immediate overlay refresh after successful subprocess call
+
+- [ ] **Step 4: Re-run the focused plugin tests**
+
+Run: `cd paperforge/plugin && npx vitest run tests/commands.test.mjs tests/errors.test.mjs`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add paperforge/plugin/main.js paperforge/plugin/styles.css paperforge/plugin/tests/commands.test.mjs paperforge/plugin/tests/errors.test.mjs
+git commit -m "feat: add local annotation editing"
+```
+
+---
+
+## Package E: Future Write-Back Readiness
+
+### Task E1: Add Explicit Pending-Push and Queue Semantics Without API Push
+
+**Files:**
+- Modify: `paperforge/annotation/schema.py`
+- Modify: `paperforge/annotation/service.py`
+- Modify: `tests/unit/annotation/test_service.py`
+
+- [ ] **Step 1: Write failing tests for queue-ready local edits**
+
+Cover:
+
+- local edits can mark rows as `pending_push`
+- `sync_queue` receives create/update/delete payloads only when write-back mode is explicitly enabled later
+- V1 default behavior remains local-only without silently pushing
+
+- [ ] **Step 2: Run the focused service tests**
+
+Run: `python -m pytest tests/unit/annotation/test_service.py -v --tb=short`
+
+Expected: FAIL if the service cannot express future queue semantics.
+
+- [ ] **Step 3: Implement the minimal readiness hooks**
+
+Do not implement Web API logic yet. Only ensure the schema/service can represent future push state without a schema rewrite.
+
+- [ ] **Step 4: Re-run the focused service tests**
+
+Run: `python -m pytest tests/unit/annotation/test_service.py -v --tb=short`
+
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add paperforge/annotation/schema.py paperforge/annotation/service.py tests/unit/annotation/test_service.py
+git commit -m "feat: prepare annotation push states"
+```
+
+---
+
+## Full Verification Slice
+
+- [ ] **Step 1: Run annotation Python test slice**
+
+Run:
+
+```bash
+python -m pytest tests/unit/annotation/ tests/integration/test_annotation_import_workflow.py tests/cli/test_annotation_commands.py tests/cli/test_json_contracts.py -v --tb=short
+```
+
+Expected: PASS.
+
+- [ ] **Step 2: Run plugin test slice**
+
+Run:
+
+```bash
+cd paperforge/plugin && npx vitest run tests/runtime.test.mjs tests/commands.test.mjs tests/errors.test.mjs
+```
+
+Expected: PASS.
+
+- [ ] **Step 3: Run broader PaperForge regression slice**
+
+Run:
+
+```bash
+python -m pytest tests/unit/memory/test_schema.py tests/test_config.py tests/test_e2e_cli.py -v --tb=short
+```
+
+Expected: PASS.
+
+- [ ] **Step 4: Run the standard project regression commands from the repo guide**
+
+Run:
+
+```bash
+python -m pytest tests/unit/ tests/cli/ -v --tb=short
+```
+
+Run:
+
+```bash
+cd paperforge/plugin && npx vitest run
+```
+
+Expected: PASS.
+
+- [ ] **Step 5: Record completion checkpoint**
+
+Expected state:
+
+- `annotations.db` exists and is schema-managed
+- Zotero SQLite import is read-only and fixture-tested
+- `paperforge annotation ...` commands are live with JSON contracts
+- plugin can fetch and render overlays in the native PDF view
+- local annotation create/edit/delete is supported
+- write-back path is designed in schema/service state, but not implemented yet
--- a/docs/superpowers/specs/2026-05-20-pdf-annotation-layer-design.md
+++ b/docs/superpowers/specs/2026-05-20-pdf-annotation-layer-design.md
@ -0,0 +1,499 @@
+# PaperForge PDF Annotation Layer Design
+
+**Date:** 2026-05-20
+**Status:** Proposed
+**Audience:** Maintainers, contributors, agentic implementers
+
+---
+
+## 1. Summary
+
+PaperForge should add a lightweight PDF annotation layer that treats Zotero as an upstream annotation source instead of trying to become a full Zotero replacement.
+
+The chosen design is:
+
+1. Read Zotero annotations from local `zotero.sqlite` in **read-only** mode.
+2. Cache normalized annotations in an independent `annotations.db` under `System/PaperForge/indexes/`.
+3. Display those annotations in Obsidian by **monkey-patching the native PDF viewer**, following the proven PDF++ approach.
+4. Allow local PaperForge-native annotation creation and editing inside Obsidian.
+5. Design write-back from day one, but route future write-back through the **Zotero Web API**, not direct SQLite writes.
+
+This explicitly avoids building a full ZotFlow-style system with its own embedded reader, large sync engine, React-heavy UI surface, or end-to-end Zotero library replacement.
+
+---
+
+## 2. Product Decision
+
+### Chosen Architecture
+
+- **Read path:** local Zotero SQLite, read-only
+- **Cache:** independent `annotations.db`
+- **Viewer integration:** patch Obsidian's native PDF viewer
+- **Write-back design:** future Zotero Web API push
+- **Version 1 scope:** import, cache, display, create local annotations, edit local comments/colors
+
+### Why
+
+This design gives PaperForge the strongest trade-off across speed, safety, and user experience:
+
+- Local SQLite gives fast, offline, zero-config reads.
+- Independent DB storage prevents memory rebuild from destroying user annotations.
+- Native PDF viewer patching preserves Obsidian UX instead of fragmenting it with a second reader.
+- Web API write-back avoids the corruption and sync-state hazards of SQLite mutation.
+
+---
+
+## 3. Problem Statement
+
+PaperForge currently has no structured annotation layer. The system can manage papers, OCR, memory state, and plugin runtime state, but it cannot:
+
+1. Read Zotero PDF annotations into a queryable PaperForge data store.
+2. Show annotations visually on top of PDFs inside Obsidian.
+3. Support local annotation editing in a way that is future-compatible with safe Zotero write-back.
+
+The nearest comparison, ZotFlow, is too large and too opinionated for PaperForge's needs:
+
+- It embeds its own reader iframe.
+- It uses a Web API-centric sync model with IndexedDB and a full conflict UI.
+- It bundles UI and note-generation concerns that PaperForge does not need for the first version.
+
+At the same time, the simpler alternatives are inadequate:
+
+- Direct SQLite writes are dangerous and explicitly discouraged by Zotero.
+- Markdown-as-annotation-storage is fragile and weak for structured query.
+- A custom PaperForge-only PDF reader would duplicate functionality the Obsidian native viewer already provides.
+
+---
+
+## 4. Goals
+
+### 4.1 Primary Goals
+
+1. Import Zotero annotations from local SQLite without writing to Zotero.
+2. Normalize those annotations into a PaperForge-owned schema.
+3. Persist them in a dedicated annotation database that survives memory rebuilds.
+4. Render them as overlays in Obsidian's native PDF viewer.
+5. Allow PaperForge-local annotation creation and editing in the Obsidian PDF view.
+6. Preserve a clean future path for safe Zotero write-back.
+
+### 4.2 Secondary Goals
+
+1. Make annotations searchable by text, comment, and tags.
+2. Support export to JSON and Markdown.
+3. Support multiple PDF attachments per paper without ambiguity.
+4. Keep plugin and CLI contracts machine-readable through `--json` output.
+
+### 4.3 Non-Goals
+
+1. No direct SQLite write-back to Zotero.
+2. No full ZotFlow feature parity.
+3. No custom embedded PDF.js reader for V1.
+4. No EPUB annotation support in V1.
+5. No collaborative multi-user annotation sync in V1.
+6. No complex ink editing in V1.
+
+---
+
+## 5. Guiding Principles
+
+1. **Zotero SQLite is read-only.** PaperForge may read and cache from it, but must never mutate it.
+2. **PaperForge owns its local annotation truth.** Once imported, normalized annotations live under PaperForge's own storage and lifecycle.
+3. **Memory rebuild must never destroy user annotations.** Annotation persistence is isolated from `paperforge.db` rebuild semantics.
+4. **Use the native viewer unless there is a hard blocker.** The default UX should stay inside Obsidian's existing PDF experience.
+5. **Future write-back must use supported contracts.** When write-back arrives, it should go through the Zotero Web API with version-aware conflict handling.
+6. **Prefer minimal structure over framework sprawl.** This repo currently uses a single plugin `main.js` plus testable helpers; the design should respect that reality unless a split is clearly necessary.
+
+---
+
+## 6. Core Research Findings
+
+### 6.1 Zotero Schema Facts
+
+Zotero stores annotation data in `itemAnnotations`, keyed to `items(itemID)` and attached to `itemAttachments(itemID)` through `parentItemID`.
+
+The relevant columns are:
+
+- `type`
+- `text`
+- `comment`
+- `color`
+- `pageLabel`
+- `sortIndex`
+- `position`
+- `isExternal`
+
+Type mapping is:
+
+- `1` → `highlight`
+- `2` → `note`
+- `3` → `image`
+- `4` → `ink`
+- `5` → `underline`
+- `6` → `text`
+
+The `position` JSON is already rich enough for rendering:
+
+- `rects` for highlight-like annotations
+- `paths` for ink
+- `pageIndex` for page scoping
+- optional `width`/`height` for image/ink data
+
+### 6.2 ZotFlow Architecture Lessons
+
+Useful parts:
+
+- normalized annotation schema
+- conflict-state thinking
+- clear read/write separation in the sync engine
+
+Parts to avoid:
+
+- reader iframe architecture
+- large worker bridge and IndexedDB runtime
+- full Zotero library replacement UI
+
+### 6.3 Obsidian PDF Feasibility Facts
+
+Overlay display is feasible by patching private PDF viewer classes and listening to PDF.js event bus events such as:
+
+- `textlayerrendered`
+- `annotationlayerrendered`
+- `pagerendered`
+- `scalechanged`
+
+There is no stable public PDF view API, so this is an accepted managed risk.
+
+---
+
+## 7. Proposed Architecture
+
+## 7.1 System Overview
+
+```text
+Zotero SQLite (read-only)
+        ↓
+Probe / Import pipeline
+        ↓
+annotations.db
+        ↓
+CLI JSON contracts
+        ↓
+Obsidian plugin bridge
+        ↓
+Native PDF viewer overlay
+```
+
+## 7.2 Storage Boundary
+
+PaperForge will add:
+
+```text
+System/PaperForge/indexes/annotations.db
+```
+
+This DB is separate from:
+
+- `paperforge.db`
+- vector DB state
+- canonical index JSON snapshots
+
+That separation is intentional. `paperforge.db` is a rebuildable memory-layer artifact. `annotations.db` is a persistent user-data store.
+
+## 7.3 Runtime Layers
+
+### Python Layer
+
+Responsibilities:
+
+- locate Zotero data directory
+- copy `zotero.sqlite` to temp when needed
+- run read-only SQL queries
+- normalize annotation rows
+- store/update/search/export annotations in `annotations.db`
+- provide CLI entrypoints with JSON output
+
+### Plugin Layer
+
+Responsibilities:
+
+- detect active PDF files
+- map PDF file to PaperForge paper key
+- fetch annotations through CLI subprocess calls
+- patch native PDF viewer load/render lifecycle
+- render and update overlay DOM
+- collect selection data for local annotation creation
+- show popovers/edit affordances
+
+### Future Sync Layer
+
+Responsibilities:
+
+- inspect `sync_state`
+- enqueue pending push operations
+- push through Zotero Web API
+- detect remote version drift and conflicts
+
+---
+
+## 8. Data Model
+
+## 8.1 Database Choice
+
+Use an independent SQLite database with its own schema version and migration path.
+
+## 8.2 Main Table
+
+The normalized table should contain:
+
+- identity (`id`, `paper_id`)
+- Zotero source tracking (`zotero_key`, `zotero_item_id`, `zotero_attachment_key`, `source_version`)
+- PDF association (`pdf_path`, `pdf_hash`)
+- annotation payload (`type`, `page_index`, `page_label`, `selected_text`, `comment`, `color`, `sort_index`, `position_json`, `selector_json`, `tags_json`)
+- sync lifecycle (`source`, `sync_state`, `is_readonly`, `source_modified_at`)
+- timestamps (`created_at`, `updated_at`, `deleted_at`)
+
+## 8.3 Search
+
+Use FTS5 on:
+
+- `selected_text`
+- `comment`
+- `tags_json`
+
+## 8.4 Write-Back Readiness
+
+Even before implementation, the schema must reserve:
+
+- `sync_state = pending_push`
+- a `sync_queue` table
+- a place to track source version and future push errors
+
+This avoids a destructive redesign when write-back is implemented later.
+
+---
+
+## 9. Read Path Design
+
+## 9.1 Import Mechanics
+
+The importer will:
+
+1. detect the Zotero DB path from CLI option or configured/default data dir
+2. **by default** copy `zotero.sqlite` to a temp directory, with an explicit opt-out only for controlled environments
+3. open the copy in SQLite `mode=ro`
+4. query annotations plus parent attachment and top-level paper context
+5. query tags through `itemTags` and `tags`
+6. normalize to PaperForge schema
+7. upsert into `annotations.db`
+
+## 9.2 Identity Rules
+
+For imported Zotero annotations:
+
+- local row `id` should be stable and PaperForge-owned
+- Zotero's annotation key should live in `zotero_key`
+- `(source='zotero_db', zotero_key)` should be treated as the external identity
+
+This avoids overloading local primary keys with source keys and leaves room for future locally-created annotations that do not yet exist in Zotero.
+
+## 9.3 Deletion Semantics
+
+If a previously imported Zotero annotation disappears from the source, PaperForge should:
+
+- mark it as remotely deleted if it was never locally modified, or
+- mark it as conflict if local edits exist and future write-back semantics would be ambiguous
+
+For V1, soft delete is sufficient.
+
+---
+
+## 10. Overlay Design
+
+## 10.1 Chosen Strategy
+
+Patch the native PDF viewer instead of creating a custom reader.
+
+The plugin should:
+
+1. patch viewer load lifecycle
+2. create per-page overlay layers
+3. render rect-based annotation DOM on page render events
+4. show popovers on click/hover
+5. collect selections for local annotation creation
+
+## 10.2 Rendering Model
+
+For each rendered page:
+
+- create `div.pf-annotation-overlay`
+- call `window.pdfjsLib.setLayerDimensions()` to align it with the page viewport
+- place child rects using percentage-based CSS positioning
+
+This mirrors PDF++ and avoids re-computing absolute pixel positions on every zoom.
+
+## 10.3 Editing Model
+
+Imported Zotero annotations:
+
+- display as read-only in V1
+- show lock state visually
+
+PaperForge-local annotations:
+
+- can be created from PDF text selection
+- can be edited for comment and color
+- can be deleted locally
+
+## 10.4 Failure Behavior
+
+If patching fails due to Obsidian version drift:
+
+- the plugin must not crash
+- the annotation feature should disable itself cleanly
+- the user should still retain CLI annotation access
+
+---
+
+## 11. Write-Back Design
+
+## 11.1 Architectural Decision
+
+When write-back is implemented, it should use the **Zotero Web API**, not direct SQLite writes.
+
+## 11.2 Why Web API for Writes
+
+Advantages over SQLite mutation:
+
+1. respects Zotero's sync and validation rules
+2. supports version-aware optimistic locking
+3. avoids corruption risk when the Zotero client is open
+4. works with supported group library semantics
+5. aligns with Zotero's official extension and automation model
+
+## 11.3 Hybrid Read/Write Model
+
+Chosen long-term model:
+
+- **Read:** local SQLite
+- **Write:** Zotero Web API
+
+This keeps reads fast and offline while reserving writes for supported contracts.
+
+## 11.4 Conflict States
+
+The design should support at least:
+
+- `zotero_synced`
+- `local_modified`
+- `pending_push`
+- `zotero_remote_changed`
+- `conflict`
+
+That state model is enough to stage future sync behavior without overbuilding V1.
+
+---
+
+## 12. CLI Surface
+
+The feature should introduce a new top-level command family:
+
+```text
+paperforge annotation import
+paperforge annotation list
+paperforge annotation create
+paperforge annotation patch
+paperforge annotation delete
+paperforge annotation export
+paperforge annotation status
+```
+
+Every command should support `--json` for plugin consumption.
+
+---
+
+## 13. Testing Strategy
+
+## 13.1 Python
+
+Need tests for:
+
+- schema initialization and migration
+- probe behavior and read-only DB open
+- annotation import/upsert/delete behavior
+- CLI JSON contracts
+- search/export behavior
+
+## 13.2 Plugin
+
+Need tests for:
+
+- path/runtime helper additions in `src/testable.js`
+- subprocess argument construction
+- annotation JSON parsing and state classification
+- overlay coordinate conversion helpers extracted into testable functions where practical
+
+## 13.3 Integration
+
+Need fixture-driven tests proving:
+
+- import from representative Zotero rows
+- imported annotations survive memory rebuild
+- plugin-visible JSON output stays stable
+
+The fixture coverage should include at least:
+
+- highlight with multi-rect `position.rects`
+- note with comment payload
+- underline with selected text
+- tags attached through `itemTags`
+- non-ASCII/CJK selected text
+- imported readonly rows and PaperForge-local rows
+
+---
+
+## 14. Risks
+
+1. **Obsidian private PDF viewer API drift**
+   - mitigation: narrow patch surface, fail soft, add plugin tests around extracted helpers
+2. **Zotero schema drift**
+   - mitigation: schema checks and clear importer errors
+3. **plugin conflicts with PDF++**
+   - mitigation: detect overlap and document incompatibility or fallback mode
+4. **large annotation counts**
+   - mitigation: render only visible pages and cache per-page annotation grouping
+
+---
+
+## 15. Rollout Strategy
+
+### Phase 1
+
+- `annotations.db`
+- probe/import pipeline
+- CLI commands
+- no viewer integration yet
+
+### Phase 2
+
+- PDF overlay rendering
+- local annotation create/edit/delete
+- read-only display of imported Zotero annotations
+
+### Phase 3
+
+- Web API write-back
+- push queue
+- remote version reconciliation and conflict handling
+
+---
+
+## 16. Final Recommendation
+
+PaperForge should proceed with a lightweight, layered annotation system centered on:
+
+- **read-only local Zotero SQLite ingestion**
+- **independent local annotation persistence**
+- **native Obsidian PDF overlay rendering**
+- **future Web API write-back**
+
+This design keeps the V1 implementation small enough to ship, avoids the major risk of SQLite writes, preserves the strongest UX available in Obsidian, and leaves a clean upgrade path to safe synchronization later.