diff --git a/docs/superpowers/plans/2026-05-20-pdf-annotation-layer.md b/docs/superpowers/plans/2026-05-20-pdf-annotation-layer.md new file mode 100644 index 00000000..be9ac987 --- /dev/null +++ b/docs/superpowers/plans/2026-05-20-pdf-annotation-layer.md @@ -0,0 +1,787 @@ +# PaperForge PDF Annotation Layer Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build a read-only Zotero annotation import pipeline, persist normalized annotations in a dedicated PaperForge database, surface them through CLI JSON contracts, and render/edit them in Obsidian's native PDF viewer via overlay patching. + +**Architecture:** The work is split into five sequential packages. Package A adds a dedicated annotation database and schema lifecycle. Package B implements read-only Zotero SQLite probing and import/update behavior. Package C exposes annotation commands through the CLI with stable JSON envelopes. Package D extends the current plain `paperforge/plugin/main.js` plugin with subprocess-backed annotation loading and viewer overlay patching. Package E prepares the future write-back lane without implementing Web API push yet. + +**Tech Stack:** Python 3.10+, SQLite/WAL/FTS5, argparse CLI, Obsidian plugin CommonJS (`main.js`), Node child_process, Vitest, pytest + +--- + +## File Map + +### New Python Annotation Files + +- `paperforge/annotation/__init__.py` — annotation package marker and narrow public exports +- `paperforge/annotation/db.py` — `annotations.db` path resolution and connection helpers +- `paperforge/annotation/schema.py` — annotation schema DDL, versioning, migrations, FTS triggers +- `paperforge/annotation/probe.py` — read-only Zotero SQLite access and normalization helpers +- `paperforge/annotation/importer.py` — import/upsert/reconciliation logic from Zotero rows to PaperForge rows +- `paperforge/annotation/service.py` — CRUD/search/export operations against `annotations.db` + +### New / Modified CLI Files + +- `paperforge/commands/annotation.py` — new command family implementation +- `paperforge/cli.py` — register `annotation` subcommands and flags +- `paperforge/core/result.py` — reuse existing result envelope only if a small helper addition is needed +- `paperforge/config.py` — only if Zotero data dir resolution needs a shared helper + +### Plugin Files + +- `paperforge/plugin/main.js` — add annotation subprocess bridge, viewer patch lifecycle, overlay UI wiring +- `paperforge/plugin/src/testable.js` — extract pure helper functions for argument building, annotation payload parsing, and coordinate helpers where possible +- `paperforge/plugin/styles.css` — overlay, popover, readonly-lock, and action styles +- `paperforge/plugin/package.json` — add `monkey-around` only if needed for patching helper ergonomics + +### New / Modified Tests + +- `fixtures/zotero/test_annotations.sqlite` — minimal Zotero SQLite fixture for importer/integration coverage +- `tests/unit/annotation/test_schema.py` — new schema creation/version tests +- `tests/unit/annotation/test_probe.py` — read-only SQLite normalization tests +- `tests/unit/annotation/test_importer.py` — import/update/delete reconciliation tests +- `tests/unit/annotation/test_service.py` — create/patch/delete/list/export behavior tests +- `tests/cli/test_annotation_commands.py` — CLI envelope and argument validation tests +- `tests/integration/test_annotation_import_workflow.py` — fixture-driven import and rebuild-survival tests +- `paperforge/plugin/tests/runtime.test.mjs` — extend with annotation runtime path/config helpers if needed +- `paperforge/plugin/tests/commands.test.mjs` — subprocess argument construction for annotation commands +- `paperforge/plugin/tests/errors.test.mjs` — plugin-side error handling for annotation subprocess and patch failures + +### Existing Reference Files To Read During Execution + +- `paperforge/memory/db.py` — existing SQLite connection patterns +- `paperforge/memory/schema.py` — versioning and trigger style reference +- `paperforge/pdf_resolver.py` — PDF path resolution rules +- `paperforge/plugin/main.js` — current plugin architecture and runtime helpers +- `paperforge/plugin/src/testable.js` — current pure helper extraction pattern +- `tests/unit/memory/test_schema.py` — schema testing style +- `tests/cli/test_json_contracts.py` — JSON contract assertions pattern + +--- + +## Package A: Dedicated Annotation Database + +### Task A1: Add Annotation DB Path and Connection Helpers + +**Files:** +- Create: `paperforge/annotation/__init__.py` +- Create: `paperforge/annotation/db.py` +- Test: `tests/unit/annotation/test_schema.py` + +- [ ] **Step 1: Write the failing tests for DB path and connection behavior** + +Cover: + +- `annotations.db` resolves under `System/PaperForge/indexes/` +- read/write connection enables WAL and foreign keys +- read-only connection uses SQLite URI `mode=ro` + +- [ ] **Step 2: Run the new targeted tests to verify failure** + +Run: `python -m pytest tests/unit/annotation/test_schema.py -v --tb=short` + +Expected: FAIL because annotation DB helpers do not exist yet. + +- [ ] **Step 3: Implement minimal path and connection helpers** + +Follow the `paperforge/memory/db.py` style: + +- `get_annotations_db_path(vault: Path) -> Path` +- `get_annotations_connection(db_path: Path, read_only: bool = False) -> sqlite3.Connection` + +- [ ] **Step 4: Re-run the targeted tests** + +Run: `python -m pytest tests/unit/annotation/test_schema.py -v --tb=short` + +Expected: PASS for the new DB helper assertions. + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/annotation/__init__.py paperforge/annotation/db.py tests/unit/annotation/test_schema.py +git commit -m "feat: add annotation db helpers" +``` + +### Task A2: Add Annotation Schema and Version Lifecycle + +**Files:** +- Create: `paperforge/annotation/schema.py` +- Modify: `tests/unit/annotation/test_schema.py` + +- [ ] **Step 1: Add failing schema lifecycle tests** + +Test: + +- schema bootstrap creates `meta`, `annotations`, `annotations_fts`, `sync_queue` +- schema version is stored as `1` +- `ensure_schema()` is idempotent + +- [ ] **Step 2: Run the targeted schema tests** + +Run: `python -m pytest tests/unit/annotation/test_schema.py -v --tb=short` + +Expected: FAIL because tables and version logic do not exist. + +- [ ] **Step 3: Implement the minimal schema module** + +Include: + +- `ANNOTATIONS_SCHEMA_VERSION = 1` +- DDL matching the columns, indexes, and FTS requirements defined in `docs/superpowers/specs/2026-05-20-pdf-annotation-layer-design.md` Section 8 +- FTS triggers +- `ensure_schema(conn)` +- `get_schema_version(conn)` + +- [ ] **Step 4: Re-run the schema tests** + +Run: `python -m pytest tests/unit/annotation/test_schema.py -v --tb=short` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/annotation/schema.py tests/unit/annotation/test_schema.py +git commit -m "feat: add annotation schema" +``` + +--- + +## Package B: Read-Only Zotero SQLite Probe and Import + +### Task B1: Implement Read-Only Zotero Annotation Probe + +**Files:** +- Create: `paperforge/annotation/probe.py` +- Create: `tests/unit/annotation/test_probe.py` +- Reference: `experiments/zotero_annotation_probe.py` + +- [ ] **Step 1: Write failing tests for probe normalization** + +Cover: + +- annotation type integer → string mapping +- `position` JSON parsing +- tag aggregation +- read-only connection open path +- invalid schema error handling + +- [ ] **Step 2: Run the probe tests to verify failure** + +Run: `python -m pytest tests/unit/annotation/test_probe.py -v --tb=short` + +Expected: FAIL because the probe module does not exist. + +- [ ] **Step 3: Implement minimal probe behavior** + +Add: + +- `copy_db_to_temp()` +- `open_readonly()` +- `fetch_annotations()` +- `fetch_annotation_tags()` +- `normalize_zotero_annotation_row()` + +Use parameterized SQL only. + +Copy-to-temp must be the default behavior. A future `--no-copy` CLI flag is the explicit escape hatch. + +- [ ] **Step 4: Re-run the probe tests** + +Run: `python -m pytest tests/unit/annotation/test_probe.py -v --tb=short` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/annotation/probe.py tests/unit/annotation/test_probe.py +git commit -m "feat: add zotero annotation probe" +``` + +### Task B1b: Add Fixture-Driven Zotero Import Coverage + +**Files:** +- Create: `fixtures/zotero/test_annotations.sqlite` +- Create: `tests/integration/test_annotation_import_workflow.py` + +- [ ] **Step 1: Write the failing fixture-driven import test** + +Cover one end-to-end import from a real SQLite fixture containing at least: + +- highlight with multi-rect position +- note annotation +- underline annotation +- tags +- non-ASCII text + +- [ ] **Step 2: Run the focused integration test** + +Run: `python -m pytest tests/integration/test_annotation_import_workflow.py -v --tb=short` + +Expected: FAIL because the fixture and end-to-end import path are not wired yet. + +- [ ] **Step 3: Build the minimal SQLite fixture and test harness** + +Use the real Zotero schema subset needed by the importer so the test exercises actual SQL joins instead of mocks. + +- [ ] **Step 4: Re-run the focused integration test** + +Run: `python -m pytest tests/integration/test_annotation_import_workflow.py -v --tb=short` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add fixtures/zotero/test_annotations.sqlite tests/integration/test_annotation_import_workflow.py +git commit -m "test: add fixture-driven annotation import coverage" +``` + +### Task B2: Implement Import/Reconciliation Logic + +**Files:** +- Create: `paperforge/annotation/importer.py` +- Create: `tests/unit/annotation/test_importer.py` +- Modify: `paperforge/annotation/schema.py` only if an index/helper is missing + +- [ ] **Step 1: Write failing importer tests** + +Cover: + +- first import inserts new rows with `source='zotero_db'` and `sync_state='zotero_synced'` +- re-import with unchanged source is a no-op +- re-import with changed source updates payload and timestamps +- missing source annotation soft-deletes stale row if not locally modified +- locally modified stale row becomes `conflict` instead of blind delete + +- [ ] **Step 2: Run the importer tests** + +Run: `python -m pytest tests/unit/annotation/test_importer.py -v --tb=short` + +Expected: FAIL because the importer does not exist. + +- [ ] **Step 3: Implement minimal reconciliation logic** + +Add: + +- stable upsert by `(source, zotero_key)` +- source version + modified time tracking +- soft-delete rules +- conflict marking rules + +- [ ] **Step 4: Re-run the importer tests** + +Run: `python -m pytest tests/unit/annotation/test_importer.py -v --tb=short` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/annotation/importer.py tests/unit/annotation/test_importer.py +git commit -m "feat: add annotation import reconciliation" +``` + +### Task B3: Implement Local Annotation CRUD/Search/Export Service + +**Files:** +- Create: `paperforge/annotation/service.py` +- Create: `tests/unit/annotation/test_service.py` + +- [ ] **Step 1: Write failing service tests** + +Cover: + +- create local annotation +- patch local comment/color +- soft delete local annotation +- list by paper/page +- export JSON +- export Markdown +- reject patching imported readonly rows unless an explicit future override exists + +- [ ] **Step 2: Run the service tests** + +Run: `python -m pytest tests/unit/annotation/test_service.py -v --tb=short` + +Expected: FAIL because the service module does not exist. + +- [ ] **Step 3: Implement the minimal service layer** + +Expose: + +- `create_annotation(...)` +- `patch_annotation(...)` +- `delete_annotation(...)` +- `list_annotations(...)` +- `export_annotations_json(...)` +- `export_annotations_markdown(...)` + +- [ ] **Step 4: Re-run the service tests** + +Run: `python -m pytest tests/unit/annotation/test_service.py -v --tb=short` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/annotation/service.py tests/unit/annotation/test_service.py +git commit -m "feat: add annotation service" +``` + +--- + +## Package C: CLI Surface and JSON Contracts + +### Task C1: Add `paperforge annotation` Parser Surface + +**Files:** +- Modify: `paperforge/cli.py` +- Create: `paperforge/commands/annotation.py` +- Create: `tests/cli/test_annotation_commands.py` + +- [ ] **Step 1: Write failing CLI parser and envelope tests** + +Cover: + +- subcommands: `import`, `list`, `create`, `patch`, `delete`, `export`, `status` +- required args and mutually exclusive flags +- `--json` envelope structure + +- [ ] **Step 2: Run the CLI tests** + +Run: `python -m pytest tests/cli/test_annotation_commands.py -v --tb=short` + +Expected: FAIL because the parser and command module do not exist. + +- [ ] **Step 3: Implement minimal parser + dispatch** + +Add the new top-level `annotation` subparser in `paperforge/cli.py` and implement the command module that delegates to the new annotation service/importer. + +Treat this CLI argument and JSON envelope contract as frozen before plugin Package D2 begins. + +- [ ] **Step 4: Re-run the CLI tests** + +Run: `python -m pytest tests/cli/test_annotation_commands.py -v --tb=short` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/cli.py paperforge/commands/annotation.py tests/cli/test_annotation_commands.py +git commit -m "feat: add annotation cli" +``` + +### Task C2: Add Targeted End-to-End JSON Contract Tests + +**Files:** +- Modify: `tests/cli/test_json_contracts.py` +- Modify: `tests/test_e2e_cli.py` or create a small focused annotation E2E test if cleaner + +- [ ] **Step 1: Write failing JSON contract assertions for annotation commands** + +Cover: + +- `annotation status --json` +- `annotation list ... --json` +- `annotation create ... --json` +- error envelope when Zotero DB is missing + +- [ ] **Step 2: Run the focused JSON contract tests** + +Run: `python -m pytest tests/cli/test_json_contracts.py tests/test_e2e_cli.py -v --tb=short` + +Expected: FAIL on missing contract coverage or incorrect envelope shape. + +- [ ] **Step 3: Implement the minimal envelope fixes** + +Keep the command outputs aligned with existing `PFResult`-style expectations where practical. + +- [ ] **Step 4: Re-run the focused JSON contract tests** + +Run: `python -m pytest tests/cli/test_json_contracts.py tests/test_e2e_cli.py -v --tb=short` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add tests/cli/test_json_contracts.py tests/test_e2e_cli.py paperforge/commands/annotation.py +git commit -m "test: cover annotation json contracts" +``` + +### Task C3: Prove Annotation Persistence Survives Memory Rebuild + +**Files:** +- Modify: `tests/integration/test_annotation_import_workflow.py` + +- [ ] **Step 1: Add the failing rebuild-survival test** + +Cover: + +- import annotations from the SQLite fixture +- run `paperforge memory build` +- run annotation list again +- assert imported annotations still exist unchanged in `annotations.db` + +- [ ] **Step 2: Run the focused integration test file** + +Run: `python -m pytest tests/integration/test_annotation_import_workflow.py -v --tb=short` + +Expected: FAIL if any code path incorrectly couples `annotations.db` to memory rebuild lifecycle. + +- [ ] **Step 3: Fix lifecycle boundaries if needed** + +The expected fix should be minimal: ensure annotation DB code is fully separate from `paperforge.memory.*` rebuild paths. + +- [ ] **Step 4: Re-run the focused integration test file** + +Run: `python -m pytest tests/integration/test_annotation_import_workflow.py -v --tb=short` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add tests/integration/test_annotation_import_workflow.py +git commit -m "test: verify annotation persistence across memory rebuild" +``` + +--- + +## Package D: Obsidian Plugin Overlay and Local Editing + +### Task D1: Extend Plugin Testable Helpers for Annotation Runtime + +**Files:** +- Modify: `paperforge/plugin/src/testable.js` +- Modify: `paperforge/plugin/tests/commands.test.mjs` +- Modify: `paperforge/plugin/tests/errors.test.mjs` + +- [ ] **Step 1: Write failing plugin helper tests** + +Cover: + +- annotation command argument builders +- parsing of CLI JSON envelopes +- readonly/local annotation classification +- safe fallback when annotation subprocess fails + +- [ ] **Step 2: Run the focused plugin tests** + +Run: `cd paperforge/plugin && npx vitest run tests/commands.test.mjs tests/errors.test.mjs` + +Expected: FAIL because the helper functions do not exist. + +- [ ] **Step 3: Add pure helper functions to `src/testable.js`** + +Add only pure logic here, for example: + +- `buildAnnotationListArgs()` +- `buildAnnotationCreateArgs()` +- `parseAnnotationResult()` +- `isReadonlyAnnotation()` + +- [ ] **Step 4: Re-run the plugin tests** + +Run: `cd paperforge/plugin && npx vitest run tests/commands.test.mjs tests/errors.test.mjs` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/plugin/src/testable.js paperforge/plugin/tests/commands.test.mjs paperforge/plugin/tests/errors.test.mjs +git commit -m "feat: add annotation plugin helpers" +``` + +### Task D2: Add Annotation Subprocess Bridge to the Plugin + +**Files:** +- Modify: `paperforge/plugin/main.js` +- Modify: `paperforge/plugin/tests/runtime.test.mjs` + +- [ ] **Step 1: Write failing runtime tests for annotation bridge behavior** + +Cover: + +- plugin resolves Python executable and calls `paperforge annotation list` +- plugin handles JSON parse failures safely +- plugin degrades cleanly when CLI returns error + +- [ ] **Step 2: Run the focused runtime tests** + +Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs tests/errors.test.mjs` + +Expected: FAIL because the bridge does not exist. + +- [ ] **Step 3: Implement minimal annotation bridge functions in `main.js`** + +Add small functions rather than a big framework: + +- `runAnnotationCommand(...)` +- `fetchAnnotationsForPaper(...)` +- `createLocalAnnotation(...)` +- `patchLocalAnnotation(...)` +- `deleteLocalAnnotation(...)` + +Fetch all annotations for the active paper once on document open and cache them in memory grouped by `page_index`. Do not spawn subprocesses on every page render event. + +- [ ] **Step 4: Re-run the focused runtime tests** + +Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs tests/errors.test.mjs` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/plugin/main.js paperforge/plugin/tests/runtime.test.mjs paperforge/plugin/tests/errors.test.mjs +git commit -m "feat: add annotation plugin bridge" +``` + +### Task D3: Add Native PDF Viewer Patch Lifecycle + +**Files:** +- Modify: `paperforge/plugin/main.js` +- Modify: `paperforge/plugin/package.json` only if `monkey-around` is needed +- Modify: `paperforge/plugin/tests/errors.test.mjs` + +- [ ] **Step 1: Write failing tests for patch setup and soft-failure behavior** + +Cover: + +- plugin does not crash when PDF internals are unavailable +- patch init exits early outside desktop / unsupported state +- patch failure surfaces a controlled warning instead of breaking startup + +- [ ] **Step 2: Run the focused patch-failure tests** + +Run: `cd paperforge/plugin && npx vitest run tests/errors.test.mjs` + +Expected: FAIL because no annotation patch lifecycle exists. + +- [ ] **Step 3: Implement minimal patch registration** + +Inside `main.js`, add a narrow patch layer that: + +- detects active `pdf` leaves +- resolves internal viewer prototypes/classes for the native PDF view and uses the PDF.js event bus events called out in the spec (`textlayerrendered`, `annotationlayerrendered`, `pagerendered`, `scalechanged`) +- installs patched `load` / `loadFile` / cleanup behavior +- bails out safely if internals are missing + +Keep the patching additive and local. `paperforge/plugin/main.js` is already monolithic, so avoid introducing another large in-file subsystem unless the logic cannot be expressed as narrow helper functions. + +- [ ] **Step 4: Re-run the patch-failure tests** + +Run: `cd paperforge/plugin && npx vitest run tests/errors.test.mjs` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/plugin/main.js paperforge/plugin/package.json paperforge/plugin/tests/errors.test.mjs +git commit -m "feat: add pdf overlay patch lifecycle" +``` + +### Task D4: Render Overlays for Imported Annotations + +**Files:** +- Modify: `paperforge/plugin/main.js` +- Modify: `paperforge/plugin/styles.css` +- Modify: `paperforge/plugin/src/testable.js` +- Modify: `paperforge/plugin/tests/commands.test.mjs` + +- [ ] **Step 1: Write failing tests for coordinate conversion and render payload shaping** + +Cover: + +- page grouping by `page_index` +- rect normalization to overlay percentages +- readonly annotation style classification + +- [ ] **Step 2: Run the focused plugin tests** + +Run: `cd paperforge/plugin && npx vitest run tests/commands.test.mjs tests/runtime.test.mjs` + +Expected: FAIL because conversion/render helpers are incomplete. + +- [ ] **Step 3: Implement minimal render helpers and overlay styles** + +In `main.js`: + +- build per-page overlay containers +- render highlight/underline/note rects +- attach hover/click listeners + +In `styles.css`: + +- add overlay, rect, popover, readonly-lock styles + +Put pure math/string helpers in `src/testable.js` when possible. + +- [ ] **Step 4: Re-run the focused plugin tests** + +Run: `cd paperforge/plugin && npx vitest run tests/commands.test.mjs tests/runtime.test.mjs` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/plugin/main.js paperforge/plugin/styles.css paperforge/plugin/src/testable.js paperforge/plugin/tests/commands.test.mjs paperforge/plugin/tests/runtime.test.mjs +git commit -m "feat: render annotation overlays" +``` + +### Task D5: Add Local Annotation Create/Edit/Delete UI Flow + +**Files:** +- Modify: `paperforge/plugin/main.js` +- Modify: `paperforge/plugin/styles.css` +- Modify: `paperforge/plugin/tests/commands.test.mjs` +- Modify: `paperforge/plugin/tests/errors.test.mjs` + +- [ ] **Step 1: Write failing tests for local annotation command flow** + +Cover: + +- selection payload becomes `annotation create` CLI args +- popover edit action becomes `annotation patch` +- delete action becomes `annotation delete` +- readonly rows never expose edit actions + +- [ ] **Step 2: Run the focused plugin tests** + +Run: `cd paperforge/plugin && npx vitest run tests/commands.test.mjs tests/errors.test.mjs` + +Expected: FAIL because local annotation UI flow is not wired. + +- [ ] **Step 3: Implement the minimal local action flow** + +Add to `main.js`: + +- selection capture +- floating create affordance or context action +- popover with edit/delete buttons for local annotations only +- immediate overlay refresh after successful subprocess call + +- [ ] **Step 4: Re-run the focused plugin tests** + +Run: `cd paperforge/plugin && npx vitest run tests/commands.test.mjs tests/errors.test.mjs` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/plugin/main.js paperforge/plugin/styles.css paperforge/plugin/tests/commands.test.mjs paperforge/plugin/tests/errors.test.mjs +git commit -m "feat: add local annotation editing" +``` + +--- + +## Package E: Future Write-Back Readiness + +### Task E1: Add Explicit Pending-Push and Queue Semantics Without API Push + +**Files:** +- Modify: `paperforge/annotation/schema.py` +- Modify: `paperforge/annotation/service.py` +- Modify: `tests/unit/annotation/test_service.py` + +- [ ] **Step 1: Write failing tests for queue-ready local edits** + +Cover: + +- local edits can mark rows as `pending_push` +- `sync_queue` receives create/update/delete payloads only when write-back mode is explicitly enabled later +- V1 default behavior remains local-only without silently pushing + +- [ ] **Step 2: Run the focused service tests** + +Run: `python -m pytest tests/unit/annotation/test_service.py -v --tb=short` + +Expected: FAIL if the service cannot express future queue semantics. + +- [ ] **Step 3: Implement the minimal readiness hooks** + +Do not implement Web API logic yet. Only ensure the schema/service can represent future push state without a schema rewrite. + +- [ ] **Step 4: Re-run the focused service tests** + +Run: `python -m pytest tests/unit/annotation/test_service.py -v --tb=short` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/annotation/schema.py paperforge/annotation/service.py tests/unit/annotation/test_service.py +git commit -m "feat: prepare annotation push states" +``` + +--- + +## Full Verification Slice + +- [ ] **Step 1: Run annotation Python test slice** + +Run: + +```bash +python -m pytest tests/unit/annotation/ tests/integration/test_annotation_import_workflow.py tests/cli/test_annotation_commands.py tests/cli/test_json_contracts.py -v --tb=short +``` + +Expected: PASS. + +- [ ] **Step 2: Run plugin test slice** + +Run: + +```bash +cd paperforge/plugin && npx vitest run tests/runtime.test.mjs tests/commands.test.mjs tests/errors.test.mjs +``` + +Expected: PASS. + +- [ ] **Step 3: Run broader PaperForge regression slice** + +Run: + +```bash +python -m pytest tests/unit/memory/test_schema.py tests/test_config.py tests/test_e2e_cli.py -v --tb=short +``` + +Expected: PASS. + +- [ ] **Step 4: Run the standard project regression commands from the repo guide** + +Run: + +```bash +python -m pytest tests/unit/ tests/cli/ -v --tb=short +``` + +Run: + +```bash +cd paperforge/plugin && npx vitest run +``` + +Expected: PASS. + +- [ ] **Step 5: Record completion checkpoint** + +Expected state: + +- `annotations.db` exists and is schema-managed +- Zotero SQLite import is read-only and fixture-tested +- `paperforge annotation ...` commands are live with JSON contracts +- plugin can fetch and render overlays in the native PDF view +- local annotation create/edit/delete is supported +- write-back path is designed in schema/service state, but not implemented yet diff --git a/docs/superpowers/specs/2026-05-20-pdf-annotation-layer-design.md b/docs/superpowers/specs/2026-05-20-pdf-annotation-layer-design.md new file mode 100644 index 00000000..1bb339a3 --- /dev/null +++ b/docs/superpowers/specs/2026-05-20-pdf-annotation-layer-design.md @@ -0,0 +1,499 @@ +# PaperForge PDF Annotation Layer Design + +**Date:** 2026-05-20 +**Status:** Proposed +**Audience:** Maintainers, contributors, agentic implementers + +--- + +## 1. Summary + +PaperForge should add a lightweight PDF annotation layer that treats Zotero as an upstream annotation source instead of trying to become a full Zotero replacement. + +The chosen design is: + +1. Read Zotero annotations from local `zotero.sqlite` in **read-only** mode. +2. Cache normalized annotations in an independent `annotations.db` under `System/PaperForge/indexes/`. +3. Display those annotations in Obsidian by **monkey-patching the native PDF viewer**, following the proven PDF++ approach. +4. Allow local PaperForge-native annotation creation and editing inside Obsidian. +5. Design write-back from day one, but route future write-back through the **Zotero Web API**, not direct SQLite writes. + +This explicitly avoids building a full ZotFlow-style system with its own embedded reader, large sync engine, React-heavy UI surface, or end-to-end Zotero library replacement. + +--- + +## 2. Product Decision + +### Chosen Architecture + +- **Read path:** local Zotero SQLite, read-only +- **Cache:** independent `annotations.db` +- **Viewer integration:** patch Obsidian's native PDF viewer +- **Write-back design:** future Zotero Web API push +- **Version 1 scope:** import, cache, display, create local annotations, edit local comments/colors + +### Why + +This design gives PaperForge the strongest trade-off across speed, safety, and user experience: + +- Local SQLite gives fast, offline, zero-config reads. +- Independent DB storage prevents memory rebuild from destroying user annotations. +- Native PDF viewer patching preserves Obsidian UX instead of fragmenting it with a second reader. +- Web API write-back avoids the corruption and sync-state hazards of SQLite mutation. + +--- + +## 3. Problem Statement + +PaperForge currently has no structured annotation layer. The system can manage papers, OCR, memory state, and plugin runtime state, but it cannot: + +1. Read Zotero PDF annotations into a queryable PaperForge data store. +2. Show annotations visually on top of PDFs inside Obsidian. +3. Support local annotation editing in a way that is future-compatible with safe Zotero write-back. + +The nearest comparison, ZotFlow, is too large and too opinionated for PaperForge's needs: + +- It embeds its own reader iframe. +- It uses a Web API-centric sync model with IndexedDB and a full conflict UI. +- It bundles UI and note-generation concerns that PaperForge does not need for the first version. + +At the same time, the simpler alternatives are inadequate: + +- Direct SQLite writes are dangerous and explicitly discouraged by Zotero. +- Markdown-as-annotation-storage is fragile and weak for structured query. +- A custom PaperForge-only PDF reader would duplicate functionality the Obsidian native viewer already provides. + +--- + +## 4. Goals + +### 4.1 Primary Goals + +1. Import Zotero annotations from local SQLite without writing to Zotero. +2. Normalize those annotations into a PaperForge-owned schema. +3. Persist them in a dedicated annotation database that survives memory rebuilds. +4. Render them as overlays in Obsidian's native PDF viewer. +5. Allow PaperForge-local annotation creation and editing in the Obsidian PDF view. +6. Preserve a clean future path for safe Zotero write-back. + +### 4.2 Secondary Goals + +1. Make annotations searchable by text, comment, and tags. +2. Support export to JSON and Markdown. +3. Support multiple PDF attachments per paper without ambiguity. +4. Keep plugin and CLI contracts machine-readable through `--json` output. + +### 4.3 Non-Goals + +1. No direct SQLite write-back to Zotero. +2. No full ZotFlow feature parity. +3. No custom embedded PDF.js reader for V1. +4. No EPUB annotation support in V1. +5. No collaborative multi-user annotation sync in V1. +6. No complex ink editing in V1. + +--- + +## 5. Guiding Principles + +1. **Zotero SQLite is read-only.** PaperForge may read and cache from it, but must never mutate it. +2. **PaperForge owns its local annotation truth.** Once imported, normalized annotations live under PaperForge's own storage and lifecycle. +3. **Memory rebuild must never destroy user annotations.** Annotation persistence is isolated from `paperforge.db` rebuild semantics. +4. **Use the native viewer unless there is a hard blocker.** The default UX should stay inside Obsidian's existing PDF experience. +5. **Future write-back must use supported contracts.** When write-back arrives, it should go through the Zotero Web API with version-aware conflict handling. +6. **Prefer minimal structure over framework sprawl.** This repo currently uses a single plugin `main.js` plus testable helpers; the design should respect that reality unless a split is clearly necessary. + +--- + +## 6. Core Research Findings + +### 6.1 Zotero Schema Facts + +Zotero stores annotation data in `itemAnnotations`, keyed to `items(itemID)` and attached to `itemAttachments(itemID)` through `parentItemID`. + +The relevant columns are: + +- `type` +- `text` +- `comment` +- `color` +- `pageLabel` +- `sortIndex` +- `position` +- `isExternal` + +Type mapping is: + +- `1` → `highlight` +- `2` → `note` +- `3` → `image` +- `4` → `ink` +- `5` → `underline` +- `6` → `text` + +The `position` JSON is already rich enough for rendering: + +- `rects` for highlight-like annotations +- `paths` for ink +- `pageIndex` for page scoping +- optional `width`/`height` for image/ink data + +### 6.2 ZotFlow Architecture Lessons + +Useful parts: + +- normalized annotation schema +- conflict-state thinking +- clear read/write separation in the sync engine + +Parts to avoid: + +- reader iframe architecture +- large worker bridge and IndexedDB runtime +- full Zotero library replacement UI + +### 6.3 Obsidian PDF Feasibility Facts + +Overlay display is feasible by patching private PDF viewer classes and listening to PDF.js event bus events such as: + +- `textlayerrendered` +- `annotationlayerrendered` +- `pagerendered` +- `scalechanged` + +There is no stable public PDF view API, so this is an accepted managed risk. + +--- + +## 7. Proposed Architecture + +## 7.1 System Overview + +```text +Zotero SQLite (read-only) + ↓ +Probe / Import pipeline + ↓ +annotations.db + ↓ +CLI JSON contracts + ↓ +Obsidian plugin bridge + ↓ +Native PDF viewer overlay +``` + +## 7.2 Storage Boundary + +PaperForge will add: + +```text +System/PaperForge/indexes/annotations.db +``` + +This DB is separate from: + +- `paperforge.db` +- vector DB state +- canonical index JSON snapshots + +That separation is intentional. `paperforge.db` is a rebuildable memory-layer artifact. `annotations.db` is a persistent user-data store. + +## 7.3 Runtime Layers + +### Python Layer + +Responsibilities: + +- locate Zotero data directory +- copy `zotero.sqlite` to temp when needed +- run read-only SQL queries +- normalize annotation rows +- store/update/search/export annotations in `annotations.db` +- provide CLI entrypoints with JSON output + +### Plugin Layer + +Responsibilities: + +- detect active PDF files +- map PDF file to PaperForge paper key +- fetch annotations through CLI subprocess calls +- patch native PDF viewer load/render lifecycle +- render and update overlay DOM +- collect selection data for local annotation creation +- show popovers/edit affordances + +### Future Sync Layer + +Responsibilities: + +- inspect `sync_state` +- enqueue pending push operations +- push through Zotero Web API +- detect remote version drift and conflicts + +--- + +## 8. Data Model + +## 8.1 Database Choice + +Use an independent SQLite database with its own schema version and migration path. + +## 8.2 Main Table + +The normalized table should contain: + +- identity (`id`, `paper_id`) +- Zotero source tracking (`zotero_key`, `zotero_item_id`, `zotero_attachment_key`, `source_version`) +- PDF association (`pdf_path`, `pdf_hash`) +- annotation payload (`type`, `page_index`, `page_label`, `selected_text`, `comment`, `color`, `sort_index`, `position_json`, `selector_json`, `tags_json`) +- sync lifecycle (`source`, `sync_state`, `is_readonly`, `source_modified_at`) +- timestamps (`created_at`, `updated_at`, `deleted_at`) + +## 8.3 Search + +Use FTS5 on: + +- `selected_text` +- `comment` +- `tags_json` + +## 8.4 Write-Back Readiness + +Even before implementation, the schema must reserve: + +- `sync_state = pending_push` +- a `sync_queue` table +- a place to track source version and future push errors + +This avoids a destructive redesign when write-back is implemented later. + +--- + +## 9. Read Path Design + +## 9.1 Import Mechanics + +The importer will: + +1. detect the Zotero DB path from CLI option or configured/default data dir +2. **by default** copy `zotero.sqlite` to a temp directory, with an explicit opt-out only for controlled environments +3. open the copy in SQLite `mode=ro` +4. query annotations plus parent attachment and top-level paper context +5. query tags through `itemTags` and `tags` +6. normalize to PaperForge schema +7. upsert into `annotations.db` + +## 9.2 Identity Rules + +For imported Zotero annotations: + +- local row `id` should be stable and PaperForge-owned +- Zotero's annotation key should live in `zotero_key` +- `(source='zotero_db', zotero_key)` should be treated as the external identity + +This avoids overloading local primary keys with source keys and leaves room for future locally-created annotations that do not yet exist in Zotero. + +## 9.3 Deletion Semantics + +If a previously imported Zotero annotation disappears from the source, PaperForge should: + +- mark it as remotely deleted if it was never locally modified, or +- mark it as conflict if local edits exist and future write-back semantics would be ambiguous + +For V1, soft delete is sufficient. + +--- + +## 10. Overlay Design + +## 10.1 Chosen Strategy + +Patch the native PDF viewer instead of creating a custom reader. + +The plugin should: + +1. patch viewer load lifecycle +2. create per-page overlay layers +3. render rect-based annotation DOM on page render events +4. show popovers on click/hover +5. collect selections for local annotation creation + +## 10.2 Rendering Model + +For each rendered page: + +- create `div.pf-annotation-overlay` +- call `window.pdfjsLib.setLayerDimensions()` to align it with the page viewport +- place child rects using percentage-based CSS positioning + +This mirrors PDF++ and avoids re-computing absolute pixel positions on every zoom. + +## 10.3 Editing Model + +Imported Zotero annotations: + +- display as read-only in V1 +- show lock state visually + +PaperForge-local annotations: + +- can be created from PDF text selection +- can be edited for comment and color +- can be deleted locally + +## 10.4 Failure Behavior + +If patching fails due to Obsidian version drift: + +- the plugin must not crash +- the annotation feature should disable itself cleanly +- the user should still retain CLI annotation access + +--- + +## 11. Write-Back Design + +## 11.1 Architectural Decision + +When write-back is implemented, it should use the **Zotero Web API**, not direct SQLite writes. + +## 11.2 Why Web API for Writes + +Advantages over SQLite mutation: + +1. respects Zotero's sync and validation rules +2. supports version-aware optimistic locking +3. avoids corruption risk when the Zotero client is open +4. works with supported group library semantics +5. aligns with Zotero's official extension and automation model + +## 11.3 Hybrid Read/Write Model + +Chosen long-term model: + +- **Read:** local SQLite +- **Write:** Zotero Web API + +This keeps reads fast and offline while reserving writes for supported contracts. + +## 11.4 Conflict States + +The design should support at least: + +- `zotero_synced` +- `local_modified` +- `pending_push` +- `zotero_remote_changed` +- `conflict` + +That state model is enough to stage future sync behavior without overbuilding V1. + +--- + +## 12. CLI Surface + +The feature should introduce a new top-level command family: + +```text +paperforge annotation import +paperforge annotation list +paperforge annotation create +paperforge annotation patch +paperforge annotation delete +paperforge annotation export +paperforge annotation status +``` + +Every command should support `--json` for plugin consumption. + +--- + +## 13. Testing Strategy + +## 13.1 Python + +Need tests for: + +- schema initialization and migration +- probe behavior and read-only DB open +- annotation import/upsert/delete behavior +- CLI JSON contracts +- search/export behavior + +## 13.2 Plugin + +Need tests for: + +- path/runtime helper additions in `src/testable.js` +- subprocess argument construction +- annotation JSON parsing and state classification +- overlay coordinate conversion helpers extracted into testable functions where practical + +## 13.3 Integration + +Need fixture-driven tests proving: + +- import from representative Zotero rows +- imported annotations survive memory rebuild +- plugin-visible JSON output stays stable + +The fixture coverage should include at least: + +- highlight with multi-rect `position.rects` +- note with comment payload +- underline with selected text +- tags attached through `itemTags` +- non-ASCII/CJK selected text +- imported readonly rows and PaperForge-local rows + +--- + +## 14. Risks + +1. **Obsidian private PDF viewer API drift** + - mitigation: narrow patch surface, fail soft, add plugin tests around extracted helpers +2. **Zotero schema drift** + - mitigation: schema checks and clear importer errors +3. **plugin conflicts with PDF++** + - mitigation: detect overlap and document incompatibility or fallback mode +4. **large annotation counts** + - mitigation: render only visible pages and cache per-page annotation grouping + +--- + +## 15. Rollout Strategy + +### Phase 1 + +- `annotations.db` +- probe/import pipeline +- CLI commands +- no viewer integration yet + +### Phase 2 + +- PDF overlay rendering +- local annotation create/edit/delete +- read-only display of imported Zotero annotations + +### Phase 3 + +- Web API write-back +- push queue +- remote version reconciliation and conflict handling + +--- + +## 16. Final Recommendation + +PaperForge should proceed with a lightweight, layered annotation system centered on: + +- **read-only local Zotero SQLite ingestion** +- **independent local annotation persistence** +- **native Obsidian PDF overlay rendering** +- **future Web API write-back** + +This design keeps the V1 implementation small enough to ship, avoids the major risk of SQLite writes, preserves the strongest UX available in Obsidian, and leaves a clean upgrade path to safe synchronization later.