mirror of
https://github.com/lllin000/PaperForge.git
synced 2026-07-22 06:50:53 +00:00
feat(annotation): WIP - pdf annotation layer with Zotero sync, schema, overlay adapter
This commit is contained in:
Research Assistant
parent
e233dbd1ca
commit
f746a22b1b
40 changed files with 3637 additions and 120 deletions
|
|
@ -0,0 +1,450 @@
|
|||
# PaperForge PDF Annotation Layer PDF.js Internal Route Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace the current DOM-only PDF annotation overlay with a PDF.js-internal, viewport-aligned overlay implementation that renders imported Zotero annotations visibly and correctly inside Obsidian's native PDF viewer.
|
||||
|
||||
**Architecture:** Keep the proven Python import/cache pipeline unchanged. Rewrite the plugin-side overlay path so that it discovers PDF.js page internals, mounts one overlay per `pageView.div`, aligns each layer with `window.pdfjsLib.setLayerDimensions(...)`, and repaints through page render events instead of generic DOM mutation retries.
|
||||
|
||||
**Tech Stack:** Python 3.10+, SQLite cache, Obsidian plugin CommonJS (`main.js`), PDF.js runtime internals, optional `monkey-around`, Vitest, pytest
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
### Python Files
|
||||
|
||||
- Modify: `paperforge/annotation/cache.py` — keep cache contract stable; preserve optional page size metadata but do not expand scope further unless strictly needed for plugin fallback
|
||||
|
||||
### Plugin Runtime Files
|
||||
|
||||
- Modify: `paperforge/plugin/main.js` — replace DOM-only overlay flow with PDF.js internal route
|
||||
- Modify: `paperforge/plugin/src/testable.js` — add pure helpers for viewport math and internal guard logic
|
||||
- Modify: `paperforge/plugin/styles.css` — simplify overlay visuals after alignment is fixed; keep only necessary visible-state styles
|
||||
|
||||
### Tests
|
||||
|
||||
- Modify: `paperforge/plugin/tests/runtime.test.mjs` — internal guard logic, viewport percentage helpers, page grouping helpers
|
||||
- Modify: `paperforge/plugin/tests/commands.test.mjs` — only if annotation command bridge arguments change
|
||||
- Modify: `paperforge/plugin/tests/errors.test.mjs` — soft-disable behavior when PDF internals unavailable
|
||||
|
||||
### Reference Inputs
|
||||
|
||||
- Read: `docs/superpowers/specs/2026-05-20-pdf-annotation-layer-pdfjs-internal-design.md`
|
||||
- Read: `patch-plan/plugin-overlay-adapter.md`
|
||||
- Read: `reports/04-obsidian-pdf-overlay-feasibility.md`
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Freeze and Test the Current Data Contract
|
||||
|
||||
**Files:**
|
||||
- Modify: `paperforge/plugin/tests/runtime.test.mjs`
|
||||
- Reference: `paperforge/annotation/cache.py`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for the plugin-side annotation cache assumptions**
|
||||
|
||||
Cover:
|
||||
|
||||
- annotation objects use full field names (`type`, `position`, `page_index`, `zotero_attachment_key`)
|
||||
- viewport renderer can consume imported Zotero-style `position.rects`
|
||||
- cache grouping by `page_index` stays zero-based
|
||||
|
||||
- [ ] **Step 2: Run the focused plugin runtime tests to verify failure**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs`
|
||||
|
||||
Expected: FAIL for the new assertions.
|
||||
|
||||
- [ ] **Step 3: Implement the minimal testable helpers needed by the new renderer**
|
||||
|
||||
Add pure helpers in `src/testable.js` for:
|
||||
|
||||
- `getAnnotationPosition(ann)`
|
||||
- `getAnnotationRectsFromPosition(position)`
|
||||
- `normalizePdfRectToViewportPercent(rect, viewBox)`
|
||||
- `groupAnnotationsByPageIndex(annotations)`
|
||||
|
||||
- [ ] **Step 4: Re-run the focused plugin runtime tests**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add paperforge/plugin/src/testable.js paperforge/plugin/tests/runtime.test.mjs
|
||||
git commit -m "test: lock annotation cache and viewport helper contracts"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Add Internal PDF.js Discovery Guards
|
||||
|
||||
**Files:**
|
||||
- Modify: `paperforge/plugin/main.js`
|
||||
- Modify: `paperforge/plugin/src/testable.js`
|
||||
- Modify: `paperforge/plugin/tests/errors.test.mjs`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for internal availability checks**
|
||||
|
||||
Cover:
|
||||
|
||||
- `window.pdfjsLib.setLayerDimensions` detection
|
||||
- internal PDF view shape guard
|
||||
- graceful disablement when page internals are missing
|
||||
|
||||
- [ ] **Step 2: Run the focused error/runtime tests to verify failure**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/errors.test.mjs tests/runtime.test.mjs`
|
||||
|
||||
Expected: FAIL for the new internal guard assertions.
|
||||
|
||||
- [ ] **Step 3: Implement minimal runtime guards**
|
||||
|
||||
Add helpers for:
|
||||
|
||||
- `hasPdfJsLayerAlignment(windowObj)`
|
||||
- `resolvePdfInternalHandle(view)`
|
||||
- `canRenderPdfInternalOverlay(handle)`
|
||||
|
||||
In `main.js`, emit one concise disablement log instead of continuing with broken rendering.
|
||||
|
||||
- [ ] **Step 4: Re-run the focused tests**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/errors.test.mjs tests/runtime.test.mjs`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add paperforge/plugin/main.js paperforge/plugin/src/testable.js paperforge/plugin/tests/errors.test.mjs paperforge/plugin/tests/runtime.test.mjs
|
||||
git commit -m "feat: add PDF.js internal overlay guards"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Replace MutationObserver Rendering with PDF Viewer Event Hooks
|
||||
|
||||
**Files:**
|
||||
- Modify: `paperforge/plugin/main.js`
|
||||
- Modify: `paperforge/plugin/tests/runtime.test.mjs`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for page-render event wiring decisions**
|
||||
|
||||
Cover:
|
||||
|
||||
- first file load fetches annotations once
|
||||
- page repaint requests are page-scoped
|
||||
- generic subtree mutation is no longer the primary repaint driver
|
||||
|
||||
- [ ] **Step 2: Run the focused runtime tests to verify failure**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs`
|
||||
|
||||
Expected: FAIL for the new page event wiring assertions.
|
||||
|
||||
- [ ] **Step 3: Implement minimal event-driven lifecycle**
|
||||
|
||||
In `main.js`:
|
||||
|
||||
- patch the native PDF view load path to capture the internal viewer handle
|
||||
- subscribe to whichever of these are available from the runtime shape:
|
||||
- `pagerendered`
|
||||
- `textlayerrendered`
|
||||
- `annotationlayerrendered`
|
||||
- `scalechanged`
|
||||
- keep MutationObserver only as a narrowly scoped fallback if an event path is unavailable
|
||||
|
||||
- [ ] **Step 4: Re-run the focused runtime tests**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add paperforge/plugin/main.js paperforge/plugin/tests/runtime.test.mjs
|
||||
git commit -m "feat: drive overlay repaint from PDF viewer events"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Add PageView-Aligned Layer Management
|
||||
|
||||
**Files:**
|
||||
- Modify: `paperforge/plugin/main.js`
|
||||
- Modify: `paperforge/plugin/styles.css`
|
||||
- Modify: `paperforge/plugin/tests/runtime.test.mjs`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for page layer creation/alignment helpers**
|
||||
|
||||
Cover:
|
||||
|
||||
- one layer per page
|
||||
- layer mounted on `pageView.div`
|
||||
- `setLayerDimensions(layer, viewport)` called when available
|
||||
- page clear only removes PaperForge-owned rects
|
||||
|
||||
- [ ] **Step 2: Run the focused runtime tests to verify failure**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs`
|
||||
|
||||
Expected: FAIL for the new layer-management assertions.
|
||||
|
||||
- [ ] **Step 3: Implement minimal layer manager behavior**
|
||||
|
||||
In `main.js` add logic equivalent to:
|
||||
|
||||
- `getOrCreateOverlayLayer(pageView)`
|
||||
- `alignOverlayLayer(layer, pageView.viewport)`
|
||||
- `clearOverlayPage(pageNumber)`
|
||||
|
||||
Simplify CSS so the layer behaves like a PDF++-style aligned layer instead of a guessed absolute box.
|
||||
|
||||
- [ ] **Step 4: Re-run the focused runtime tests**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add paperforge/plugin/main.js paperforge/plugin/styles.css paperforge/plugin/tests/runtime.test.mjs
|
||||
git commit -m "feat: align overlay layers to PDF.js page viewports"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Rewrite Rect Rendering to Use Viewport ViewBox
|
||||
|
||||
**Files:**
|
||||
- Modify: `paperforge/plugin/main.js`
|
||||
- Modify: `paperforge/plugin/src/testable.js`
|
||||
- Modify: `paperforge/plugin/tests/runtime.test.mjs`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for viewport-based rect normalization**
|
||||
|
||||
Cover:
|
||||
|
||||
- input rect format is `[left, bottom, right, top]`
|
||||
- Y mirroring uses viewport/viewBox coordinates
|
||||
- output percentages are derived from `viewBox`, not guessed page sizes
|
||||
|
||||
- [ ] **Step 2: Run the focused runtime tests to verify failure**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs`
|
||||
|
||||
Expected: FAIL for the new normalization assertions.
|
||||
|
||||
- [ ] **Step 3: Implement minimal viewport-based renderer**
|
||||
|
||||
In `main.js`:
|
||||
|
||||
- stop using hardcoded letter/A4 fallback as the primary mapping path
|
||||
- normalize each rect against `pageView.viewport.viewBox`
|
||||
- preserve readonly state, popover hooks, and local action metadata
|
||||
|
||||
In `src/testable.js`, expose the pure geometry transform for tests.
|
||||
|
||||
- [ ] **Step 4: Re-run the focused runtime tests**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add paperforge/plugin/main.js paperforge/plugin/src/testable.js paperforge/plugin/tests/runtime.test.mjs
|
||||
git commit -m "feat: render annotation rects from PDF.js viewport coordinates"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Reconnect Popovers and Local Actions on the New Layer
|
||||
|
||||
**Files:**
|
||||
- Modify: `paperforge/plugin/main.js`
|
||||
- Modify: `paperforge/plugin/styles.css`
|
||||
- Modify: `paperforge/plugin/tests/errors.test.mjs`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for local interaction behavior on aligned layers**
|
||||
|
||||
Cover:
|
||||
|
||||
- click target still opens popover
|
||||
- readonly imported annotations stay readonly
|
||||
- local annotations still expose delete/edit path
|
||||
|
||||
- [ ] **Step 2: Run the focused tests to verify failure**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/errors.test.mjs tests/runtime.test.mjs`
|
||||
|
||||
Expected: FAIL for the new interaction assertions.
|
||||
|
||||
- [ ] **Step 3: Implement the minimal interaction reconnect**
|
||||
|
||||
Ensure the new viewport-aligned layer still attaches:
|
||||
|
||||
- click handlers
|
||||
- hover handlers
|
||||
- popover placement
|
||||
- local delete/edit actions
|
||||
|
||||
- [ ] **Step 4: Re-run the focused tests**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/errors.test.mjs tests/runtime.test.mjs`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add paperforge/plugin/main.js paperforge/plugin/styles.css paperforge/plugin/tests/errors.test.mjs paperforge/plugin/tests/runtime.test.mjs
|
||||
git commit -m "feat: reconnect popovers and local actions on aligned overlay layers"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Preserve or Soft-Disable Selection-to-Create
|
||||
|
||||
**Files:**
|
||||
- Modify: `paperforge/plugin/main.js`
|
||||
- Modify: `paperforge/plugin/tests/runtime.test.mjs`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for selection create behavior under the new lifecycle**
|
||||
|
||||
Cover one of two explicitly acceptable outcomes:
|
||||
|
||||
- selection-create still works after text layer readiness, or
|
||||
- selection-create is soft-disabled behind a clear console/user message pending follow-up
|
||||
|
||||
- [ ] **Step 2: Run the focused runtime tests to verify failure**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs`
|
||||
|
||||
Expected: FAIL for the new selection-path assertions.
|
||||
|
||||
- [ ] **Step 3: Implement the minimal supported outcome**
|
||||
|
||||
Preferred:
|
||||
|
||||
- bind selection-create only after page text-layer readiness on the internal page route
|
||||
|
||||
Fallback if runtime shape prevents safe support in this phase:
|
||||
|
||||
- disable selection-create temporarily with a documented message and follow-up note in the spec/plan
|
||||
|
||||
- [ ] **Step 4: Re-run the focused runtime tests**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/runtime.test.mjs`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add paperforge/plugin/main.js paperforge/plugin/tests/runtime.test.mjs
|
||||
git commit -m "feat: stabilize selection create for internal PDF overlay route"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 8: Clean Up Temporary Debug Instrumentation
|
||||
|
||||
**Files:**
|
||||
- Modify: `paperforge/plugin/main.js`
|
||||
- Modify: `paperforge/plugin/styles.css`
|
||||
- Modify: `paperforge/plugin/tests/errors.test.mjs`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for production logging expectations**
|
||||
|
||||
Cover:
|
||||
|
||||
- plugin keeps concise lifecycle logs only
|
||||
- no `visual-probe`, `hitTest`, `pageBox`, `layerBox`, `rectBox` debug spam in the stable path
|
||||
|
||||
- [ ] **Step 2: Run the focused tests to verify failure**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/errors.test.mjs tests/runtime.test.mjs`
|
||||
|
||||
Expected: FAIL because the temporary debugging code is still present.
|
||||
|
||||
- [ ] **Step 3: Remove or gate temporary diagnostics**
|
||||
|
||||
Keep only concise logs such as:
|
||||
|
||||
- overlay enabled/disabled
|
||||
- annotation count fetched
|
||||
- one clear internal-shape failure message when applicable
|
||||
|
||||
- [ ] **Step 4: Re-run the focused tests**
|
||||
|
||||
Run: `cd paperforge/plugin && npx vitest run tests/errors.test.mjs tests/runtime.test.mjs`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add paperforge/plugin/main.js paperforge/plugin/styles.css paperforge/plugin/tests/errors.test.mjs paperforge/plugin/tests/runtime.test.mjs
|
||||
git commit -m "chore: remove temporary PDF overlay debug instrumentation"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 9: Real-World Verification on a Zotero-Backed PDF
|
||||
|
||||
**Files:**
|
||||
- Modify only if verification reveals real defects
|
||||
|
||||
- [ ] **Step 1: Run Python and plugin automated test slices**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/unit/ tests/cli/ -v --tb=short
|
||||
cd paperforge/plugin && npx vitest run
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 2: Verify against the real Literature-hub vault PDF**
|
||||
|
||||
Use the existing real paper already exercised during debugging:
|
||||
|
||||
- vault PDF path under `System/Zotero/storage/5AWBHQ59/...pdf`
|
||||
- expected paper id `9ILH6L6W`
|
||||
|
||||
Confirm:
|
||||
|
||||
- imported Zotero highlights are visible on the correct lines
|
||||
- zoom preserves alignment
|
||||
- scrolling to later annotated pages preserves alignment
|
||||
- popover opens on click
|
||||
|
||||
- [ ] **Step 3: If verification fails, fix only the discovered root cause and re-run Step 1 and Step 2**
|
||||
|
||||
- [ ] **Step 4: Commit final verification-backed fixes**
|
||||
|
||||
```bash
|
||||
git add paperforge/plugin/main.js paperforge/plugin/src/testable.js paperforge/plugin/styles.css paperforge/plugin/tests/runtime.test.mjs paperforge/plugin/tests/errors.test.mjs paperforge/annotation/cache.py
|
||||
git commit -m "fix: finalize PDF.js internal annotation overlay route"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
Do not consider this plan complete until all of the following are true:
|
||||
|
||||
- the plugin no longer relies on generic page DOM geometry as its primary placement method
|
||||
- layer alignment uses PDF.js viewer internals or a clearly documented fallback path
|
||||
- a real imported Zotero annotation is visibly aligned on the target PDF inside Obsidian
|
||||
- broad debug instrumentation has been removed from the normal path
|
||||
- automated test slices pass
|
||||
|
|
@ -37,8 +37,9 @@
|
|||
|
||||
### New / Modified Tests
|
||||
|
||||
- `fixtures/zotero/test_annotations.sqlite` — minimal Zotero SQLite fixture for importer/integration coverage
|
||||
- `tests/unit/annotation/test_schema.py` — new schema creation/version tests
|
||||
- `tests/audit/test_hardcoded_paths.py` — gate that fails if any Python source hardcodes config-driven path literals; run by `pre-commit` hook
|
||||
- `fixtures/zotero/test_annotations.sqlite` — minimal Zotero SQLite fixture for importer/integration coverage
|
||||
- `tests/unit/annotation/test_schema.py` — new schema creation/version tests
|
||||
- `tests/unit/annotation/test_probe.py` — read-only SQLite normalization tests
|
||||
- `tests/unit/annotation/test_importer.py` — import/update/delete reconciliation tests
|
||||
- `tests/unit/annotation/test_service.py` — create/patch/delete/list/export behavior tests
|
||||
|
|
@ -59,7 +60,29 @@
|
|||
- `tests/cli/test_json_contracts.py` — JSON contract assertions pattern
|
||||
|
||||
---
|
||||
|
||||
|
||||
## Pre-Phase Gate: Hardcoded Path Audit
|
||||
|
||||
Every commit must pass the hardcoded-path audit before it is allowed. This prevents new code from bypassing the config system with literal directory strings.
|
||||
|
||||
Requires `pre-commit` installed (`pip install pre-commit && pre-commit install`).
|
||||
|
||||
- [ ] **Step 1: Verify audit passes on baseline**
|
||||
|
||||
Run: `python -m pytest tests/audit/test_hardcoded_paths.py -v`
|
||||
|
||||
If it fails, fix or explicitly exempt the violation before proceeding.
|
||||
|
||||
- [ ] **Step 2: Install pre-commit hook**
|
||||
|
||||
Run: `pre-commit install`
|
||||
|
||||
- [ ] **Step 3 (future): When adding new Python code to `paperforge/`, always use the config system (`paperforge_paths()`, `cfg.get()`, `load_vault_config()`) instead of `vault / "System" / "PaperForge" / ...` patterns.**
|
||||
|
||||
The audit test scans for these patterns and will reject commits that violate this rule.
|
||||
|
||||
---
|
||||
|
||||
## Package A: Dedicated Annotation Database
|
||||
|
||||
### Task A1: Add Annotation DB Path and Connection Helpers
|
||||
|
|
|
|||
320
docs/superpowers/specs/2026-05-20-annotation-ui-and-tools.md
Normal file
320
docs/superpowers/specs/2026-05-20-annotation-ui-and-tools.md
Normal file
|
|
@ -0,0 +1,320 @@
|
|||
# Annotation UI Refresh & Tool Suite
|
||||
|
||||
> PaperForge PDF annotation overlay — redesigned for Zotero parity + macOS Preview feel
|
||||
|
||||
**Date:** 2026-05-20
|
||||
**Branch:** `feat/pdf-annotation-layer`
|
||||
**Status:** Draft
|
||||
|
||||
---
|
||||
|
||||
## 1. UX Design Research Summary
|
||||
|
||||
### macOS Preview (reference model)
|
||||
|
||||
| Pattern | Detail |
|
||||
|---------|--------|
|
||||
| **Persistent Markup Toolbar** | Top-of-viewer bar with tool buttons. Click a tool (Highlight becomes gray = active mode), then select text → auto-annotates. Click tool again to exit mode. |
|
||||
| **Sticky Note** | Click Note button → cursor becomes crosshair → click anywhere on page → opens inline yellow note. |
|
||||
| **Undo** | Cmd+Z standard system undo. No toast, no timeout. |
|
||||
| **Delete** | Control-click annotation → "Remove Highlight" context menu. Or select in sidebar → Delete key. |
|
||||
| **Highlights & Notes sidebar** | View > Highlights and Notes → sidebar lists every annotation with text, author, date. Click navigates to page. |
|
||||
|
||||
### Zotero PDF Reader (feature parity target)
|
||||
|
||||
| Pattern | Detail |
|
||||
|---------|--------|
|
||||
| **Unlocked mode** | Nothing selected in toolbar. Select text → popup appears near cursor with color swatches → click color → creates highlight/underline. |
|
||||
| **Locked mode** | Click highlight/underline tool → pick color. Then every text selection auto-creates annotation with that color. |
|
||||
| **Annotation toolbar** | Highlight | Underline | Sticky Note | Select Area | Color picker. Persistent top bar. |
|
||||
| **Sticky Note** | Click tool → click page → places note + opens comment. |
|
||||
| **Annotations sidebar** | Left sidebar showing all annotations sorted by page. Click annotation → "Show on Page" navigates. |
|
||||
| **Color palette** | 8 colors: yellow, red, green, blue, purple, magenta, orange, gray. |
|
||||
|
||||
### PaperForge Design Decision
|
||||
|
||||
**Hybrid: Preview-style persistent toolbar + Zotero-style color popup.** The toolbar stays docked at top of PDF viewer (like Preview Markup). Default mode is Zotero's "unlocked" — select text, see color/type popup near cursor. Clicking a tool in toolbar enters "locked" mode — every selection auto-applies.
|
||||
|
||||
---
|
||||
|
||||
## 2. Persistent Annotation Toolbar
|
||||
|
||||
### 2.1 Layout
|
||||
|
||||
Docked at the TOP of `.pdf-viewer-container`, above the PDF content. Always visible when a PDF is open with annotations.
|
||||
|
||||
```
|
||||
┌── PDF Viewer ───────────────────────────────────────────────────────┐
|
||||
│ [◀] [_undoRedo] [🖍 Highlight] [T̲ Underline] [📝 Note] [✎ Text] │ [⬤ yellow] [⋮] │
|
||||
│ │
|
||||
│ ── separator ── │
|
||||
│ │
|
||||
│ PDF content area │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Elements (left to right):**
|
||||
1. **`[◀]`** — Back navigation button. Hidden when no navigation history exists.
|
||||
2. **`undoRedo`** — Undo (↩) + Redo (↪). Disabled states when stack is empty.
|
||||
3. **`[Highlight]`** — Tool button. Active state = blue highlight (enters locked mode).
|
||||
4. **`[Underline]`** — Tool button. Active state = blue highlight.
|
||||
5. **`[Note]`** — Tool button. Active = cursor becomes crosshair for page-click placement.
|
||||
6. **`[Text]`** — Tool button. Active = crosshair placement.
|
||||
7. **`[⬤ yellow]`** — Color indicator. Shows current color. Click opens 8-color palette popup.
|
||||
8. **`[⋮]`** — More menu: Dashboard toggle, Export annotations, Import from Zotero.
|
||||
|
||||
### 2.2 State
|
||||
|
||||
```javascript
|
||||
let _annotationToolMode = null; // null (unlocked), 'highlight', 'underline', 'note', 'text'
|
||||
let _currentAnnotationColor = '#ffd400'; // persists across sessions via plugin settings
|
||||
|
||||
// Undo system — behaves like standard editor undo
|
||||
let _undoStack = []; // [{action:'create'|'delete', ann, pageNum}], max 50
|
||||
let _redoStack = []; // [{same}]
|
||||
// On create: clear redo, push to undo. On delete: push to undo, clear redo.
|
||||
// On undo: pop undo → push redo → execute reverse action.
|
||||
// On redo: pop redo → push undo → execute original action.
|
||||
// Cull limit: if _undoStack.length > 50, shift oldest entries and commit those subprocesses.
|
||||
|
||||
// Toolbar DOM
|
||||
let _annotationToolbarEl = null;
|
||||
let _annotationToolBtns = []; // references to each tool button
|
||||
```
|
||||
|
||||
### 2.3 Behavior
|
||||
|
||||
**Unlocked mode** (`_annotationToolMode === null`, default):
|
||||
- User selects text → popup appears near cursor: `<color swatches 1-8>`
|
||||
- Click color → creates annotation with `type='highlight'` and that color
|
||||
- Click outside popup → popup closes, selection stays
|
||||
|
||||
**Locked mode** (`_annotationToolMode === 'highlight'` or `'underline'`):
|
||||
- User selects text → annotation auto-created with current mode + color
|
||||
- No popup appears
|
||||
- Same behavior as Zotero's locked mode
|
||||
|
||||
**Click-to-place mode** (`_annotationToolMode === 'note'` or `'text'`):
|
||||
- Cursor changes to crosshair over PDF
|
||||
- Click on page → creates annotation at click point (24pt × 24pt default rect)
|
||||
- Immediately opens popover with comment textarea
|
||||
- After creation, mode resets to unlocked (single-shot)
|
||||
|
||||
**Color palette popup** (click `[⬤ color]`):
|
||||
- 8 colored circles in a horizontal or 2×4 grid
|
||||
- Click a color → update `_currentAnnotationColor`, close palette
|
||||
- Palette persists across tool modes
|
||||
|
||||
---
|
||||
|
||||
## 3. Annotation Popover Redesign
|
||||
|
||||
### 3.1 Layout
|
||||
|
||||
Appears on clicking a rendered annotation rect. Uses fixed positioning anchored near the rect.
|
||||
|
||||
```
|
||||
┌───────────────────────────────┐
|
||||
│ ⬤ yellow highlight · page 3 │ ← type + color dot + page
|
||||
│ │
|
||||
│ "The selected text that │
|
||||
│ was highlighted appears │ ← scrollable if > 3 lines
|
||||
│ here with context..." │
|
||||
│ │
|
||||
│ ┌─────────────────────────┐ │
|
||||
│ │ Comment (editable) │ │ ← textarea, initially collapsed
|
||||
│ └─────────────────────────┘ │
|
||||
│ │
|
||||
│ tags: #important #result │ ← click tags to filter
|
||||
│ │
|
||||
│ [↩ Copy] [✕ Delete] [⋮] │ ← action buttons
|
||||
└───────────────────────────────┘
|
||||
```
|
||||
|
||||
### 3.2 Actions
|
||||
|
||||
| Button | Action |
|
||||
|--------|--------|
|
||||
| Copy | Copy selected text to clipboard (no subprocess) |
|
||||
| Delete | Instant: remove rect DOM + push to undo stack + fire subprocess (no confirm) |
|
||||
| ⋮ | Menu: Edit comment, Change color, Add tag |
|
||||
|
||||
### 3.3 Delete Flow (no confirm)
|
||||
|
||||
```
|
||||
User clicks "✕ Delete" in popover
|
||||
→ 1. Hide popover immediately
|
||||
→ 2. Remove rect from DOM (fade-out transition 200ms)
|
||||
→ 3. Push {action:'delete', ann} to _undoStack (clears _redoStack)
|
||||
→ 4. _removeAnnotationFromMemory(ann.id)
|
||||
→ 5. fire deleteLocalAnnotation() subprocess in background
|
||||
→ 6. On success: _scheduleAnnotationCacheFlush()
|
||||
→ 7. On failure: re-push ann to memory + re-render (undo handles this case)
|
||||
```
|
||||
|
||||
### 3.4 Undo Flow
|
||||
|
||||
```
|
||||
User clicks ↩ (undo) in toolbar
|
||||
→ Pop last item from _undoStack
|
||||
→ If action is 'delete':
|
||||
push {action:'undelete', ann} to _redoStack
|
||||
_appendAnnotationToMemory(ann)
|
||||
_removeAnnotationRectsFromDom(ann.id) // remove old rects if any
|
||||
_renderAnnotationToPage(ann, ...)
|
||||
fire createLocalAnnotation() subprocess in background
|
||||
→ If action is 'create':
|
||||
push {action:'uncreate', ann} to _redoStack
|
||||
_removeAnnotationRectsFromDom(ann.id)
|
||||
_removeAnnotationFromMemory(ann.id)
|
||||
fire deleteLocalAnnotation() subprocess
|
||||
→ After each undo: _scheduleAnnotationCacheFlush()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Annotation Sidebar (Replaces Dashboard)
|
||||
|
||||
### 4.1 Layout
|
||||
|
||||
Toggle via `[⋮]` menu → "Toggle Sidebar" or ribbon icon. Opens as a right-side panel within the PDF viewer leaf.
|
||||
|
||||
```
|
||||
┌── Annotation Sidebar ──────────── 280px ──┐
|
||||
│ Annotations (35) [✕] │
|
||||
│ │
|
||||
│ 📊 35 highlights 2 notes │
|
||||
│ │
|
||||
│ ── By Page ── │
|
||||
│ Page 1 ████████████ 12 │
|
||||
│ Page 2 ██████ 6 │
|
||||
│ Page 3 ████████████████████ 23 │
|
||||
│ ... │
|
||||
│ │
|
||||
│ ── Recent ── │
|
||||
│ + "electric field stimulation..." p.2 │
|
||||
│ - "biomaterial scaffolds..." p.5 │
|
||||
│ + Note "Important finding" p.3 │
|
||||
│ │
|
||||
│ ── All Annotations ── │
|
||||
│ ▸ Page 1 (12) │
|
||||
│ ▸ Page 2 (6) │
|
||||
│ ▸ Page 3 (23) ← expanded │
|
||||
│ ⬤ "Advances in electrical..." ¶1 │
|
||||
│ ⬤ "The piezoelectric effect..." ¶2 │
|
||||
│ ⬤ "Bone regeneration requires..." ¶3 │
|
||||
│ ... │
|
||||
│ │
|
||||
│ [Export JSON] [Import Zotero] │
|
||||
└──────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 4.2 Features
|
||||
|
||||
- **Page bars**: horizontal CSS-bar chart, click a bar → navigate to that page
|
||||
- **Expandable page sections**: click page header → shows annotation list for that page
|
||||
- **Click annotation in list** → navigate to page + highlight rect with a brief flash animation
|
||||
- **Export JSON** → `paperforge annotation export --json` subprocess
|
||||
- **Import Zotero** → `paperforge annotation import` subprocess
|
||||
- **Recently created/deleted** list from `_undoStack` entries marked with action type
|
||||
|
||||
---
|
||||
|
||||
## 5. PDF Back-Navigation
|
||||
|
||||
### 5.1 Trigger
|
||||
|
||||
PDF.js fires `pagechanging` event when:
|
||||
- User clicks internal hyperlink / cross-reference
|
||||
- User clicks outline item
|
||||
- Programmatic page change
|
||||
|
||||
### 5.2 Stack Model
|
||||
|
||||
```javascript
|
||||
let _pdfNavStack = []; // [{pageNumber, scrollY, scale}]
|
||||
let _pdfNavPos = -1; // current position in stack (not always length-1 after back)
|
||||
```
|
||||
|
||||
### 5.3 Behavior
|
||||
|
||||
```
|
||||
On PDF open: push {pageNumber: initial, scrollY: 0, scale: 1}
|
||||
On pagechanging event (source IS linkService or outline):
|
||||
→ If current page != event pageNumber:
|
||||
push current {pageNumber, scrollY, scale} to stack
|
||||
advance _pdfNavPos
|
||||
trim any entries after _pdfNavPos (like browser history)
|
||||
show [◀] button
|
||||
|
||||
Click [◀]:
|
||||
→ _pdfNavPos--
|
||||
→ navigate to stack[_pdfNavPos].pageNumber + scrollY
|
||||
→ enable [▶] forward button
|
||||
→ if _pdfNavPos === 0: hide [◀]
|
||||
|
||||
Click [▶]:
|
||||
→ _pdfNavPos++
|
||||
→ navigate
|
||||
→ if _pdfNavPos === stack.length - 1: hide [▶]
|
||||
|
||||
Stack cap: 100 entries. Oldest evicted.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Implementation Plan (Phased)
|
||||
|
||||
### Phase A: Persistent Toolbar + Locked Mode (core UI)
|
||||
**Files:** `main.js` (~150 lines new), `styles.css` (~80 lines new)
|
||||
1. Create toolbar DOM element, inject into `.pdf-viewer-container`
|
||||
2. Tool buttons: Highlight, Underline, Note, Text with active state toggle
|
||||
3. `_annotationToolMode` state + locked/unlocked behavior
|
||||
4. Color palette popup from `[⬤]` button
|
||||
5. Tests: toolbar DOM structure, mode state transitions
|
||||
|
||||
### Phase B: Undo/Redo System
|
||||
**Files:** `main.js` (~100 lines)
|
||||
1. `_undoStack` / `_redoStack` with 50-entry cap
|
||||
2. Undo/Redo buttons in toolbar
|
||||
3. Undo/Redo action dispatch (create ↔ delete)
|
||||
4. Stack culling: commit oldest entries every 10 new ops
|
||||
5. Tests: undo re-renders rect, redo restores, cap enforcement
|
||||
|
||||
### Phase C: Popover Redesign (no-confirm delete)
|
||||
**Files:** `main.js` (~60 lines), `styles.css` (~40 lines)
|
||||
1. Remove `confirm()` from delete path
|
||||
2. Add fade-out transition on rect removal
|
||||
3. Add Copy button to popover
|
||||
4. Add ⋮ menu with Edit comment / Change color
|
||||
5. Tests: delete flow, undo after delete
|
||||
|
||||
### Phase D: Annotation Sidebar
|
||||
**Files:** `main.js` (~200 lines), `styles.css` (~150 lines)
|
||||
1. Sidebar panel DOM template
|
||||
2. Toggle via ⋮ menu + ribbon icon
|
||||
3. Page bar chart from `getGroupedAnnotationsForCurrentPaper()`
|
||||
4. Expandable per-page annotation lists
|
||||
5. Click-to-navigate on annotation list items
|
||||
6. Export/Import buttons
|
||||
7. Tests: sidebar open/close, bar chart rendering, navigation
|
||||
|
||||
### Phase E: PDF Back-Navigation
|
||||
**Files:** `main.js` (~80 lines)
|
||||
1. `_pdfNavStack` + `_pdfNavPos` state
|
||||
2. Hook `pagechanging` event for link-driven navigation
|
||||
3. Back/Forward buttons in toolbar
|
||||
4. Stock cap + eviction
|
||||
5. Tests: nav push/pop, button visibility, stack overflow
|
||||
|
||||
---
|
||||
|
||||
## 7. Rules & Constraints
|
||||
|
||||
1. No new Python dependencies. All subprocess calls use existing CLI.
|
||||
2. No new npm dependencies. Use inline SVG for icons. No icon library.
|
||||
3. All CSS uses `--background-*` and `--text-*` Obsidian variables. No hardcoded colors for theme.
|
||||
4. Sidebar must not break Obsidian's native PDF sidebar (thumbnails/outline). Use CSS containment.
|
||||
5. Toolbar height ≤ 40px to avoid pushing PDF content down significantly.
|
||||
6. All annotations rendered MUST have `data-annotation-id` attribute for delete targeting.
|
||||
|
|
@ -0,0 +1,315 @@
|
|||
# PaperForge PDF Annotation Layer: PDF.js Internal Route Design
|
||||
|
||||
**Date:** 2026-05-20
|
||||
**Status:** Proposed
|
||||
**Audience:** Maintainers, contributors, agentic implementers
|
||||
|
||||
---
|
||||
|
||||
## 1. Summary
|
||||
|
||||
PaperForge should replace its current DOM-only PDF overlay experiment with a PDF.js-internal overlay architecture modeled on PDF++.
|
||||
|
||||
The critical design change is:
|
||||
|
||||
1. Stop treating Obsidian's native PDF viewer as a generic DOM tree.
|
||||
2. Patch the viewer lifecycle to obtain PDF.js page objects (`pageView`, `viewport`).
|
||||
3. Mount annotation layers on `pageView.div`.
|
||||
4. Align each layer with `window.pdfjsLib.setLayerDimensions(layer, pageView.viewport)`.
|
||||
5. Render annotation rects in the same viewport coordinate space used by PDF.js.
|
||||
|
||||
This design keeps the existing Python-side import/cache model, but replaces the plugin-side rendering path.
|
||||
|
||||
---
|
||||
|
||||
## 2. Why A New Design Is Needed
|
||||
|
||||
The first overlay implementation proved that PaperForge can:
|
||||
|
||||
- import real Zotero annotations from `zotero.sqlite`
|
||||
- normalize them correctly into `annotations.db`
|
||||
- expose them to the plugin through `annotation-cache.json`
|
||||
- compute real screen-space rect DOM nodes
|
||||
|
||||
However, real-world debugging against a live paper showed that the current rendering strategy is still wrong.
|
||||
|
||||
### 2.1 Proven Facts From Debugging
|
||||
|
||||
For the real Zotero annotation `LJ8FR3BS`:
|
||||
|
||||
- Zotero SQLite, `annotations.db`, and `annotation-cache.json` all agree on:
|
||||
- `parent_key = 5AWBHQ59`
|
||||
- `type = highlight`
|
||||
- `color = #aaaaaa`
|
||||
- `pageIndex = 0`
|
||||
- `rects = [[61.137, 323.189, 291.762, 331.788], ...]`
|
||||
- The plugin creates real overlay rect DOM nodes.
|
||||
- The overlay rect has a valid `getBoundingClientRect()`.
|
||||
- `document.elementFromPoint()` at the rect center returns the overlay node itself (`SELF`).
|
||||
|
||||
So the failure is not:
|
||||
|
||||
- SQLite parsing
|
||||
- import normalization
|
||||
- JSON cache shape
|
||||
- missing DOM nodes
|
||||
- trivial `z-index` loss
|
||||
|
||||
### 2.2 Actual Failure Mode
|
||||
|
||||
The failure is architectural:
|
||||
|
||||
- the plugin currently renders into a generic absolute-positioned layer inferred from `.page`, `canvas`, and related DOM boxes
|
||||
- PDF++ does not do that
|
||||
- PDF++ renders through PDF.js page internals and explicitly aligns the layer with `setLayerDimensions(..., viewport)`
|
||||
|
||||
The DOM-only route is therefore considered disproven for PaperForge.
|
||||
|
||||
---
|
||||
|
||||
## 3. Product Decision
|
||||
|
||||
### Chosen Plugin-Side Route
|
||||
|
||||
- **Keep:** Python import pipeline, `annotations.db`, CLI/cache bridge
|
||||
- **Replace:** DOM-only overlay placement logic
|
||||
- **Adopt:** PDF.js internal page/viewport aligned rendering
|
||||
|
||||
### Why
|
||||
|
||||
This is the smallest change that addresses the proven root cause without reopening the Python architecture.
|
||||
|
||||
---
|
||||
|
||||
## 4. Goals
|
||||
|
||||
### 4.1 Primary Goals
|
||||
|
||||
1. Render imported Zotero annotations visibly and reliably inside Obsidian's native PDF viewer.
|
||||
2. Use the same page viewport coordinate system as PDF.js.
|
||||
3. Re-render overlays from PDF viewer lifecycle events instead of generic DOM mutation heuristics.
|
||||
4. Preserve existing local annotation create/edit/delete capabilities where possible.
|
||||
|
||||
### 4.2 Secondary Goals
|
||||
|
||||
1. Remove noisy debug-only rendering paths from the plugin.
|
||||
2. Reduce overlay re-render churn during scroll/zoom.
|
||||
3. Keep the current `annotation-cache.json` read path intact.
|
||||
|
||||
### 4.3 Non-Goals
|
||||
|
||||
1. No rewrite of the Python annotation import layer.
|
||||
2. No custom embedded PDF reader.
|
||||
3. No Zotero write-back in this phase.
|
||||
4. No attempt to support every Obsidian version; private PDF internals remain a managed compatibility risk.
|
||||
|
||||
---
|
||||
|
||||
## 5. Design Principles
|
||||
|
||||
1. **Python data path stays stable.** The rendering rewrite should not disturb the proven SQLite import/cache pipeline.
|
||||
2. **Render in PDF.js coordinate space, not guessed DOM space.**
|
||||
3. **Use native viewer lifecycle events, not broad MutationObserver retries.**
|
||||
4. **Patch the smallest internal surface that exposes `pageView` and `viewport`.**
|
||||
5. **Fail soft.** If PDF internals are unavailable, the plugin should disable overlays for that session rather than degrade the whole viewer.
|
||||
|
||||
---
|
||||
|
||||
## 6. Research Basis
|
||||
|
||||
### 6.1 PDF++ Reference Pattern
|
||||
|
||||
PDF++ creates its layer like this conceptually:
|
||||
|
||||
```ts
|
||||
const pageDiv = pageView.div;
|
||||
let layer = pageDiv.querySelector('div.pdf-plus-backlink-highlight-layer');
|
||||
if (!layer) {
|
||||
layer = pageDiv.createDiv('pdf-plus-backlink-highlight-layer');
|
||||
window.pdfjsLib.setLayerDimensions(layer, pageView.viewport);
|
||||
}
|
||||
```
|
||||
|
||||
It also renders rectangles using the viewport's `viewBox`, not guessed page DOM dimensions.
|
||||
|
||||
### 6.2 Current PaperForge Gap
|
||||
|
||||
PaperForge currently:
|
||||
|
||||
- appends `.pf-annotation-overlay` directly to `.page`
|
||||
- infers layer geometry from visible DOM boxes
|
||||
- derives coordinates from cached page sizes and percentages
|
||||
- uses MutationObserver as a coarse refresh trigger
|
||||
|
||||
This is close enough to create DOM nodes, but not close enough to guarantee native viewer alignment.
|
||||
|
||||
---
|
||||
|
||||
## 7. Proposed Architecture
|
||||
|
||||
## 7.1 System Overview
|
||||
|
||||
```text
|
||||
annotation-cache.json
|
||||
↓
|
||||
Obsidian plugin file→paper resolution
|
||||
↓
|
||||
PDF view internal patch
|
||||
↓
|
||||
PDF.js pageView / viewport discovery
|
||||
↓
|
||||
viewport-aligned overlay layer
|
||||
↓
|
||||
annotation rect rendering + local actions
|
||||
```
|
||||
|
||||
## 7.2 Internal Patch Boundary
|
||||
|
||||
The plugin should patch the native PDF viewer only far enough to gain access to:
|
||||
|
||||
- current PDF file path
|
||||
- `pdfViewer`
|
||||
- `pdfViewer.getPageView(pageNumberIndex)`
|
||||
- PDF.js event bus events such as:
|
||||
- `pagerendered`
|
||||
- `textlayerrendered`
|
||||
- `annotationlayerrendered`
|
||||
- `scalechanged`
|
||||
|
||||
## 7.3 Layer Alignment
|
||||
|
||||
For each annotated page:
|
||||
|
||||
1. Get `pageView`.
|
||||
2. Get or create `.pf-annotation-overlay` on `pageView.div`.
|
||||
3. Call `window.pdfjsLib.setLayerDimensions(layer, pageView.viewport)`.
|
||||
4. Render rects as percentages of `pageView.viewport.viewBox`.
|
||||
|
||||
## 7.4 Coordinate Mapping
|
||||
|
||||
Input rects remain Zotero/PDF coordinates:
|
||||
|
||||
```text
|
||||
[left, bottom, right, top]
|
||||
```
|
||||
|
||||
Rendering should:
|
||||
|
||||
1. mirror Y into viewport display space
|
||||
2. normalize against `viewport.viewBox`
|
||||
3. set CSS percentages relative to the aligned overlay layer
|
||||
|
||||
This removes the need to guess page dimensions from:
|
||||
|
||||
- canvas pixels
|
||||
- DOM boxes
|
||||
- hardcoded letter/A4 fallbacks
|
||||
|
||||
## 7.5 Event-Driven Refresh
|
||||
|
||||
Replace broad mutation-driven rerenders with page/viewer events:
|
||||
|
||||
- initial file load → fetch annotations once
|
||||
- `pagerendered` / `textlayerrendered` → ensure page layer exists, repaint that page only
|
||||
- `scalechanged` → clear/rebuild visible layers
|
||||
- annotation create/delete/update → invalidate local page cache and repaint affected page only
|
||||
|
||||
---
|
||||
|
||||
## 8. Data Contracts
|
||||
|
||||
## 8.1 Cache Contract
|
||||
|
||||
The existing `annotation-cache.json` contract remains valid.
|
||||
|
||||
Required fields per annotation:
|
||||
|
||||
- `id`
|
||||
- `paper_id`
|
||||
- `zotero_key`
|
||||
- `zotero_attachment_key`
|
||||
- `type`
|
||||
- `page_index`
|
||||
- `selected_text`
|
||||
- `comment`
|
||||
- `color`
|
||||
- `sort_index`
|
||||
- `position.rects`
|
||||
- `is_readonly`
|
||||
|
||||
`page_width/page_height` may remain in the cache as optional debug metadata, but the plugin should no longer depend on them for final placement when `viewport` is available.
|
||||
|
||||
## 8.2 Plugin Runtime State
|
||||
|
||||
The plugin should maintain:
|
||||
|
||||
- current `pdfPath`
|
||||
- current `paperId`
|
||||
- current annotations grouped by `page_index`
|
||||
- per-page overlay layer references
|
||||
- per-page rendered annotation ids for cheap repaint decisions
|
||||
|
||||
---
|
||||
|
||||
## 9. Module/File Strategy
|
||||
|
||||
Respect current repo reality:
|
||||
|
||||
- `paperforge/plugin/main.js` remains the runtime entry point
|
||||
- `paperforge/plugin/src/testable.js` remains the home for pure helpers
|
||||
|
||||
Recommended logical slices inside `main.js`:
|
||||
|
||||
1. PDF internal discovery helpers
|
||||
2. overlay layer manager
|
||||
3. viewport-based rect renderer
|
||||
4. event registration / teardown
|
||||
5. selection-to-create and popover actions
|
||||
|
||||
No forced multi-file runtime split is required for this phase.
|
||||
|
||||
---
|
||||
|
||||
## 10. Risks
|
||||
|
||||
### Risk 1: Obsidian PDF internals differ across versions
|
||||
|
||||
Mitigation:
|
||||
|
||||
- add version/shape guards around internal access
|
||||
- log a single concise disablement message
|
||||
- avoid crashing plugin startup
|
||||
|
||||
### Risk 2: Cannot reliably obtain `pageView` from current patch point
|
||||
|
||||
Mitigation:
|
||||
|
||||
- probe multiple internal access paths already documented by PDF++ research
|
||||
- patch at file-load and render-event boundaries
|
||||
- fall back to disabling overlays rather than continuing with DOM-only placement
|
||||
|
||||
### Risk 3: Selection-create path depends on text layer timing
|
||||
|
||||
Mitigation:
|
||||
|
||||
- register selection handlers only after page text layer render readiness
|
||||
- keep current create command bridge unchanged until overlay rendering is stable
|
||||
|
||||
---
|
||||
|
||||
## 11. Acceptance Criteria
|
||||
|
||||
The phase is complete when all of the following are true on a real Zotero-backed PDF in Obsidian:
|
||||
|
||||
1. Imported annotations appear visibly on the correct page and over the correct text region.
|
||||
2. Overlays remain aligned after zoom changes.
|
||||
3. Overlays remain aligned after scrolling to newly rendered pages.
|
||||
4. Console no longer depends on broad debug spam to understand rendering state.
|
||||
5. The plugin no longer depends on guessed letter/A4 sizing for primary placement.
|
||||
6. Existing local create/delete behavior still works, or is explicitly soft-disabled with a documented reason.
|
||||
|
||||
---
|
||||
|
||||
## 12. Rollout Note
|
||||
|
||||
This design supersedes only the plugin-side rendering path of the earlier 2026-05-20 annotation-layer design. The Python-side architecture, database boundary, and cache contract remain in force.
|
||||
22
experiments/obsidian-overlay-prototype/README.md
Normal file
22
experiments/obsidian-overlay-prototype/README.md
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
# Obsidian PDF Overlay Prototype
|
||||
|
||||
> Placeholder for overlay prototype code.
|
||||
>
|
||||
> After Phase 2 implementation, this directory will contain:
|
||||
> - `patch-pdf-viewer.ts` — monkey-around patches for PDFViewerChild
|
||||
> - `overlay-layer.ts` — overlay DOM management
|
||||
> - `rect-renderer.ts` — rectangle rendering with coordinates
|
||||
> - `selection-handler.ts` — text selection detection
|
||||
> - `popover.ts` — annotation edit popover
|
||||
> - `annotation-fetcher.ts` — execFile Python bridge
|
||||
> - `types.ts` — shared interfaces
|
||||
|
||||
## Reference
|
||||
|
||||
PDF++ source: https://github.com/RyotaUshio/obsidian-pdf-plus
|
||||
|
||||
Core techniques:
|
||||
- `around()` from `monkey-around` library for prototype patching
|
||||
- `window.pdfjsLib.setLayerDimensions()` for coordinate alignment
|
||||
- Percentage-based CSS positioning for zoom-independence
|
||||
- EventBus: `textlayerrendered`, `pagerendered`, `scalechanged`
|
||||
126
experiments/sample_zotero_annotations.json
Normal file
126
experiments/sample_zotero_annotations.json
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
{
|
||||
"ok": true,
|
||||
"probe": {
|
||||
"zotero_db": "/Users/researcher/Zotero/zotero.sqlite",
|
||||
"tables_found": [
|
||||
"itemAnnotations",
|
||||
"itemAttachments",
|
||||
"itemCreators",
|
||||
"itemData",
|
||||
"itemDataValues",
|
||||
"itemNotes",
|
||||
"itemRelations",
|
||||
"itemTags",
|
||||
"itemTypeCreatorTypes",
|
||||
"itemTypeFields",
|
||||
"itemTypeFieldsCombined",
|
||||
"itemTypes",
|
||||
"items",
|
||||
"tags",
|
||||
"creators",
|
||||
"collections",
|
||||
"collectionItems",
|
||||
"deletedItems",
|
||||
"fulltextWords",
|
||||
"groupItems",
|
||||
"groups",
|
||||
"libraries",
|
||||
"relations",
|
||||
"settings",
|
||||
"syncCache",
|
||||
"syncDeleteLog",
|
||||
"syncQueue",
|
||||
"version"
|
||||
],
|
||||
"annotation_count": 3,
|
||||
"schema_version_detected": "itemAnnotations v125+"
|
||||
},
|
||||
"annotations": [
|
||||
{
|
||||
"libraryID": 1,
|
||||
"annotationKey": "ABC12345",
|
||||
"annotationTypeInt": 1,
|
||||
"annotationType": "highlight",
|
||||
"attachmentKey": "PDFE7890",
|
||||
"parentItemKey": "PAPER1234",
|
||||
"parentItemTitle": "Deep Learning for Biomedical Image Analysis",
|
||||
"isExternal": false,
|
||||
"selectedText": "Deep learning approaches have demonstrated remarkable performance across various biomedical imaging tasks, including segmentation, classification, and detection.",
|
||||
"comment": "Key finding for introduction section",
|
||||
"color": "#ffd400",
|
||||
"pageLabel": "3",
|
||||
"sortIndex": "00002|000000|00000",
|
||||
"position": {
|
||||
"pageIndex": 2,
|
||||
"rects": [
|
||||
[72.0, 520.0, 540.0, 536.0],
|
||||
[72.0, 504.0, 540.0, 520.0],
|
||||
[72.0, 488.0, 540.0, 504.0]
|
||||
]
|
||||
},
|
||||
"tags": ["deep_learning", "review"],
|
||||
"authorName": "",
|
||||
"dateAdded": "2025-11-15 10:30:00",
|
||||
"dateModified": "2025-11-15 10:35:00",
|
||||
"version": 5,
|
||||
"attachment_path": "storage:PDFE7890/paper.pdf",
|
||||
"attachment_link_mode": 0
|
||||
},
|
||||
{
|
||||
"libraryID": 1,
|
||||
"annotationKey": "DEF67890",
|
||||
"annotationTypeInt": 2,
|
||||
"annotationType": "note",
|
||||
"attachmentKey": "PDFE7890",
|
||||
"parentItemKey": "PAPER1234",
|
||||
"parentItemTitle": "Deep Learning for Biomedical Image Analysis",
|
||||
"isExternal": false,
|
||||
"selectedText": "",
|
||||
"comment": "<p>This figure shows the U-Net architecture that became the de facto standard for medical image segmentation. Note the skip connections that preserve spatial information across the contracting path.</p>",
|
||||
"color": "#2ea8e5",
|
||||
"pageLabel": "7",
|
||||
"sortIndex": "00006|000000|00000",
|
||||
"position": {
|
||||
"pageIndex": 6,
|
||||
"rects": [
|
||||
[420.0, 360.0, 540.0, 400.0]
|
||||
]
|
||||
},
|
||||
"tags": ["architecture", "u-net"],
|
||||
"authorName": "",
|
||||
"dateAdded": "2025-11-15 14:00:00",
|
||||
"dateModified": "2025-11-16 09:15:00",
|
||||
"version": 7,
|
||||
"attachment_path": "storage:PDFE7890/paper.pdf",
|
||||
"attachment_link_mode": 0
|
||||
},
|
||||
{
|
||||
"libraryID": 1,
|
||||
"annotationKey": "GHI12345",
|
||||
"annotationTypeInt": 5,
|
||||
"annotationType": "underline",
|
||||
"attachmentKey": "PDFE7890",
|
||||
"parentItemKey": "PAPER1234",
|
||||
"parentItemTitle": "Deep Learning for Biomedical Image Analysis",
|
||||
"isExternal": false,
|
||||
"selectedText": "The primary limitation of current approaches is the requirement for large annotated datasets.",
|
||||
"comment": "Research gap — mention in related work",
|
||||
"color": "#ff6666",
|
||||
"pageLabel": "12",
|
||||
"sortIndex": "00011|000000|00000",
|
||||
"position": {
|
||||
"pageIndex": 11,
|
||||
"rects": [
|
||||
[72.0, 480.0, 540.0, 496.0]
|
||||
]
|
||||
},
|
||||
"tags": ["research_gap"],
|
||||
"authorName": "",
|
||||
"dateAdded": "2025-11-16 11:00:00",
|
||||
"dateModified": "2025-11-16 11:00:00",
|
||||
"version": 8,
|
||||
"attachment_path": "storage:PDFE7890/paper.pdf",
|
||||
"attachment_link_mode": 0
|
||||
}
|
||||
]
|
||||
}
|
||||
242
experiments/zotero_annotation_probe.py
Normal file
242
experiments/zotero_annotation_probe.py
Normal file
|
|
@ -0,0 +1,242 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Zotero Annotation Probe — read-only SQLite parser for Zotero annotations.
|
||||
|
||||
Usage:
|
||||
python experiments/zotero_annotation_probe.py --zotero-db "<path-to-zotero.sqlite>" [--limit 20]
|
||||
|
||||
Outputs unified JSON annotations to stdout. Never writes to Zotero SQLite.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ANNOTATION_TYPE_MAP = {
|
||||
1: "highlight",
|
||||
2: "note",
|
||||
3: "image",
|
||||
4: "ink",
|
||||
5: "underline",
|
||||
6: "text",
|
||||
}
|
||||
|
||||
|
||||
def copy_db_to_temp(zotero_db: Path) -> Path:
|
||||
"""Copy zotero.sqlite to a temp file to avoid locks and inconsistent reads."""
|
||||
tmp = Path(tempfile.mkdtemp()) / "zotero_copy.sqlite"
|
||||
shutil.copy2(str(zotero_db), str(tmp))
|
||||
return tmp
|
||||
|
||||
|
||||
def open_readonly(db_path: Path) -> sqlite3.Connection:
|
||||
uri = "file:" + db_path.as_posix() + "?mode=ro"
|
||||
conn = sqlite3.connect(uri, uri=True)
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
|
||||
|
||||
def probe_schema(conn: sqlite3.Connection) -> dict:
|
||||
"""Return detected table schemas for verification."""
|
||||
tables = {}
|
||||
cursor = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"
|
||||
)
|
||||
for row in cursor.fetchall():
|
||||
tables[row["name"]] = True
|
||||
return tables
|
||||
|
||||
|
||||
def fetch_annotations(conn: sqlite3.Connection, limit: int = 20) -> list[dict]:
|
||||
"""Fetch annotations and their parent attachment info."""
|
||||
query = """
|
||||
SELECT
|
||||
ia.itemID,
|
||||
ia.parentItemID,
|
||||
ia.type AS annotation_type,
|
||||
ia.authorName,
|
||||
ia.text AS annotation_text,
|
||||
ia.comment AS annotation_comment,
|
||||
ia.color AS annotation_color,
|
||||
ia.pageLabel,
|
||||
ia.sortIndex,
|
||||
ia.position AS position_json,
|
||||
ia.isExternal,
|
||||
|
||||
i.key AS annotation_key,
|
||||
i.libraryID,
|
||||
i.dateAdded,
|
||||
i.dateModified,
|
||||
i.version,
|
||||
|
||||
att.key AS attachment_key,
|
||||
ia_att.path AS attachment_path,
|
||||
ia_att.linkMode AS attachment_link_mode,
|
||||
ia_att.contentType AS attachment_content_type,
|
||||
|
||||
parent.key AS parent_item_key
|
||||
FROM itemAnnotations ia
|
||||
JOIN items i ON i.itemID = ia.itemID
|
||||
LEFT JOIN itemAttachments ia_att ON ia_att.itemID = ia.parentItemID
|
||||
LEFT JOIN items att ON att.itemID = ia.parentItemID
|
||||
LEFT JOIN items parent ON parent.itemID = ia_att.parentItemID
|
||||
LIMIT ?
|
||||
"""
|
||||
rows = conn.execute(query, (limit,)).fetchall()
|
||||
|
||||
results = []
|
||||
for row in rows:
|
||||
ann_type_int = row["annotation_type"]
|
||||
ann_type_str = ANNOTATION_TYPE_MAP.get(ann_type_int, f"unknown_{ann_type_int}")
|
||||
|
||||
position = {}
|
||||
raw_pos = row["position_json"]
|
||||
if raw_pos:
|
||||
try:
|
||||
position = json.loads(raw_pos)
|
||||
except json.JSONDecodeError:
|
||||
position = {"_parse_error": True, "_raw_preview": raw_pos[:200]}
|
||||
|
||||
# Fetch tags for this annotation
|
||||
tags = []
|
||||
try:
|
||||
tag_rows = conn.execute(
|
||||
"""SELECT t.name
|
||||
FROM itemTags it
|
||||
JOIN tags t ON t.tagID = it.tagID
|
||||
WHERE it.itemID = ?""",
|
||||
(row["itemID"],),
|
||||
).fetchall()
|
||||
tags = [t["name"] for t in tag_rows]
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
entry = {
|
||||
"libraryID": row["libraryID"],
|
||||
"annotationKey": row["annotation_key"],
|
||||
"annotationTypeInt": ann_type_int,
|
||||
"annotationType": ann_type_str,
|
||||
"attachmentKey": row["attachment_key"],
|
||||
"parentItemKey": row["parent_item_key"],
|
||||
"isExternal": bool(row["isExternal"]),
|
||||
"selectedText": row["annotation_text"] or "",
|
||||
"comment": row["annotation_comment"] or "",
|
||||
"color": row["annotation_color"] or "#ffd400",
|
||||
"pageLabel": row["pageLabel"] or "",
|
||||
"sortIndex": row["sortIndex"] or "",
|
||||
"position": position,
|
||||
"tags": tags,
|
||||
"authorName": row["authorName"] or "",
|
||||
"dateAdded": row["dateAdded"] or "",
|
||||
"dateModified": row["dateModified"] or "",
|
||||
"version": row["version"] or 0,
|
||||
"attachment_path": row["attachment_path"] or "",
|
||||
"attachment_link_mode": row["attachment_link_mode"],
|
||||
}
|
||||
|
||||
results.append(entry)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Read-only Zotero annotation probe"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--zotero-db",
|
||||
required=True,
|
||||
help="Path to zotero.sqlite",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--limit",
|
||||
type=int,
|
||||
default=20,
|
||||
help="Max annotations to return",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--copy-db",
|
||||
action="store_true",
|
||||
default=True,
|
||||
help="Copy DB to temp before reading (default: True)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-copy",
|
||||
action="store_true",
|
||||
help="Skip DB copy (read directly, risky if Zotero is running)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
zotero_db = Path(args.zotero_db).expanduser().resolve()
|
||||
if not zotero_db.exists():
|
||||
print(json.dumps({
|
||||
"ok": False,
|
||||
"error": f"File not found: {zotero_db}",
|
||||
}, ensure_ascii=False))
|
||||
sys.exit(1)
|
||||
|
||||
probe_path = zotero_db
|
||||
cleanup_path = None
|
||||
|
||||
if not args.no_copy:
|
||||
try:
|
||||
probe_path = copy_db_to_temp(zotero_db)
|
||||
cleanup_path = probe_path.parent
|
||||
except Exception as e:
|
||||
print(json.dumps({
|
||||
"ok": False,
|
||||
"error": f"Failed to copy DB: {e}",
|
||||
}, ensure_ascii=False))
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
conn = open_readonly(probe_path)
|
||||
|
||||
tables = probe_schema(conn)
|
||||
if "itemAnnotations" not in tables:
|
||||
print(json.dumps({
|
||||
"ok": False,
|
||||
"error": "itemAnnotations table not found — not a valid Zotero SQLite?",
|
||||
"detected_tables": list(tables.keys()),
|
||||
}, ensure_ascii=False))
|
||||
sys.exit(1)
|
||||
|
||||
annotations = fetch_annotations(conn, limit=args.limit)
|
||||
|
||||
output = {
|
||||
"ok": True,
|
||||
"probe": {
|
||||
"zotero_db": str(zotero_db),
|
||||
"tables_found": list(tables.keys()),
|
||||
"annotation_count": len(annotations),
|
||||
"schema_version_detected": "itemAnnotations v125+",
|
||||
},
|
||||
"annotations": annotations,
|
||||
}
|
||||
|
||||
print(json.dumps(output, ensure_ascii=False, indent=2))
|
||||
|
||||
except sqlite3.DatabaseError as e:
|
||||
print(json.dumps({
|
||||
"ok": False,
|
||||
"error": f"SQLite error: {e}",
|
||||
}, ensure_ascii=False))
|
||||
sys.exit(1)
|
||||
finally:
|
||||
conn.close()
|
||||
if cleanup_path and cleanup_path.exists():
|
||||
shutil.rmtree(cleanup_path, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -13,6 +13,8 @@ from pathlib import Path
|
|||
|
||||
|
||||
def build_test_db(path: Path) -> None:
|
||||
if path.exists():
|
||||
path.unlink()
|
||||
conn = sqlite3.connect(str(path))
|
||||
conn.execute("PRAGMA journal_mode=OFF")
|
||||
|
||||
|
|
|
|||
|
|
@ -7,25 +7,22 @@ import json
|
|||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from paperforge.core.errors import ErrorCode
|
||||
|
||||
from paperforge import __version__ as PF_VERSION
|
||||
from paperforge.annotation.cache import write_cache
|
||||
from paperforge.annotation.db import get_annotations_db_path, get_annotations_connection
|
||||
from paperforge.annotation.db import get_annotations_connection, get_annotations_db_path
|
||||
from paperforge.annotation.importer import run_import
|
||||
from paperforge.annotation.probe import copy_db_to_temp, open_readonly, fetch_annotations
|
||||
from paperforge.annotation.probe import copy_db_to_temp, fetch_annotations, open_readonly
|
||||
from paperforge.annotation.schema import ensure_schema, get_schema_version
|
||||
from paperforge.annotation.service import (
|
||||
create_annotation,
|
||||
delete_annotation,
|
||||
export_annotations_json,
|
||||
export_annotations_markdown,
|
||||
get_annotation,
|
||||
hard_delete,
|
||||
list_annotations,
|
||||
patch_annotation,
|
||||
)
|
||||
from paperforge.config import paperforge_paths
|
||||
from paperforge.core.errors import ErrorCode
|
||||
from paperforge.core.result import PFError, PFResult
|
||||
|
||||
|
||||
|
|
@ -165,7 +162,10 @@ def _resolve_paper_key(vault, pdf_path):
|
|||
if not pdf_path:
|
||||
return ""
|
||||
import sqlite3
|
||||
db = Path(str(vault)) / "System" / "PaperForge" / "indexes" / "paperforge.db"
|
||||
|
||||
from paperforge.memory.db import get_memory_db_path
|
||||
|
||||
db = get_memory_db_path(Path(str(vault)))
|
||||
if not db.exists():
|
||||
return ""
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -53,15 +53,13 @@ def run(args) -> int:
|
|||
|
||||
def _dashboard_from_db(vault: Path) -> dict | None:
|
||||
"""Build dashboard stats from paperforge.db. Returns None if DB unavailable."""
|
||||
paths = paperforge_paths(vault)
|
||||
index_path = paths.get("index")
|
||||
if not index_path:
|
||||
return None
|
||||
db_path = index_path.parent / "paperforge.db"
|
||||
paths = paperforge_paths(vault, load_vault_config(vault))
|
||||
db_path = paths["index"].parent / "paperforge.db"
|
||||
if not db_path.exists():
|
||||
return None
|
||||
try:
|
||||
import sqlite3
|
||||
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
conn.row_factory = sqlite3.Row
|
||||
# Aggregate stats via single GROUP BY
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ def run(args: argparse.Namespace) -> int:
|
|||
vault = getattr(args, "vault_path", None)
|
||||
if vault is None:
|
||||
from paperforge.config import resolve_vault
|
||||
|
||||
vault = resolve_vault(cli_vault=getattr(args, "vault", None))
|
||||
|
||||
verbose = getattr(args, "verbose", False)
|
||||
|
|
@ -54,8 +55,14 @@ def run(args: argparse.Namespace) -> int:
|
|||
from paperforge.services.sync_service import SyncService
|
||||
|
||||
svc = SyncService(vault)
|
||||
result = svc.run(verbose=verbose, json_output=json_output, selection_only=selection_only, index_only=index_only,
|
||||
prune=prune_flag, prune_force=prune_force)
|
||||
result = svc.run(
|
||||
verbose=verbose,
|
||||
json_output=json_output,
|
||||
selection_only=selection_only,
|
||||
index_only=index_only,
|
||||
prune=prune_flag,
|
||||
prune_force=prune_force,
|
||||
)
|
||||
|
||||
_write_orphan_state(vault, result)
|
||||
|
||||
|
|
@ -64,15 +71,19 @@ def run(args: argparse.Namespace) -> int:
|
|||
if result.ok and not dry_run and not index_only and not selection_only:
|
||||
try:
|
||||
from paperforge.memory.builder import build_from_index
|
||||
|
||||
build_from_index(vault)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
import subprocess, sys
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
subprocess.Popen(
|
||||
[sys.executable, "-m", "paperforge", "embed", "build", "--resume"],
|
||||
cwd=str(vault),
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
creationflags=subprocess.CREATE_NEW_PROCESS_GROUP if sys.platform == "win32" else 0,
|
||||
)
|
||||
except Exception:
|
||||
|
|
@ -84,6 +95,7 @@ def run(args: argparse.Namespace) -> int:
|
|||
|
||||
try:
|
||||
from paperforge.memory.builder import build_from_index
|
||||
|
||||
counts = build_from_index(vault)
|
||||
tag = " (fast)" if counts.get("hash_match") else ""
|
||||
print(f"memory: {counts.get('papers_indexed', 0)} papers{tag}")
|
||||
|
|
@ -91,11 +103,14 @@ def run(args: argparse.Namespace) -> int:
|
|||
print(f"memory: deferred ({e})")
|
||||
|
||||
try:
|
||||
import subprocess, sys
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
subprocess.Popen(
|
||||
[sys.executable, "-m", "paperforge", "embed", "build", "--resume"],
|
||||
cwd=str(vault),
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
creationflags=subprocess.CREATE_NEW_PROCESS_GROUP if sys.platform == "win32" else 0,
|
||||
)
|
||||
except Exception:
|
||||
|
|
@ -108,8 +123,7 @@ def _write_orphan_state(vault, result: PFResult) -> None:
|
|||
from paperforge.config import paperforge_paths
|
||||
preview = (result.data or {}).get("prune", {}) if result.data else {}
|
||||
items = preview.get("preview", []) if isinstance(preview, dict) else []
|
||||
paths = paperforge_paths(vault)
|
||||
orphan_path = paths["index"].parent / "sync-orphan-state.json"
|
||||
orphan_path = paperforge_paths(vault)["index"].parent / "sync-orphan-state.json"
|
||||
if not items:
|
||||
try:
|
||||
orphan_path.unlink(missing_ok=True)
|
||||
|
|
|
|||
|
|
@ -8,12 +8,14 @@ logger = logging.getLogger(__name__)
|
|||
|
||||
def get_vector_db_path(vault: Path) -> Path:
|
||||
from paperforge.config import paperforge_paths
|
||||
|
||||
paths = paperforge_paths(vault)
|
||||
return (paths.get("memory_db", paths.get("index", vault / "System" / "PaperForge"))).parent / "vectors"
|
||||
return paths["index"].parent / "vectors"
|
||||
|
||||
|
||||
def _get_chroma():
|
||||
import chromadb
|
||||
|
||||
return chromadb
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -6,8 +6,9 @@ from pathlib import Path
|
|||
|
||||
def get_vector_build_state_path(vault: Path) -> Path:
|
||||
from paperforge.config import paperforge_paths
|
||||
|
||||
paths = paperforge_paths(vault)
|
||||
index_dir = (paths.get("memory_db", paths.get("index", vault / "System" / "PaperForge"))).parent
|
||||
index_dir = paths["index"].parent
|
||||
return index_dir / "vector-build-state.json"
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -48,11 +48,13 @@ def _preflight_check(vault: Path, settings: dict | None = None) -> dict:
|
|||
}
|
||||
|
||||
# 4. OCR done papers
|
||||
from paperforge.worker._utils import pipeline_paths
|
||||
paths = pipeline_paths(vault)
|
||||
idx_path = paths.get("indexes", Path("")) / "formal-library.json" if paths.get("indexes") else None
|
||||
from paperforge.config import paperforge_paths
|
||||
|
||||
paths = paperforge_paths(vault)
|
||||
idx_path = paths.get("index")
|
||||
if idx_path and idx_path.exists():
|
||||
import json
|
||||
|
||||
data = json.loads(idx_path.read_text(encoding="utf-8"))
|
||||
items = data.get("items", []) if isinstance(data, dict) else data
|
||||
done = sum(1 for i in (items or []) if i.get("ocr_status") == "done")
|
||||
|
|
|
|||
|
|
@ -33,22 +33,29 @@ def _layer(status: str, evidence: list, next_action: str = "", repair_command: s
|
|||
def _check_bootstrap(vault: Path) -> dict:
|
||||
pf_json = vault / "paperforge.json"
|
||||
if not pf_json.exists():
|
||||
return _layer("blocked", [f"paperforge.json not found at {vault}"],
|
||||
"Run paperforge setup to initialize vault",
|
||||
"paperforge setup --headless")
|
||||
return _layer(
|
||||
"blocked",
|
||||
[f"paperforge.json not found at {vault}"],
|
||||
"Run paperforge setup to initialize vault",
|
||||
"paperforge setup --headless",
|
||||
)
|
||||
try:
|
||||
import json
|
||||
|
||||
cfg = json.loads(pf_json.read_text(encoding="utf-8"))
|
||||
except Exception as e:
|
||||
return _layer("blocked", [f"Cannot read paperforge.json: {e}"],
|
||||
"Fix paperforge.json syntax",
|
||||
"paperforge doctor")
|
||||
return _layer(
|
||||
"blocked", [f"Cannot read paperforge.json: {e}"], "Fix paperforge.json syntax", "paperforge doctor"
|
||||
)
|
||||
system_dir = cfg.get("vault_config", {}).get("system_dir") or cfg.get("system_dir", "System")
|
||||
pf_root = vault / system_dir / "PaperForge"
|
||||
if not pf_root.exists():
|
||||
return _layer("degraded", [f"PaperForge directory not found at {pf_root}"],
|
||||
"Run setup to create directory structure",
|
||||
"paperforge doctor")
|
||||
return _layer(
|
||||
"degraded",
|
||||
[f"PaperForge directory not found at {pf_root}"],
|
||||
"Run setup to create directory structure",
|
||||
"paperforge doctor",
|
||||
)
|
||||
return _layer("ok", [f"paperforge.json ok at {pf_json}"])
|
||||
|
||||
|
||||
|
|
@ -59,29 +66,36 @@ def _check_read(vault: Path) -> dict:
|
|||
paths = paperforge_paths(vault)
|
||||
index_path = paths.get("index")
|
||||
if index_path and not Path(index_path).exists():
|
||||
return _layer("degraded", ["Canonical index not found"],
|
||||
"Run paperforge sync --rebuild-index",
|
||||
"paperforge sync --rebuild-index")
|
||||
return _layer(
|
||||
"degraded",
|
||||
["Canonical index not found"],
|
||||
"Run paperforge sync --rebuild-index",
|
||||
"paperforge sync --rebuild-index",
|
||||
)
|
||||
|
||||
db_path = get_memory_db_path(vault)
|
||||
if not db_path.exists():
|
||||
return _layer("degraded", ["Memory DB not found, can be rebuilt"],
|
||||
"Run paperforge memory build",
|
||||
"paperforge memory build")
|
||||
return _layer(
|
||||
"degraded",
|
||||
["Memory DB not found, can be rebuilt"],
|
||||
"Run paperforge memory build",
|
||||
"paperforge memory build",
|
||||
)
|
||||
|
||||
logs_dir = paths.get("paperforge", vault / "System" / "PaperForge") / "logs"
|
||||
logs_dir = paths["paperforge"] / "logs"
|
||||
if not logs_dir.exists():
|
||||
return _layer("degraded", ["JSONL logs directory missing"],
|
||||
"Create logs directory or run sync",
|
||||
"mkdir -p \"$logs_dir\"")
|
||||
return _layer(
|
||||
"degraded", ["JSONL logs directory missing"], "Create logs directory or run sync", 'mkdir -p "$logs_dir"'
|
||||
)
|
||||
|
||||
return _layer("ok", ["Canonical index found", "Memory DB exists", "JSONL logs dir found"])
|
||||
|
||||
|
||||
def _check_write(vault: Path) -> dict:
|
||||
from paperforge.config import paperforge_paths
|
||||
|
||||
paths = paperforge_paths(vault)
|
||||
pf_root = paths.get("paperforge", vault / "System" / "PaperForge")
|
||||
pf_root = paths["paperforge"]
|
||||
logs_dir = pf_root / "logs"
|
||||
evidence = []
|
||||
try:
|
||||
|
|
@ -91,30 +105,25 @@ def _check_write(vault: Path) -> dict:
|
|||
test_file.unlink()
|
||||
evidence.append("JSONL logs dir writable")
|
||||
except Exception as e:
|
||||
return _layer("blocked", [f"JSONL logs dir not writable: {e}"],
|
||||
"Check filesystem permissions", "")
|
||||
return _layer("blocked", [f"JSONL logs dir not writable: {e}"], "Check filesystem permissions", "")
|
||||
return _layer("ok", evidence)
|
||||
|
||||
|
||||
def _check_index(vault: Path) -> dict:
|
||||
from paperforge.memory.db import get_connection, get_memory_db_path
|
||||
|
||||
db_path = get_memory_db_path(vault)
|
||||
if not db_path.exists():
|
||||
return _layer("blocked", ["Memory DB not found"],
|
||||
"Run paperforge memory build",
|
||||
"paperforge memory build")
|
||||
return _layer("blocked", ["Memory DB not found"], "Run paperforge memory build", "paperforge memory build")
|
||||
try:
|
||||
conn = get_connection(db_path, read_only=True)
|
||||
version = conn.execute("SELECT value FROM meta WHERE key='schema_version'").fetchone()
|
||||
conn.close()
|
||||
if version:
|
||||
return _layer("ok", [f"Schema version: {version['value']}"])
|
||||
return _layer("degraded", ["Schema version unknown"],
|
||||
"Rebuild memory DB", "paperforge memory build")
|
||||
return _layer("degraded", ["Schema version unknown"], "Rebuild memory DB", "paperforge memory build")
|
||||
except Exception as e:
|
||||
return _layer("blocked", [f"DB query failed: {e}"],
|
||||
"Rebuild memory DB",
|
||||
"paperforge memory build")
|
||||
return _layer("blocked", [f"DB query failed: {e}"], "Rebuild memory DB", "paperforge memory build")
|
||||
|
||||
|
||||
def _check_vector(vault: Path) -> dict:
|
||||
|
|
@ -125,6 +134,7 @@ def _check_vector(vault: Path) -> dict:
|
|||
if settings_path.exists():
|
||||
try:
|
||||
import json
|
||||
|
||||
data = json.loads(settings_path.read_text(encoding="utf-8"))
|
||||
vector_enabled = bool(data.get("features", {}).get("vector_db", False))
|
||||
except Exception:
|
||||
|
|
@ -136,19 +146,20 @@ def _check_vector(vault: Path) -> dict:
|
|||
build_state = read_vector_build_state(vault)
|
||||
job_status = build_state.get("status", "idle")
|
||||
if job_status == "running":
|
||||
return _layer("degraded", ["Vector build in progress"],
|
||||
"Wait for build to complete",
|
||||
"paperforge embed status --json")
|
||||
return _layer(
|
||||
"degraded", ["Vector build in progress"], "Wait for build to complete", "paperforge embed status --json"
|
||||
)
|
||||
if job_status == "failed":
|
||||
return _layer("degraded", [f"Last build failed: {build_state.get('message', '')}"],
|
||||
"Check error and rebuild",
|
||||
"paperforge embed build --resume")
|
||||
return _layer(
|
||||
"degraded",
|
||||
[f"Last build failed: {build_state.get('message', '')}"],
|
||||
"Check error and rebuild",
|
||||
"paperforge embed build --resume",
|
||||
)
|
||||
|
||||
db_path = get_vector_db_path(vault)
|
||||
if not db_path.exists():
|
||||
return _layer("degraded", ["Vector DB not built yet"],
|
||||
"Run embed build",
|
||||
"paperforge embed build --resume")
|
||||
return _layer("degraded", ["Vector DB not built yet"], "Run embed build", "paperforge embed build --resume")
|
||||
|
||||
return _layer("ok", ["Vector DB exists"])
|
||||
|
||||
|
|
|
|||
|
|
@ -2192,8 +2192,8 @@ class PaperForgeStatusView extends ItemView {
|
|||
}
|
||||
|
||||
_fallbackFetchStats(quiet, vp, plugin) {
|
||||
const systemDir = plugin?.settings?.system_dir || 'System';
|
||||
const indexPath = path.join(vp, systemDir, 'PaperForge', 'indexes', 'formal-library.json');
|
||||
const runtimePaths = resolveVaultPaths(vp);
|
||||
const indexPath = path.join(runtimePaths.indexesDir, 'formal-library.json');
|
||||
|
||||
try {
|
||||
const raw = fs.readFileSync(indexPath, 'utf-8');
|
||||
|
|
@ -2301,9 +2301,8 @@ class PaperForgeStatusView extends ItemView {
|
|||
/* ── Index Loading (D-11, D-17, D-19) ── */
|
||||
_loadIndex() {
|
||||
const vp = this.app.vault.adapter.basePath;
|
||||
const plugin = this.app.plugins.plugins['paperforge'];
|
||||
const systemDir = plugin?.settings?.system_dir || 'System';
|
||||
const indexPath = path.join(vp, systemDir, 'PaperForge', 'indexes', 'formal-library.json');
|
||||
const runtimePaths = resolveVaultPaths(vp);
|
||||
const indexPath = path.join(runtimePaths.indexesDir, 'formal-library.json');
|
||||
try {
|
||||
const raw = fs.readFileSync(indexPath, 'utf-8');
|
||||
return JSON.parse(raw);
|
||||
|
|
@ -2754,11 +2753,11 @@ class PaperForgeStatusView extends ItemView {
|
|||
indexOk ? (index.items.length + ' entries') : 'formal-library.json not found');
|
||||
|
||||
// Zotero export (check if any JSON export exists)
|
||||
const systemDir = plugin?.settings?.system_dir || 'System';
|
||||
const vp = this.app.vault.adapter.basePath;
|
||||
const runtimePaths = resolveVaultPaths(vp);
|
||||
let exportOk = false, exportDetail = 'No exports found';
|
||||
try {
|
||||
const exportsDir = path.join(vp, systemDir, 'PaperForge', 'exports');
|
||||
const exportsDir = runtimePaths.exportsDir;
|
||||
if (fs.existsSync(exportsDir)) {
|
||||
const files = fs.readdirSync(exportsDir).filter(f => f.endsWith('.json'));
|
||||
exportOk = files.length > 0;
|
||||
|
|
@ -2771,8 +2770,7 @@ class PaperForgeStatusView extends ItemView {
|
|||
let tokenOk = !!(plugin?.settings?.paddleocr_api_key);
|
||||
if (!tokenOk) {
|
||||
try {
|
||||
const sysDir = plugin?.settings?.system_dir || 'System';
|
||||
const envPath = path.join(vp, sysDir, 'PaperForge', '.env');
|
||||
const envPath = path.join(runtimePaths.systemDir, '.env');
|
||||
if (fs.existsSync(envPath)) {
|
||||
const envContent = fs.readFileSync(envPath, 'utf-8');
|
||||
const tokenMatch = envContent.match(/^PADDLEOCR_API_TOKEN\s*=\s*(.+)$/m);
|
||||
|
|
|
|||
BIN
paperforge/plugin/sql-wasm.wasm
Normal file
BIN
paperforge/plugin/sql-wasm.wasm
Normal file
Binary file not shown.
|
|
@ -42,11 +42,9 @@ class SyncService:
|
|||
|
||||
def load_exports(self) -> list[dict]:
|
||||
"""Load all BBT export JSON files."""
|
||||
from paperforge.config import load_vault_config
|
||||
from paperforge.config import paperforge_paths
|
||||
|
||||
cfg = load_vault_config(self.vault)
|
||||
system_dir = self.vault / cfg.get("system_dir", "99_System")
|
||||
exports_dir = system_dir / "PaperForge" / "exports"
|
||||
exports_dir = paperforge_paths(self.vault)["exports"]
|
||||
if not exports_dir.exists():
|
||||
logger.warning("Exports directory not found: %s", exports_dir)
|
||||
return []
|
||||
|
|
@ -199,8 +197,13 @@ class SyncService:
|
|||
return candidates
|
||||
|
||||
def run(
|
||||
self, verbose: bool = False, json_output: bool = False, selection_only: bool = False, index_only: bool = False,
|
||||
prune: bool = False, prune_force: bool = False,
|
||||
self,
|
||||
verbose: bool = False,
|
||||
json_output: bool = False,
|
||||
selection_only: bool = False,
|
||||
index_only: bool = False,
|
||||
prune: bool = False,
|
||||
prune_force: bool = False,
|
||||
) -> PFResult:
|
||||
"""Full sync orchestration. Returns PFResult contract.
|
||||
|
||||
|
|
@ -228,9 +231,10 @@ class SyncService:
|
|||
|
||||
# ── Phase 1: Select ──
|
||||
if not index_only:
|
||||
from paperforge.worker.sync import run_selection_sync
|
||||
import time as _time
|
||||
|
||||
from paperforge.worker.sync import run_selection_sync
|
||||
|
||||
_t0 = _time.time()
|
||||
try:
|
||||
selection_result = run_selection_sync(self.vault, verbose=verbose, json_output=json_output)
|
||||
|
|
@ -303,9 +307,11 @@ class SyncService:
|
|||
print("prune: dry-run (pass --prune-force to delete)")
|
||||
else:
|
||||
counts = pdata.get("counts", {})
|
||||
msg = (f"prune: deleted {len(pdata.get('deleted', []))} paper(s) "
|
||||
f"(ws={counts.get('workspace',0)} ocr={counts.get('ocr',0)} "
|
||||
f"vec={counts.get('vectors',0)})")
|
||||
msg = (
|
||||
f"prune: deleted {len(pdata.get('deleted', []))} paper(s) "
|
||||
f"(ws={counts.get('workspace', 0)} ocr={counts.get('ocr', 0)} "
|
||||
f"vec={counts.get('vectors', 0)})"
|
||||
)
|
||||
print(msg)
|
||||
else:
|
||||
print("prune: no orphans found")
|
||||
|
|
@ -342,8 +348,10 @@ class SyncService:
|
|||
"""Run orphan paper cleanup. Default dry_run=True (preview only)."""
|
||||
if fresh_index is None:
|
||||
from paperforge.worker.asset_index import read_index
|
||||
|
||||
fresh_index = read_index(self.vault)
|
||||
from paperforge.worker.prune import prune_orphan_papers
|
||||
|
||||
return prune_orphan_papers(self.vault, fresh_index=fresh_index, dry_run=dry_run)
|
||||
|
||||
# ── Legacy passthrough (backward-compat for commands/sync.py) ──
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import sys
|
|||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from paperforge.config import load_vault_config
|
||||
from paperforge.setup import SetupStepResult
|
||||
|
||||
|
||||
|
|
@ -45,7 +46,7 @@ class SetupChecker:
|
|||
details["zotero_found"] = zotero_path is not None
|
||||
|
||||
# Check Better BibTeX exports
|
||||
system_dir = self.vault / "99_System"
|
||||
system_dir = self.vault / load_vault_config(self.vault).get("system_dir", "System")
|
||||
exports_dir = system_dir / "PaperForge" / "exports"
|
||||
bbt_exists = exports_dir.exists() and len(list(exports_dir.glob("*.json"))) > 0
|
||||
details["bbt_exports_found"] = bbt_exists
|
||||
|
|
|
|||
|
|
@ -13,8 +13,8 @@ class VaultInitializer:
|
|||
"""Create vault directory structure, Zotero junction, and .env file."""
|
||||
|
||||
DEFAULT_DIRS = {
|
||||
"system_dir": "99_System",
|
||||
"resources_dir": "03_Resources",
|
||||
"system_dir": "System",
|
||||
"resources_dir": "Resources",
|
||||
"literature_dir": "Literature",
|
||||
# control_dir 已淘汰 (v2.1+ workspace 架构)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -269,6 +269,7 @@ def _build_entry(item: dict, vault: Path, paths: dict, domain: str, zotero_dir:
|
|||
import shutil
|
||||
|
||||
from paperforge import __version__ as PAPERFORGE_VERSION
|
||||
from paperforge.adapters.obsidian_frontmatter import has_deep_reading_content
|
||||
from paperforge.worker._utils import lookup_impact_factor, read_json, slugify_filename, write_json, yaml_quote
|
||||
from paperforge.worker.asset_state import (
|
||||
compute_health,
|
||||
|
|
@ -278,7 +279,6 @@ def _build_entry(item: dict, vault: Path, paths: dict, domain: str, zotero_dir:
|
|||
)
|
||||
from paperforge.worker.ocr import validate_ocr_meta
|
||||
from paperforge.worker.paper_meta import write_paper_meta
|
||||
from paperforge.adapters.obsidian_frontmatter import has_deep_reading_content
|
||||
from paperforge.worker.sync import (
|
||||
collection_fields,
|
||||
frontmatter_note,
|
||||
|
|
@ -369,7 +369,7 @@ def _build_entry(item: dict, vault: Path, paths: dict, domain: str, zotero_dir:
|
|||
note_text = fp.read_text(encoding="utf-8")
|
||||
fm = read_frontmatter_dict(note_text)
|
||||
fm_cached_text = note_text
|
||||
fm_was_main = (fp == main_note_path)
|
||||
fm_was_main = fp == main_note_path
|
||||
break
|
||||
except Exception:
|
||||
continue
|
||||
|
|
@ -469,8 +469,10 @@ def _build_entry(item: dict, vault: Path, paths: dict, domain: str, zotero_dir:
|
|||
else:
|
||||
main_note_path.write_text(frontmatter_note(entry, text), encoding="utf-8")
|
||||
else:
|
||||
existing_text = fm_cached_text if not fm_was_main and fm_cached_text else (
|
||||
note_path.read_text(encoding="utf-8") if note_path.exists() else ""
|
||||
existing_text = (
|
||||
fm_cached_text
|
||||
if not fm_was_main and fm_cached_text
|
||||
else (note_path.read_text(encoding="utf-8") if note_path.exists() else "")
|
||||
)
|
||||
main_note_path.write_text(frontmatter_note(entry, existing_text), encoding="utf-8")
|
||||
|
||||
|
|
@ -495,6 +497,7 @@ def _vec_auto_embed_if_new(vault: Path, entry: dict) -> None:
|
|||
from paperforge.embedding import embed_paper, get_vector_db_path
|
||||
from paperforge.embedding._config import _read_plugin_settings
|
||||
from paperforge.memory.chunker import chunk_fulltext
|
||||
|
||||
settings = _read_plugin_settings(vault)
|
||||
if not settings.get("features", {}).get("vector_db", False):
|
||||
return
|
||||
|
|
@ -508,6 +511,7 @@ def _vec_auto_embed_if_new(vault: Path, entry: dict) -> None:
|
|||
except Exception:
|
||||
pass # ChromaDB / model not installed — silently skip
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Full index build
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -533,7 +537,6 @@ def build_index(vault: Path, verbose: bool = False) -> int:
|
|||
Returns:
|
||||
Number of items written to (or already present in) the index.
|
||||
"""
|
||||
from paperforge.config import load_vault_config
|
||||
from paperforge.worker._utils import pipeline_paths # noqa: F811
|
||||
from paperforge.worker.base_views import ensure_base_views
|
||||
from paperforge.worker.sync import load_domain_config, load_export_rows
|
||||
|
|
@ -553,8 +556,7 @@ def build_index(vault: Path, verbose: bool = False) -> int:
|
|||
f"need {CURRENT_SCHEMA_VERSION}. Rebuilding..."
|
||||
)
|
||||
|
||||
cfg = load_vault_config(vault)
|
||||
zotero_dir = vault / cfg.get("system_dir", "System") / "Zotero"
|
||||
zotero_dir = paths["zotero_dir"]
|
||||
|
||||
export_hash = _compute_export_hash(paths)
|
||||
if isinstance(existing_data, dict) and existing_data.get("export_hash") == export_hash:
|
||||
|
|
@ -608,7 +610,6 @@ def refresh_index_entry(vault: Path, key: str) -> bool:
|
|||
``True`` if incremental refresh was performed,
|
||||
``False`` if a full rebuild was triggered instead.
|
||||
"""
|
||||
from paperforge.config import load_vault_config
|
||||
from paperforge.worker._utils import pipeline_paths
|
||||
from paperforge.worker.sync import load_domain_config, load_export_rows
|
||||
|
||||
|
|
@ -627,8 +628,7 @@ def refresh_index_entry(vault: Path, key: str) -> bool:
|
|||
# Find the export item for the requested key
|
||||
config = load_domain_config(paths)
|
||||
domain_lookup = {entry["export_file"]: entry["domain"] for entry in config["domains"]}
|
||||
cfg = load_vault_config(vault)
|
||||
zotero_dir = vault / cfg.get("system_dir", "System") / "Zotero"
|
||||
zotero_dir = paths["zotero_dir"]
|
||||
|
||||
found_item = None
|
||||
found_domain = ""
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ def _collect_orphan_candidates(lit_dir: Path, fresh_keys: set[str]) -> list[dict
|
|||
|
||||
def _resolve_ocr_dir(vault: Path, key: str) -> Path:
|
||||
cfg = paperforge_paths(vault)
|
||||
ocr_root = cfg.get("ocr", vault / "System" / "PaperForge" / "ocr")
|
||||
ocr_root = cfg["ocr"]
|
||||
return ocr_root / key
|
||||
|
||||
|
||||
|
|
@ -76,18 +76,20 @@ def _enrich_orphan_preview(vault: Path, candidates: list[dict]) -> list[dict]:
|
|||
if isinstance(coll, list):
|
||||
coll = " | ".join(coll)
|
||||
|
||||
enriched.append({
|
||||
"key": key,
|
||||
"citation_key": fm.get("citation_key") or key,
|
||||
"title": title,
|
||||
"year": str(fm.get("year", "")),
|
||||
"authors": first_author,
|
||||
"has_pdf": fm.get("has_pdf", False) in (True, "true"),
|
||||
"collection_path": coll,
|
||||
"domain": c["domain"],
|
||||
"workspace": str(ws),
|
||||
"ocr_dir": str(c["ocr_dir"]) if c["ocr_dir"] and c["ocr_dir"].exists() else None,
|
||||
})
|
||||
enriched.append(
|
||||
{
|
||||
"key": key,
|
||||
"citation_key": fm.get("citation_key") or key,
|
||||
"title": title,
|
||||
"year": str(fm.get("year", "")),
|
||||
"authors": first_author,
|
||||
"has_pdf": fm.get("has_pdf", False) in (True, "true"),
|
||||
"collection_path": coll,
|
||||
"domain": c["domain"],
|
||||
"workspace": str(ws),
|
||||
"ocr_dir": str(c["ocr_dir"]) if c["ocr_dir"] and c["ocr_dir"].exists() else None,
|
||||
}
|
||||
)
|
||||
|
||||
return enriched
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,6 @@ import logging
|
|||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from paperforge.config import load_vault_config
|
||||
from paperforge.worker._utils import (
|
||||
read_json,
|
||||
write_json,
|
||||
|
|
@ -89,8 +88,7 @@ def repair_pdf_paths(
|
|||
fixed = 0
|
||||
from paperforge.pdf_resolver import resolve_pdf_path
|
||||
|
||||
cfg = load_vault_config(vault)
|
||||
zotero_dir = vault / cfg.get("system_dir", "System") / "Zotero"
|
||||
zotero_dir = paths["zotero_dir"]
|
||||
|
||||
# Cache export rows by domain to avoid reloading
|
||||
domain_exports: dict[str, list[dict]] = {}
|
||||
|
|
|
|||
|
|
@ -34,7 +34,6 @@ from paperforge.adapters.zotero_paths import (
|
|||
obsidian_wikilink_for_path,
|
||||
obsidian_wikilink_for_pdf,
|
||||
)
|
||||
from paperforge.config import load_vault_config
|
||||
from paperforge.worker._domain import load_domain_collections, load_domain_config
|
||||
from paperforge.worker._utils import (
|
||||
_extract_year,
|
||||
|
|
@ -218,8 +217,7 @@ def run_selection_sync(vault: Path, verbose: bool = False, json_output: bool = F
|
|||
raw_pdf_path = pdf_attachments[0].get("path", "") if pdf_attachments else ""
|
||||
from paperforge.pdf_resolver import resolve_pdf_path
|
||||
|
||||
cfg = load_vault_config(vault)
|
||||
zotero_dir = vault / cfg.get("system_dir", "System") / "Zotero"
|
||||
zotero_dir = paths["zotero_dir"]
|
||||
resolved_pdf = resolve_pdf_path(raw_pdf_path, has_pdf, vault, zotero_dir)
|
||||
collection_meta = collection_fields(item.get("collections", []))
|
||||
meta_path = paths["ocr"] / item["key"] / "meta.json"
|
||||
|
|
@ -1221,8 +1219,7 @@ def run_index_refresh(
|
|||
ensure_base_views(vault, paths, config)
|
||||
domain_lookup = {entry["export_file"]: entry["domain"] for entry in config["domains"]}
|
||||
|
||||
cfg = load_vault_config(vault)
|
||||
zotero_dir = vault / cfg.get("system_dir", "System") / "Zotero"
|
||||
zotero_dir = paths["zotero_dir"]
|
||||
exports = {}
|
||||
for export_path in sorted(paths["exports"].glob("*.json")):
|
||||
domain = domain_lookup.get(export_path.name, export_path.stem)
|
||||
|
|
|
|||
77
patch-plan/annotation-schema-v3.md
Normal file
77
patch-plan/annotation-schema-v3.md
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
# Annotation Schema v1 (Independent, not v3)
|
||||
|
||||
> Note: While PaperForge's memory layer is at schema v2, annotation schema starts at v1 independently.
|
||||
|
||||
## DB Location
|
||||
|
||||
```
|
||||
<vault>/System/PaperForge/indexes/annotations.db
|
||||
```
|
||||
|
||||
## Schema Version Management
|
||||
|
||||
```python
|
||||
ANNOTATIONS_SCHEMA_VERSION = 1
|
||||
|
||||
def get_annotations_schema_version(conn):
|
||||
try:
|
||||
row = conn.execute("SELECT value FROM meta WHERE key = 'schema_version'").fetchone()
|
||||
return int(row["value"]) if row else 0
|
||||
except sqlite3.OperationalError:
|
||||
return 0
|
||||
|
||||
def ensure_annotations_schema(conn):
|
||||
stored = get_annotations_schema_version(conn)
|
||||
if stored == 0:
|
||||
_create_annotations_schema(conn)
|
||||
conn.execute("INSERT OR REPLACE INTO meta (key, value) VALUES ('schema_version', '1')")
|
||||
conn.commit()
|
||||
elif stored < ANNOTATIONS_SCHEMA_VERSION:
|
||||
_migrate_annotations_schema(conn, stored)
|
||||
```
|
||||
|
||||
## Full DDL
|
||||
|
||||
See `reports/03-paperforge-schema-design.md` for complete DDL.
|
||||
|
||||
### Tables
|
||||
|
||||
| Table | Purpose |
|
||||
|-------|---------|
|
||||
| `meta` | Schema version, last import timestamp, stats |
|
||||
| `annotations` | Main annotation data |
|
||||
| `annotations_fts` | FTS5 virtual table for full-text search |
|
||||
| `sync_queue` | Pending write-back operations (future) |
|
||||
|
||||
### Indexes
|
||||
|
||||
```sql
|
||||
idx_annotations_paper — paper_id
|
||||
idx_annotations_type — type
|
||||
idx_annotations_sync_state — sync_state
|
||||
idx_annotations_source — source
|
||||
idx_annotations_page — (paper_id, page_index)
|
||||
idx_annotations_zotero_key — zotero_key
|
||||
idx_annotations_deleted — partial index on deleted_at IS NOT NULL
|
||||
```
|
||||
|
||||
## Migration Strategy (Future)
|
||||
|
||||
| Version | Changes |
|
||||
|---------|---------|
|
||||
| 1 (MVP) | Initial schema as designed |
|
||||
| 2 | Web API write-back support: add `api_response_json`, `last_push_attempt` columns |
|
||||
| 3 | Group library support: add `zotero_group_id` column |
|
||||
|
||||
## Cross-DB Note
|
||||
|
||||
Annotations DB is independent from paperforge.db. To correlate:
|
||||
|
||||
```python
|
||||
# Get all annotations for a paper
|
||||
def get_paper_annotations(ann_conn, paper_zotero_key):
|
||||
return ann_conn.execute(
|
||||
"SELECT * FROM annotations WHERE paper_id = ? AND deleted_at IS NULL ORDER BY sort_index",
|
||||
(paper_zotero_key,)
|
||||
).fetchall()
|
||||
```
|
||||
150
patch-plan/annotation-service-api.md
Normal file
150
patch-plan/annotation-service-api.md
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
# Annotation Service API Design
|
||||
|
||||
> Python service layer for annotation operations.
|
||||
|
||||
## Module: `paperforge/annotation/__init__.py`
|
||||
|
||||
```python
|
||||
"""
|
||||
PaperForge Annotation Layer.
|
||||
|
||||
Read-only Zotero SQLite parsing → annotations.db cache → CLI commands → Obsidian overlay.
|
||||
"""
|
||||
```
|
||||
|
||||
## Module: `paperforge/annotation/probe.py`
|
||||
|
||||
```python
|
||||
class ZoteroAnnotationProbe:
|
||||
"""Read-only probe into Zotero SQLite for annotations."""
|
||||
|
||||
def __init__(self, zotero_db_path: Path):
|
||||
self.zotero_db_path = zotero_db_path
|
||||
|
||||
def copy_to_temp(self) -> Path:
|
||||
"""Copy zotero.sqlite to temp file. Returns temp path."""
|
||||
|
||||
def open_readonly(self, db_path: Path) -> sqlite3.Connection:
|
||||
"""Open SQLite in read-only mode (URI mode=ro)."""
|
||||
|
||||
def fetch_annotations(self, conn, paper_key: str = "", limit: int = 100) -> list[dict]:
|
||||
"""Fetch annotations with parent attachment info."""
|
||||
|
||||
def fetch_tags_for_annotations(self, conn, item_ids: list[int]) -> dict[int, list[str]]:
|
||||
"""Fetch tags for annotation items."""
|
||||
|
||||
def probe(self, limit: int = 20) -> dict:
|
||||
"""Full probe: schema check → fetch → return unified JSON."""
|
||||
```
|
||||
|
||||
## Module: `paperforge/annotation/db.py`
|
||||
|
||||
```python
|
||||
class AnnotationDB:
|
||||
"""Management of annotations.db."""
|
||||
|
||||
def __init__(self, vault: Path):
|
||||
self.vault = vault
|
||||
self.db_path = vault / "System" / "PaperForge" / "indexes" / "annotations.db"
|
||||
|
||||
def get_connection(self, read_only: bool = False) -> sqlite3.Connection:
|
||||
"""Open connection with WAL mode."""
|
||||
|
||||
def ensure_schema(self):
|
||||
"""Create tables if not exist, migrate if needed."""
|
||||
|
||||
def integrity_check(self) -> bool:
|
||||
"""PRAGMA integrity_check."""
|
||||
|
||||
def get_stats(self) -> dict:
|
||||
"""Return count by type, source, sync_state."""
|
||||
```
|
||||
|
||||
## Module: `paperforge/annotation/import_annotations.py`
|
||||
|
||||
```python
|
||||
def import_from_zotero(
|
||||
vault: Path,
|
||||
zotero_db_path: Path | None = None,
|
||||
paper_key: str = "",
|
||||
dry_run: bool = False,
|
||||
) -> dict:
|
||||
"""Import annotations from Zotero SQLite into annotations.db.
|
||||
|
||||
Steps:
|
||||
1. Detect/copy Zotero SQLite
|
||||
2. Open read-only connection
|
||||
3. Fetch annotations (optionally filtered by paper_key)
|
||||
4. For each annotation:
|
||||
a. Check if already imported (by zotero_key + source_version)
|
||||
b. If new: INSERT with sync_state='zotero_synced', is_readonly=1
|
||||
c. If updated: UPDATE with new position/comment/text, bump version
|
||||
d. If deleted: soft delete (set deleted_at)
|
||||
5. Commit
|
||||
6. Return import stats
|
||||
"""
|
||||
```
|
||||
|
||||
## Module: `paperforge/annotation/export.py`
|
||||
|
||||
```python
|
||||
def export_json(ann_conn, paper_key: str) -> str:
|
||||
"""Export annotations for a paper as pretty JSON."""
|
||||
|
||||
def export_markdown(ann_conn, paper_key: str, paper_title: str) -> str:
|
||||
"""Export annotations as formatted Markdown."""
|
||||
```
|
||||
|
||||
## CLI: `paperforge/commands/annotation.py`
|
||||
|
||||
```python
|
||||
def run(args) -> int:
|
||||
"""Dispatch annotation subcommands."""
|
||||
if args.annotation_action == "import":
|
||||
return _cmd_import(args)
|
||||
elif args.annotation_action == "list":
|
||||
return _cmd_list(args)
|
||||
elif args.annotation_action == "create":
|
||||
return _cmd_create(args)
|
||||
elif args.annotation_action == "patch":
|
||||
return _cmd_patch(args)
|
||||
elif args.annotation_action == "delete":
|
||||
return _cmd_delete(args)
|
||||
elif args.annotation_action == "export":
|
||||
return _cmd_export(args)
|
||||
elif args.annotation_action == "status":
|
||||
return _cmd_status(args)
|
||||
```
|
||||
|
||||
## JSON Output Contract
|
||||
|
||||
All commands support `--json` flag. Output envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"command": "annotation <subcommand>",
|
||||
"data": {
|
||||
"paper_id": "ABCD1234",
|
||||
"annotations": [...],
|
||||
"count": 15
|
||||
},
|
||||
"meta": {
|
||||
"db_path": "/path/to/annotations.db",
|
||||
"elapsed_ms": 123
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Error envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": false,
|
||||
"command": "annotation <subcommand>",
|
||||
"error": {
|
||||
"code": "ZOTERO_DB_NOT_FOUND",
|
||||
"message": "Zotero SQLite not found at /path/to/zotero.sqlite"
|
||||
}
|
||||
}
|
||||
```
|
||||
396
patch-plan/plugin-overlay-adapter.md
Normal file
396
patch-plan/plugin-overlay-adapter.md
Normal file
|
|
@ -0,0 +1,396 @@
|
|||
# Plugin Overlay Adapter Design
|
||||
|
||||
> Obsidian plugin side of the annotation overlay system.
|
||||
|
||||
## Module Structure
|
||||
|
||||
```
|
||||
plugin/src/
|
||||
├── main.ts ← extended: add annotation commands
|
||||
├── pdf-overlay/
|
||||
│ ├── patch-pdf-viewer.ts ← monkey-around patches
|
||||
│ ├── overlay-layer.ts ← overlay DOM management
|
||||
│ ├── rect-renderer.ts ← rect placement + colors
|
||||
│ ├── selection-handler.ts ← text selection → annotation
|
||||
│ ├── popover.ts ← annotation popover
|
||||
│ ├── annotation-fetcher.ts ← execFile bridge
|
||||
│ └── types.ts ← interfaces
|
||||
└── testable.js ← extended
|
||||
```
|
||||
|
||||
## Patch Sequence (in `patch-pdf-viewer.ts`)
|
||||
|
||||
```typescript
|
||||
import { around } from 'monkey-around';
|
||||
import { Plugin } from 'obsidian';
|
||||
|
||||
export function patchPDFViewer(plugin: Plugin): boolean {
|
||||
// Step 1: Get PDFView constructor
|
||||
const pdfLeaves = plugin.app.workspace.getLeavesOfType('pdf');
|
||||
if (!pdfLeaves.length) return false;
|
||||
const pdfView = pdfLeaves[0].view as any;
|
||||
const PDFViewerChildProto = pdfView.constructor.prototype;
|
||||
|
||||
// Step 2: Patch loadFile to inject annotation overlay
|
||||
plugin.register(around(PDFViewerChildProto, {
|
||||
loadFile(old: Function) {
|
||||
return function (this: any, file: any) {
|
||||
const result = old.call(this, file);
|
||||
initOverlayForFile(this, file);
|
||||
return result;
|
||||
};
|
||||
}
|
||||
}));
|
||||
|
||||
// Step 3: Patch load to register event listeners
|
||||
plugin.register(around(PDFViewerChildProto, {
|
||||
load(old: Function) {
|
||||
return function (this: any) {
|
||||
const result = old.call(this);
|
||||
registerOverlayEvents(this);
|
||||
return result;
|
||||
};
|
||||
},
|
||||
unload(old: Function) {
|
||||
return function (this: any) {
|
||||
cleanupOverlay(this);
|
||||
return old.call(this);
|
||||
};
|
||||
}
|
||||
}));
|
||||
|
||||
return true;
|
||||
}
|
||||
```
|
||||
|
||||
## Overlay Layer Management (in `overlay-layer.ts`)
|
||||
|
||||
```typescript
|
||||
export class OverlayLayerManager {
|
||||
private pageLayers: Map<number, HTMLElement> = new Map();
|
||||
|
||||
getOrCreateLayer(pageView: any): HTMLElement {
|
||||
const pageDiv = pageView.div;
|
||||
let layer = pageDiv.querySelector('div.pf-annotation-overlay') as HTMLElement;
|
||||
if (!layer) {
|
||||
layer = pageDiv.createDiv('pf-annotation-overlay');
|
||||
window.pdfjsLib.setLayerDimensions(layer, pageView.viewport);
|
||||
}
|
||||
return layer;
|
||||
}
|
||||
|
||||
clearPage(pageNumber: number): void {
|
||||
const layer = this.pageLayers.get(pageNumber);
|
||||
if (layer) layer.empty();
|
||||
}
|
||||
|
||||
clearAll(): void {
|
||||
this.pageLayers.forEach(layer => layer.empty());
|
||||
this.pageLayers.clear();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Rect Renderer (in `rect-renderer.ts`)
|
||||
|
||||
```typescript
|
||||
export interface AnnotationRect {
|
||||
id: string;
|
||||
type: 'highlight' | 'underline' | 'note';
|
||||
rect: [number, number, number, number]; // PDF coords [left, bottom, right, top]
|
||||
color: string;
|
||||
comment?: string;
|
||||
isReadonly: boolean;
|
||||
}
|
||||
|
||||
export class RectRenderer {
|
||||
render(rects: AnnotationRect[], pageView: any, layer: HTMLElement): void {
|
||||
const viewport = pageView.viewport;
|
||||
const [pageX, pageY, pageX2, pageY2] = viewport.viewBox;
|
||||
const pageW = pageX2 - pageX;
|
||||
const pageH = pageY2 - pageY;
|
||||
|
||||
for (const ann of rects) {
|
||||
const [left, bottom, right, top] = ann.rect;
|
||||
const mirrored = window.pdfjsLib.Util.normalizeRect([
|
||||
left, pageY2 - bottom + pageY,
|
||||
right, pageY2 - top + pageY
|
||||
]);
|
||||
|
||||
const el = layer.createDiv('pf-annotation-rect');
|
||||
el.dataset.annotationId = ann.id;
|
||||
el.dataset.type = ann.type;
|
||||
|
||||
el.setCssStyles({
|
||||
left: `${100 * (mirrored[0] - pageX) / pageW}%`,
|
||||
top: `${100 * (mirrored[1] - pageY) / pageH}%`,
|
||||
width: `${100 * (mirrored[2] - mirrored[0]) / pageW}%`,
|
||||
height: `${100 * (mirrored[3] - mirrored[1]) / pageH}%`,
|
||||
});
|
||||
|
||||
if (ann.type === 'highlight') {
|
||||
el.style.background = hexToRgba(ann.color, 0.25);
|
||||
} else if (ann.type === 'underline') {
|
||||
el.style.borderBottom = `2px solid ${ann.color}`;
|
||||
} else if (ann.type === 'note') {
|
||||
el.style.background = hexToRgba(ann.color, 0.15);
|
||||
el.style.border = `1px solid ${ann.color}`;
|
||||
}
|
||||
|
||||
if (ann.isReadonly) {
|
||||
el.classList.add('pf-annotation-readonly');
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function hexToRgba(hex: string, alpha: number): string {
|
||||
const r = parseInt(hex.slice(1, 3), 16);
|
||||
const g = parseInt(hex.slice(3, 5), 16);
|
||||
const b = parseInt(hex.slice(5, 7), 16);
|
||||
return `rgba(${r}, ${g}, ${b}, ${alpha})`;
|
||||
}
|
||||
```
|
||||
|
||||
## Selection Handler (in `selection-handler.ts`)
|
||||
|
||||
```typescript
|
||||
export class SelectionHandler {
|
||||
private pendingButton: HTMLElement | null = null;
|
||||
|
||||
handleSelection(pageView: any): void {
|
||||
const selection = window.getSelection();
|
||||
if (!selection || selection.isCollapsed) {
|
||||
this.removePendingButton();
|
||||
return;
|
||||
}
|
||||
|
||||
const range = selection.getRangeAt(0);
|
||||
const textLayer = pageView.textLayer;
|
||||
if (!textLayer) return;
|
||||
|
||||
// Determine page index and selected rects
|
||||
const pageNumber = pageView.id;
|
||||
const selectedText = selection.toString().trim();
|
||||
if (!selectedText) return;
|
||||
|
||||
// Show floating "Add Highlight" button
|
||||
this.showAddButton(pageView, range, selectedText);
|
||||
}
|
||||
|
||||
private async addAnnotation(pageView: any, selectedText: string, rects: number[][]): Promise<void> {
|
||||
const sortIndex = this.buildSortIndex(pageView.id, 0, 0);
|
||||
const position = {
|
||||
pageIndex: pageView.id - 1,
|
||||
rects: rects.map(r => [r[0], r[1], r[2], r[3]]),
|
||||
};
|
||||
|
||||
// execFile → paperforge annotation create
|
||||
const result = await createAnnotation({
|
||||
paperKey: currentPaperKey,
|
||||
type: 'highlight',
|
||||
pageIndex: pageView.id - 1,
|
||||
selectedText,
|
||||
color: '#ffd400',
|
||||
position,
|
||||
sortIndex,
|
||||
});
|
||||
|
||||
// Optimistic add to overlay
|
||||
renderSingleAnnotation(result, pageView);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Popover (in `popover.ts`)
|
||||
|
||||
```typescript
|
||||
export class AnnotationPopover {
|
||||
private popoverEl: HTMLElement | null = null;
|
||||
|
||||
show(annotation: Annotation, rect: DOMRect, pageView: any): void {
|
||||
this.dismiss();
|
||||
|
||||
this.popoverEl = pageView.div.createDiv('pf-annotation-popover');
|
||||
this.popoverEl.setCssStyles({
|
||||
position: 'absolute',
|
||||
left: `${rect.left}px`,
|
||||
top: `${rect.bottom + 8}px`,
|
||||
});
|
||||
|
||||
this.popoverEl.createDiv('pf-popover-header', (el) => {
|
||||
el.setText(annotation.type.toUpperCase());
|
||||
el.style.color = annotation.color;
|
||||
});
|
||||
|
||||
if (annotation.selectedText) {
|
||||
this.popoverEl.createDiv('pf-popover-text', el => el.setText(`"${annotation.selectedText.slice(0, 100)}..."`));
|
||||
}
|
||||
|
||||
if (annotation.comment) {
|
||||
this.popoverEl.createDiv('pf-popover-comment', el => el.setText(annotation.comment));
|
||||
}
|
||||
|
||||
if (!annotation.isReadonly) {
|
||||
const actions = this.popoverEl.createDiv('pf-popover-actions');
|
||||
actions.createEl('button', { text: 'Edit' }, (btn) => {
|
||||
btn.onclick = () => this.enableEdit(annotation, pageView);
|
||||
});
|
||||
actions.createEl('button', { text: 'Delete' }, (btn) => {
|
||||
btn.onclick = () => this.deleteAnnotation(annotation.id, pageView);
|
||||
});
|
||||
} else {
|
||||
this.popoverEl.createDiv('pf-popover-readonly', el => {
|
||||
el.setText('🔒 Zotero annotation (read-only in PaperForge)');
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
dismiss(): void {
|
||||
this.popoverEl?.remove();
|
||||
this.popoverEl = null;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## execFile Bridge (in `annotation-fetcher.ts`)
|
||||
|
||||
```typescript
|
||||
import { execFile } from 'child_process';
|
||||
|
||||
let currentPaperKey: string = '';
|
||||
|
||||
export async function fetchAnnotations(vaultPath: string, pythonExe: string, paperKey: string): Promise<Annotation[]> {
|
||||
currentPaperKey = paperKey;
|
||||
return runAnnotationCommand(vaultPath, pythonExe, ['list', paperKey, '--json']);
|
||||
}
|
||||
|
||||
export async function createAnnotation(vaultPath: string, pythonExe: string, data: CreatePayload): Promise<Annotation> {
|
||||
const args = [
|
||||
'create',
|
||||
'--paper', data.paperKey,
|
||||
'--type', data.type,
|
||||
'--page-index', String(data.pageIndex),
|
||||
'--selected-text', data.selectedText || '',
|
||||
'--comment', data.comment || '',
|
||||
'--color', data.color || '#ffd400',
|
||||
'--position', JSON.stringify(data.position),
|
||||
'--sort-index', data.sortIndex,
|
||||
'--json',
|
||||
];
|
||||
return runAnnotationCommand(vaultPath, pythonExe, args);
|
||||
}
|
||||
|
||||
async function runAnnotationCommand(vaultPath: string, pythonExe: string, args: string[]): Promise<any> {
|
||||
return new Promise((resolve, reject) => {
|
||||
execFile(
|
||||
pythonExe,
|
||||
['-m', 'paperforge', 'annotation', ...args],
|
||||
{ cwd: vaultPath, timeout: 15000 },
|
||||
(err, stdout) => {
|
||||
if (err) {
|
||||
reject(new Error(`Annotation command failed: ${err.message}`));
|
||||
return;
|
||||
}
|
||||
const result = JSON.parse(stdout);
|
||||
if (!result.ok) {
|
||||
reject(new Error(result.error?.message || 'Annotation command failed'));
|
||||
return;
|
||||
}
|
||||
resolve(result.data);
|
||||
}
|
||||
);
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
## CSS Styles (to add to `styles.css`)
|
||||
|
||||
```css
|
||||
/* Annotation overlay layer */
|
||||
.pf-annotation-overlay {
|
||||
position: absolute;
|
||||
top: 0;
|
||||
left: 0;
|
||||
right: 0;
|
||||
bottom: 0;
|
||||
pointer-events: none;
|
||||
z-index: 10;
|
||||
}
|
||||
|
||||
/* Individual annotation rect */
|
||||
.pf-annotation-rect {
|
||||
position: absolute;
|
||||
pointer-events: auto;
|
||||
cursor: pointer;
|
||||
border-radius: 2px;
|
||||
transition: opacity 0.15s ease;
|
||||
}
|
||||
|
||||
.pf-annotation-rect:hover {
|
||||
opacity: 0.6;
|
||||
}
|
||||
|
||||
.pf-annotation-readonly {
|
||||
cursor: default;
|
||||
}
|
||||
|
||||
.pf-annotation-readonly::after {
|
||||
content: '🔒';
|
||||
position: absolute;
|
||||
top: -8px;
|
||||
right: -8px;
|
||||
font-size: 10px;
|
||||
opacity: 0.5;
|
||||
}
|
||||
|
||||
/* Popover */
|
||||
.pf-annotation-popover {
|
||||
position: fixed;
|
||||
z-index: 1000;
|
||||
background: var(--background-primary);
|
||||
border: 1px solid var(--background-modifier-border);
|
||||
border-radius: 8px;
|
||||
padding: 12px;
|
||||
min-width: 200px;
|
||||
max-width: 400px;
|
||||
box-shadow: var(--shadow-l);
|
||||
font-size: var(--font-small);
|
||||
}
|
||||
|
||||
.pf-popover-header {
|
||||
font-weight: bold;
|
||||
font-size: var(--font-smaller);
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.pf-popover-text {
|
||||
font-style: italic;
|
||||
color: var(--text-muted);
|
||||
margin-bottom: 6px;
|
||||
padding: 4px 8px;
|
||||
border-left: 2px solid var(--background-modifier-border);
|
||||
}
|
||||
|
||||
.pf-popover-comment {
|
||||
margin-bottom: 8px;
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
|
||||
.pf-popover-actions {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
justify-content: flex-end;
|
||||
}
|
||||
|
||||
.pf-popover-readonly {
|
||||
color: var(--text-faint);
|
||||
font-size: var(--font-smallest);
|
||||
text-align: center;
|
||||
padding: 4px;
|
||||
}
|
||||
|
||||
/* Dark mode adjustments */
|
||||
.theme-dark .pf-annotation-rect {
|
||||
opacity: 0.7;
|
||||
}
|
||||
112
reports/00-executive-summary.md
Normal file
112
reports/00-executive-summary.md
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
# Executive Summary — PaperForge PDF Annotation Layer
|
||||
|
||||
> ROBCO INDUSTRIES (TM) TERMLINK REPORT
|
||||
> Status: RESEARCH COMPLETE | DESIGN READY
|
||||
|
||||
## The 10 Key Questions Answered
|
||||
|
||||
### 1. ZotFlow 哪些设计值得 PaperForge 借鉴？
|
||||
|
||||
- **AnnotationJSON schema** — ZotFlow 的 `AnnotationJSON` 接口（id, type, position, sortIndex, color, comment, text, tags 等字段）是经过 production 验证的 Zotero 兼容模型
|
||||
- **Sync conflict resolution** — 字段级别 diff + version-based optimistic locking + 三种 conflict action（keep-local / accept-remote / mark-conflict）
|
||||
- **LiquidJS template pipeline** — 模板驱动的 source note 生成可直接复用
|
||||
- **Dexie/IndexedDB 缓存模式** — 在 Python 端可用 SQLite 等价实现，设计思路一致（syncStatus 追踪、全 raw payload 存储）
|
||||
|
||||
### 2. ZotFlow 哪些部分太重，PaperForge 应该避免？
|
||||
|
||||
- **内嵌 Zotero reader iframe** — 整个 reader/ 子模块（Zotero PDF.js + Penpal + Comlink），约 2/3 的代码量
|
||||
- **CodeMirror 6 扩展** — 可编辑区域、锁、comments、citation suggest 等 O(N) 级别的 editor 集成
|
||||
- **React UI 组件** — TreeView（react-arborist）、Activity Center、settings tab 等
|
||||
- **完整的自适应同步引擎** — 42,918 行的 sync.ts，涵盖双向 pull/push、冲突检测、批量重试等，对只读场景完全不需要
|
||||
|
||||
### 3. 本地解析 Zotero SQLite 是否足够支持只读 annotation sync？
|
||||
|
||||
**完全足够。** 所有 annotation 数据存储在 `itemAnnotations` 表 + `items` 表 + `tags` 表 + `itemTags` 表，通过 standard SQL JOIN 可获取全部字段（type, text, comment, color, pageLabel, sortIndex, position JSON, tags, dateModified）。
|
||||
|
||||
Zotero 官方文档明确标注 SQLite 可以被外部工具只读提取。
|
||||
|
||||
注意点：Zotero 运行时 SQLite 有缓存层，必须先 COPY DB 到临时目录再读，避免锁和不一致。
|
||||
|
||||
### 4. Zotero annotation 到 PaperForge annotation 的字段映射
|
||||
|
||||
| Zotero 字段 | PaperForge 字段 | 说明 |
|
||||
| ------------------------- | ------------------ | --------------------------------- |
|
||||
| `itemAnnotations.type` | `type` | 1→highlight, 2→note, 3→image... |
|
||||
| `items.key` | `zotero_key` | 8 字符 base32 key |
|
||||
| `itemAnnotations.text` | `selected_text` | 仅 highlight/underline 有值 |
|
||||
| `itemAnnotations.comment` | `comment` | 所有类型都可能含 comment |
|
||||
| `itemAnnotations.color` | `color` | Hex 色号，如 #ffd400 |
|
||||
| `itemAnnotations.pageLabel`| `page_label` | 标页码字符串 |
|
||||
| `itemAnnotations.sortIndex`| `sort_index` | 格式化零填充排序索引 |
|
||||
| `itemAnnotations.position` | `position_json` | 原始 JSON 字符串，含 rects/paths |
|
||||
| `itemTags → tags.name` | `tags_json` | JSON 数组 |
|
||||
| `items.dateModified` | `source_modified_at` | ISO 8601 |
|
||||
|
||||
### 5. PaperForge 是否应该修改现有 paperforge.db schema？
|
||||
|
||||
**不应该。** Annotation 是用户数据，现有 `drop_all_tables()` 会在 memory rebuild 时无差别删除。
|
||||
|
||||
**决策：独立 `annotations.db`**，与 `paperforge.db` 并列存放。两者通过 `zotero_key` 关联。rebuild memory layer 时 annotations.db 完全不受影响。
|
||||
|
||||
### 6. Annotation 表应该如何避免被 memory rebuild 删除？
|
||||
|
||||
采用独立 DB + 独立 schema version 管理：
|
||||
|
||||
```
|
||||
paperforge/indexes/
|
||||
├── paperforge.db ← memory layer (schema v2, 可 rebuild)
|
||||
├── annotations.db ← annotation layer (schema v1, 独立管理)
|
||||
├── formal-library.json ← canonical index
|
||||
├── memory-runtime-state.json
|
||||
└── vector-runtime-state.json
|
||||
```
|
||||
|
||||
`annotations.db` 有自己的 `meta` 表记录 `schema_version`，由 annotation CLI 命令独立管理 migration。
|
||||
|
||||
### 7. Obsidian PDF overlay 第一版是否可行？
|
||||
|
||||
**可行。** PDF++（2095 stars）已验证了 monkey-patching 路线的可行性。核心机制：
|
||||
|
||||
1. 用 `monkey-around` 库的 `around()` 函数 patch `PDFViewerChild.prototype`
|
||||
2. 在每个 `div.page` 内创建 overlay layer div
|
||||
3. 用 `window.pdfjsLib.setLayerDimensions()` 对齐坐标空间
|
||||
4. 用百分比 CSS 定位实现缩放无关渲染
|
||||
5. 监听 `textlayerrendered` / `annotationlayerrendered` 事件触发重绘
|
||||
|
||||
### 8. 是否需要自建 PDF.js reader？
|
||||
|
||||
**不需要** 且 **不推荐**。Obsidian 内置了 PDF.js（通过 `loadPdfJs()` 可从 plugin 访问），ZotFlow 的自建 reader iframe 是其最重的部分。PaperForge 应该直接 hook 原生 PDF viewer，仅在需要 overlay 的页面上添加 DOM 层。
|
||||
|
||||
### 9. 第一版是否应该写回 Zotero？
|
||||
|
||||
**架构层面设计写回，但 v1 不实现。** 推荐 hybrid 路线：
|
||||
|
||||
- **读路径**：本地 SQLite 只读解析（快、离线、零配置）
|
||||
- **写路径**：通过 Zotero Web API 写回（安全、version 乐观锁、官方支持）
|
||||
- **v1**：本地 SQLite 读取 + 本地 annotations.db 编辑，`sync_state` = `local` / `pending_push`
|
||||
- **v2**：实现 Web API push，`sync_state` = `zotero_synced`
|
||||
|
||||
schema 从第一天就包含 `sync_state`、`source`、`source_version` 等字段，为写回预留。
|
||||
|
||||
### 10. 最推荐的 MVP 开发路线是什么？
|
||||
|
||||
```
|
||||
Phase 1 (1 周): DB + CLI
|
||||
annotations.db schema
|
||||
paperforge annotation import (Zotero SQLite read-only)
|
||||
paperforge annotation list/patch/delete
|
||||
独立 schema version 管理，sync_state 预留
|
||||
|
||||
Phase 2 (1-2 周): Plugin Overlay
|
||||
monkey-patch PDF viewer
|
||||
render highlight/underline/note rects
|
||||
selection → create annotation
|
||||
click → edit popover
|
||||
Zotero annotations show as read-only (lock icon)
|
||||
PaperForge local annotations are editable
|
||||
|
||||
Phase 3 (后续): Web API Write-back
|
||||
Web API 客户端 (zotero-api-client 封装)
|
||||
sync queue → pending_push → API push
|
||||
冲突检测与手动解决
|
||||
```
|
||||
113
reports/01-zotflow-architecture.md
Normal file
113
reports/01-zotflow-architecture.md
Normal file
|
|
@ -0,0 +1,113 @@
|
|||
# ZotFlow Architecture Analysis
|
||||
|
||||
> Status: COMPLETE | Repo: duanxianpi/obsidian-zotflow v1.0.9 | License: AGPL-3.0
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
ZotFlow 采用 **Main Thread + Web Worker + Reader Iframe** 三层架构：
|
||||
|
||||
```
|
||||
Obsidian Main Thread
|
||||
├── Plugin (main.ts)
|
||||
│ ├── WorkerBridge (Comlink) ←→ Web Worker
|
||||
│ ├── ZoteroReaderView (Penpal) ←→ Reader Iframe (zotero/reader)
|
||||
│ ├── LocalReaderView (vault files)
|
||||
│ ├── NoteEditorView
|
||||
│ └── TreeView (React + react-arborist)
|
||||
│
|
||||
├── Services (main-thread)
|
||||
│ ├── ServiceLocator
|
||||
│ ├── IndexService (zotero_key → file path from frontmatter)
|
||||
│ ├── ViewStateService
|
||||
│ ├── CitationService
|
||||
│ └── EventBus / TaskMonitor / LogService / NotificationService
|
||||
│
|
||||
└── Ui Components
|
||||
├── reader/view.ts / bridge.ts / local-view.ts
|
||||
├── tree-view/ (React)
|
||||
├── activity-center/ (React modal)
|
||||
├── editor/ (CodeMirror 6 extensions)
|
||||
└── modals/
|
||||
```
|
||||
|
||||
```
|
||||
Web Worker (worker.ts)
|
||||
├── ZoteroService (zotero-api-client wrapper)
|
||||
├── SyncService (42,918 bytes — bidirectional sync engine)
|
||||
├── AnnotationService (CRUD + diff)
|
||||
├── LibraryService (per-library permissions)
|
||||
├── AttachmentService (download + cache from API/WebDAV)
|
||||
├── TemplateService (LiquidJS rendering)
|
||||
├── NoteService (source note generation)
|
||||
├── ConflictService (field-level diff + resolve)
|
||||
├── IndexedDB (Dexie) — local cache for all Zotero data
|
||||
└── PDFProcessWorker (nested PDF.js worker for export/import)
|
||||
```
|
||||
|
||||
## Communication Protocols
|
||||
|
||||
| Boundary | Protocol | Mechanism |
|
||||
|----------|----------|-----------|
|
||||
| Main ↔ Worker | Comlink | Proxy-based RPC, `Comlink.wrap<WorkerAPI>()` |
|
||||
| Main ↔ Reader Iframe | Penpal | WindowMessenger, structured clone |
|
||||
| Worker ↔ PDF.js Worker | postMessage | Promise-based request/response |
|
||||
|
||||
## Annotation Data Flow
|
||||
|
||||
### Read Path (Zotero → Plugin → Iframe)
|
||||
|
||||
```
|
||||
Zotero API → Worker SyncService → IndexedDB (items table)
|
||||
→ Plugin.annotationService.getAnnotations()
|
||||
→ getAnnotationJson() (db/annotation.ts)
|
||||
→ AnnotationJSON[] → ZoteroReaderView
|
||||
→ IframeBridge.initReader({ annotations })
|
||||
→ AnnotationManager.setAnnotations() → render overlays
|
||||
```
|
||||
|
||||
### Write Path (Iframe → Plugin → Zotero)
|
||||
|
||||
```
|
||||
User creates annotation in reader iframe
|
||||
→ annotationsSaved event (Penpal)
|
||||
→ ZoteroReaderView.handleAnnotationsSaved()
|
||||
→ WorkerBridge.annotation.saveAnnotations()
|
||||
→ AnnotationService.saveAnnotations()
|
||||
→ diff with existing (annotationDataDiff)
|
||||
→ upsert IndexedDB items (syncStatus = "created"/"updated")
|
||||
→ trigger source note re-render (debounced)
|
||||
→ Next push cycle: SyncService.pushDirtyItems()
|
||||
→ Zotero API (version-based optimistic locking)
|
||||
```
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
### Why Web API instead of SQLite?
|
||||
- Full bidirectional sync requires version management
|
||||
- SQLite write would bypass Zotero's validation + sync state
|
||||
- Web API supports group libraries, conflict detection
|
||||
|
||||
### Local Cache
|
||||
- Dexie/IndexedDB with schema migration (v1→v2→v3)
|
||||
- `syncStatus` tracking per item: synced / created / updated / deleted / ignore / conflict
|
||||
- Raw API response stored in `raw` field for field-level diff
|
||||
|
||||
### Reader Embedding
|
||||
- Git submodule to `zotero/reader`
|
||||
- Built via webpack, inlined as blob URLs
|
||||
- Modified Zotero reader engine with Obsidian theme integration
|
||||
- Iframe sandbox restricted to `allow-scripts allow-same-origin allow-forms`
|
||||
|
||||
## What PaperForge Should Borrow
|
||||
|
||||
1. **AnnotationJSON schema** — well-designed, Zotero-compatible data model
|
||||
2. **Conflict resolution model** — field-level diff with keep-local / accept-remote
|
||||
3. **LiquidJS template pipeline** — if PaperForge wants template-driven notes
|
||||
4. **SyncStatus pattern** — clean state machine for sync tracking
|
||||
|
||||
## What PaperForge Should NOT Borrow
|
||||
|
||||
1. **Embedded reader iframe** — too heavy, PaperForge should hook native PDF viewer
|
||||
2. **CodeMirror 6 extensions** — Obsidian-specific, PaperForge doesn't do real-time editing
|
||||
3. **React UI components** — PaperForge's plugin is minimal (ribbon icons + status bar)
|
||||
4. **Full bidirectional sync engine** — PaperForge is read-first, write-limited
|
||||
65
reports/01-zotflow-dataflow.mmd
Normal file
65
reports/01-zotflow-dataflow.mmd
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
```mermaid
|
||||
graph TB
|
||||
subgraph "Zotero Cloud"
|
||||
ZAPI[Zotero Web API]
|
||||
ZStorage[Zotero File Storage]
|
||||
end
|
||||
|
||||
subgraph "Web Worker"
|
||||
direction TB
|
||||
Sync[SyncService]
|
||||
Annotation[AnnotationService]
|
||||
IDB[(IndexedDB\nDexie)]
|
||||
Tpl[TemplateService\nLiquidJS]
|
||||
Attach[AttachmentService]
|
||||
PDFW[PDF Process Worker]
|
||||
Conflict[ConflictService]
|
||||
end
|
||||
|
||||
subgraph "Main Thread (Obsidian Plugin)"
|
||||
direction TB
|
||||
WB[WorkerBridge\nComlink]
|
||||
Plugin[ZotFlow Plugin]
|
||||
RB[ReaderBridge\nPenpal]
|
||||
CM6[CodeMirror 6\nExtensions]
|
||||
Tree[Tree View\nReact]
|
||||
Activity[Activity Center]
|
||||
end
|
||||
|
||||
subgraph "Reader Iframe"
|
||||
direction TB
|
||||
Reader[Zotero Reader Engine]
|
||||
AM[Annotation Manager]
|
||||
PDFJS[PDF.js]
|
||||
end
|
||||
|
||||
subgraph "Obsidian Vault"
|
||||
SN[Source Notes\n.md]
|
||||
ZF[.zf.json Sidecar\nfor local files]
|
||||
end
|
||||
|
||||
ZAPI -->|"REST API\nversion-based"| Sync
|
||||
ZStorage -->|"File download"| Attach
|
||||
Sync --> IDB
|
||||
IDB --> Annotation
|
||||
Annotation -->|"getAnnotations()"| WB
|
||||
Annotation -->|"saveAnnotations()"| Tpl
|
||||
Tpl --> SN
|
||||
Attach -->|"Blob"| WB
|
||||
WB <--> Plugin
|
||||
Plugin --> RB
|
||||
RB -->|"initReader(data)"| Reader
|
||||
Reader --> PDFJS
|
||||
Reader --> AM
|
||||
AM -->|"annotationsSaved"| RB
|
||||
AM -->|"annotationsDeleted"| RB
|
||||
WB <-->|"extractExternalAnnotations"| PDFW
|
||||
Plugin --> CM6
|
||||
Plugin --> Tree
|
||||
Plugin --> Activity
|
||||
Plugin -->|"LocalReader"| ZF
|
||||
|
||||
style ZAPI fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style IDB fill:#bbf,stroke:#333,stroke-width:2px
|
||||
style Reader fill:#bfb,stroke:#333,stroke-width:2px
|
||||
```
|
||||
160
reports/02-zotero-sqlite-schema.md
Normal file
160
reports/02-zotero-sqlite-schema.md
Normal file
|
|
@ -0,0 +1,160 @@
|
|||
# Zotero Annotation SQLite Schema Report
|
||||
|
||||
> Status: COMPLETE | Source: zotero/zotero (main branch)
|
||||
|
||||
## Table: itemAnnotations
|
||||
|
||||
```sql
|
||||
CREATE TABLE itemAnnotations (
|
||||
itemID INTEGER PRIMARY KEY,
|
||||
parentItemID INT NOT NULL,
|
||||
type INTEGER NOT NULL,
|
||||
authorName TEXT,
|
||||
text TEXT,
|
||||
comment TEXT,
|
||||
color TEXT,
|
||||
pageLabel TEXT,
|
||||
sortIndex TEXT NOT NULL,
|
||||
position TEXT NOT NULL,
|
||||
isExternal INT NOT NULL,
|
||||
FOREIGN KEY (itemID) REFERENCES items(itemID) ON DELETE CASCADE,
|
||||
FOREIGN KEY (parentItemID) REFERENCES itemAttachments(itemID)
|
||||
);
|
||||
CREATE INDEX itemAnnotations_parentItemID ON itemAnnotations(parentItemID);
|
||||
```
|
||||
|
||||
## Annotation Type Mapping
|
||||
|
||||
| Integer | String | Description |
|
||||
|---------|-------------|----------------------|
|
||||
| 1 | `highlight` | Text highlight |
|
||||
| 2 | `note` | Sticky note |
|
||||
| 3 | `image` | Image region capture |
|
||||
| 4 | `ink` | Freehand drawing |
|
||||
| 5 | `underline` | Text underline |
|
||||
| 6 | `text` | Text box (resizable) |
|
||||
|
||||
Source: `chrome/content/zotero/xpcom/annotations.js`
|
||||
```javascript
|
||||
Zotero.Annotations.ANNOTATION_TYPE_HIGHLIGHT = 1;
|
||||
Zotero.Annotations.ANNOTATION_TYPE_NOTE = 2;
|
||||
Zotero.Annotations.ANNOTATION_TYPE_IMAGE = 3;
|
||||
Zotero.Annotations.ANNOTATION_TYPE_INK = 4;
|
||||
Zotero.Annotations.ANNOTATION_TYPE_UNDERLINE = 5;
|
||||
Zotero.Annotations.ANNOTATION_TYPE_TEXT = 6;
|
||||
```
|
||||
|
||||
## Position JSON Structures
|
||||
|
||||
### Highlight / Underline / Note / Text (type 1, 2, 5, 6)
|
||||
```json
|
||||
{
|
||||
"pageIndex": 1,
|
||||
"rects": [
|
||||
[231.284, 402.126, 293.107, 410.142],
|
||||
[54.222, 392.164, 293.107, 400.18]
|
||||
]
|
||||
}
|
||||
```
|
||||
Each rect: `[left, top, right, bottom]` in PDF points (1/72 inch), origin bottom-left.
|
||||
|
||||
### Image (type 3)
|
||||
```json
|
||||
{
|
||||
"pageIndex": 123,
|
||||
"rects": [[314.4, 412.8, 556.2, 609.6]],
|
||||
"width": 400,
|
||||
"height": 200
|
||||
}
|
||||
```
|
||||
|
||||
### Ink (type 4)
|
||||
```json
|
||||
{
|
||||
"pageIndex": 1,
|
||||
"width": 2,
|
||||
"paths": [
|
||||
[x0, y0, x1, y1, x2, y2, ...],
|
||||
[x3, y3, x4, y4, ...]
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Key Fields
|
||||
|
||||
| Column | Type | Notes |
|
||||
|-------------------|----------|-----------------------------------------------|
|
||||
| `text` | TEXT | Extracted text for highlight/underline only |
|
||||
| `comment` | TEXT | User's annotation note (may contain HTML) |
|
||||
| `color` | TEXT | Hex color, e.g. `#ffd400` (default yellow) |
|
||||
| `pageLabel` | TEXT | Page label string, e.g. "15", "XVI" |
|
||||
| `sortIndex` | TEXT | Zero-padded: `"<page:05d>|<rect:06d>|<char:05d>"` |
|
||||
| `position` | TEXT | JSON string (can be up to 65KB before split) |
|
||||
| `isExternal` | INT | 1 = embedded PDF annotation (read-only) |
|
||||
|
||||
## Relationship Traversal
|
||||
|
||||
```sql
|
||||
-- Get all annotations for a paper (top-level item)
|
||||
SELECT
|
||||
ia.itemID,
|
||||
ia.type,
|
||||
ia.text,
|
||||
ia.comment,
|
||||
ia.color,
|
||||
ia.pageLabel,
|
||||
ia.sortIndex,
|
||||
ia.position,
|
||||
ia.isExternal,
|
||||
i.key AS annotation_key,
|
||||
i.dateModified,
|
||||
i.libraryID,
|
||||
att.path AS attachment_path,
|
||||
att.key AS attachment_key,
|
||||
att.linkMode AS attachment_link_mode
|
||||
FROM items paper
|
||||
JOIN items att ON att.parentItemID = paper.itemID
|
||||
AND att.itemTypeID = (SELECT itemTypeID FROM itemTypes WHERE typeName = 'attachment')
|
||||
JOIN itemAnnotations ia ON ia.parentItemID = att.itemID
|
||||
JOIN items i ON i.itemID = ia.itemID
|
||||
WHERE paper.key = ?;
|
||||
|
||||
-- Get tags for annotations
|
||||
SELECT
|
||||
i.key AS annotation_key,
|
||||
t.name AS tag_name
|
||||
FROM items i
|
||||
JOIN itemTags it ON it.itemID = i.itemID
|
||||
JOIN tags t ON t.tagID = it.tagID
|
||||
WHERE i.key IN (?);
|
||||
```
|
||||
|
||||
## Annotation Color Presets
|
||||
|
||||
Source: `zotero/reader/src/common/defines.js`
|
||||
```javascript
|
||||
const ANNOTATION_COLORS = [
|
||||
['yellow', '#ffd400'],
|
||||
['red', '#ff6666'],
|
||||
['green', '#5fb236'],
|
||||
['blue', '#2ea8e5'],
|
||||
['purple', '#a28ae5'],
|
||||
['magenta', '#e56eee'],
|
||||
['orange', '#f19837'],
|
||||
['gray', '#aaaaaa'],
|
||||
];
|
||||
```
|
||||
|
||||
## Position Size Limit
|
||||
|
||||
`ANNOTATION_POSITION_MAX_SIZE = 65000` bytes. Annotations whose serialized `position` JSON exceeds this are split into multiple annotation items (by rect or path segment).
|
||||
|
||||
## Important: Read-Only Access Only
|
||||
|
||||
Zotero documentation explicitly warns:
|
||||
> "access to the SQLite database should be done only in a read-only manner. Modifying the database while Zotero is running can easily result in a corrupted database."
|
||||
|
||||
PaperForge must:
|
||||
1. COPY `zotero.sqlite` to a temp path before reading (if Zotero is running)
|
||||
2. Never write to any Zotero SQLite table
|
||||
3. Never assume schema stability across Zotero versions
|
||||
190
reports/03-paperforge-schema-design.md
Normal file
190
reports/03-paperforge-schema-design.md
Normal file
|
|
@ -0,0 +1,190 @@
|
|||
# PaperForge Annotation Schema Integration Design
|
||||
|
||||
> Status: DESIGN COMPLETE | Schema Version: v1 (independent)
|
||||
|
||||
## Decision: Independent annotations.db
|
||||
|
||||
Annotation data will live in a **separate SQLite database** (`annotations.db`) co-located with `paperforge.db`:
|
||||
|
||||
```
|
||||
<vault>/System/PaperForge/indexes/
|
||||
├── paperforge.db ← Memory Layer (rebuildable)
|
||||
├── annotations.db ← Annotation Layer (never dropped)
|
||||
├── formal-library.json
|
||||
└── runtime-state*.json
|
||||
```
|
||||
|
||||
### Rationale
|
||||
|
||||
| Concern | Solution |
|
||||
|---------|----------|
|
||||
| `drop_all_tables()` would destroy user data | Separate DB is immune to rebuild |
|
||||
| Schema version coupling | Independent schema version management |
|
||||
| Rebuild frequency | Memory rebuild is frequent, annotations should persist |
|
||||
| Backup granularity | Users may want to backup annotations independently |
|
||||
| Concurrent access | Annotations can be written while memory is being rebuilt |
|
||||
|
||||
## Schema: annotations.db
|
||||
|
||||
```sql
|
||||
-- Schema version management
|
||||
CREATE TABLE IF NOT EXISTS meta (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
);
|
||||
|
||||
-- Main annotation table
|
||||
CREATE TABLE IF NOT EXISTS annotations (
|
||||
id TEXT PRIMARY KEY, -- UUID for local, zotero_key for imported
|
||||
|
||||
-- Paper association (via zotero_key)
|
||||
paper_id TEXT NOT NULL,
|
||||
|
||||
-- Zotero source tracking
|
||||
zotero_library_id INTEGER,
|
||||
zotero_item_id INTEGER,
|
||||
zotero_key TEXT DEFAULT '',
|
||||
zotero_attachment_key TEXT DEFAULT '',
|
||||
|
||||
-- PDF tracking
|
||||
pdf_path TEXT DEFAULT '',
|
||||
pdf_hash TEXT DEFAULT '',
|
||||
|
||||
-- Annotation data
|
||||
type TEXT NOT NULL, -- highlight|underline|note|image|ink|text
|
||||
page_index INTEGER, -- 0-based
|
||||
page_label TEXT DEFAULT '',
|
||||
|
||||
selected_text TEXT DEFAULT '',
|
||||
comment TEXT DEFAULT '',
|
||||
color TEXT DEFAULT '', -- hex: #ffd400
|
||||
sort_index TEXT DEFAULT '',
|
||||
|
||||
-- Position data
|
||||
position_json TEXT DEFAULT '{}',
|
||||
selector_json TEXT DEFAULT '{}', -- for EPUB/web annotations (future)
|
||||
|
||||
-- Tags
|
||||
tags_json TEXT DEFAULT '[]',
|
||||
|
||||
-- Sync state (from Zotero or local)
|
||||
source TEXT NOT NULL DEFAULT 'paperforge', -- paperforge|zotero_db|pdf_embedded|imported_json
|
||||
source_key TEXT DEFAULT '', -- original key in source system
|
||||
source_version INTEGER, -- Zotero version number
|
||||
source_modified_at TEXT DEFAULT '', -- ISO 8601 from Zotero
|
||||
|
||||
sync_state TEXT NOT NULL DEFAULT 'local', -- local|zotero_synced|zotero_remote_changed|local_modified|conflict|pending_push
|
||||
is_readonly INTEGER NOT NULL DEFAULT 0, -- 1 = Zotero-sourced, not editable
|
||||
|
||||
-- Metadata
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
deleted_at TEXT, -- soft delete
|
||||
|
||||
FOREIGN KEY (paper_id) REFERENCES papers(zotero_key) -- conceptual, not enforced cross-DB
|
||||
);
|
||||
|
||||
-- Indexes
|
||||
CREATE INDEX IF NOT EXISTS idx_annotations_paper ON annotations(paper_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_annotations_type ON annotations(type);
|
||||
CREATE INDEX IF NOT EXISTS idx_annotations_sync_state ON annotations(sync_state);
|
||||
CREATE INDEX IF NOT EXISTS idx_annotations_source ON annotations(source);
|
||||
CREATE INDEX IF NOT EXISTS idx_annotations_page ON annotations(paper_id, page_index);
|
||||
CREATE INDEX IF NOT EXISTS idx_annotations_zotero_key ON annotations(zotero_key);
|
||||
CREATE INDEX IF NOT EXISTS idx_annotations_deleted ON annotations(deleted_at) WHERE deleted_at IS NOT NULL;
|
||||
|
||||
-- FTS5 full-text search on annotation text and comments
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS annotations_fts USING fts5(
|
||||
paper_id,
|
||||
selected_text,
|
||||
comment,
|
||||
tags_json,
|
||||
content='annotations',
|
||||
content_rowid='rowid'
|
||||
);
|
||||
|
||||
-- Triggers to keep FTS in sync
|
||||
CREATE TRIGGER IF NOT EXISTS annotations_ai AFTER INSERT ON annotations BEGIN
|
||||
INSERT INTO annotations_fts(rowid, paper_id, selected_text, comment, tags_json)
|
||||
VALUES (new.rowid, new.paper_id, new.selected_text, new.comment, new.tags_json);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS annotations_ad AFTER DELETE ON annotations BEGIN
|
||||
INSERT INTO annotations_fts(annotations_fts, rowid, paper_id, selected_text, comment, tags_json)
|
||||
VALUES ('delete', old.rowid, old.paper_id, old.selected_text, old.comment, old.tags_json);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS annotations_au AFTER UPDATE ON annotations BEGIN
|
||||
INSERT INTO annotations_fts(annotations_fts, rowid, paper_id, selected_text, comment, tags_json)
|
||||
VALUES ('delete', old.rowid, old.paper_id, old.selected_text, old.comment, old.tags_json);
|
||||
INSERT INTO annotations_fts(rowid, paper_id, selected_text, comment, tags_json)
|
||||
VALUES (new.rowid, new.paper_id, new.selected_text, new.comment, new.tags_json);
|
||||
END;
|
||||
|
||||
-- Sync queue for pending write-back operations
|
||||
CREATE TABLE IF NOT EXISTS sync_queue (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
annotation_id TEXT NOT NULL,
|
||||
operation TEXT NOT NULL, -- create|update|delete
|
||||
payload_json TEXT NOT NULL,
|
||||
retry_count INTEGER DEFAULT 0,
|
||||
last_error TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
FOREIGN KEY (annotation_id) REFERENCES annotations(id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_sync_queue_pending ON sync_queue(annotation_id, operation)
|
||||
WHERE retry_count < 3;
|
||||
```
|
||||
|
||||
## Sync State Machine
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ local │ ← PaperForge-native annotation (no Zotero source)
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌──────▼──────┐
|
||||
│ pending_push│ ← local edit pending Zotero API push
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌──────▼──────┐
|
||||
│zotero_synced│ ← successfully pushed to Zotero
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌────────────┼────────────┐
|
||||
│ │ │
|
||||
┌────────▼───┐ ┌──────▼──────┐ ┌──▼─────────┐
|
||||
│local_modif.│ │remote_change│ │ conflict │
|
||||
│ (push) │ │ (re-pull) │ │ (manual fix)│
|
||||
└────────────┘ └─────────────┘ └─────────────┘
|
||||
```
|
||||
|
||||
## How to Avoid Rebuild Conflict
|
||||
|
||||
```python
|
||||
# In memory/builder.py build_from_index():
|
||||
# DO NOT drop annotations.db tables
|
||||
# Only operate on paperforge.db tables
|
||||
|
||||
ANNOTATIONS_DB = "annotations.db" # managed separately
|
||||
|
||||
def build_from_index(vault: Path) -> dict:
|
||||
# ... existing code touches paperforge.db only ...
|
||||
# annotations.db is never dropped
|
||||
```
|
||||
|
||||
## Cross-DB Join Pattern
|
||||
|
||||
Since annotations and papers are in separate databases, queries that need both do a two-step lookup:
|
||||
|
||||
```python
|
||||
# Step 1: Get paper
|
||||
paper = lookup_paper(conn_paperforge, query)
|
||||
|
||||
# Step 2: Get annotations for that paper
|
||||
annotations = conn_annotations.execute(
|
||||
"SELECT * FROM annotations WHERE paper_id = ? AND deleted_at IS NULL ORDER BY sort_index",
|
||||
(paper["zotero_key"],)
|
||||
).fetchall()
|
||||
```
|
||||
213
reports/04-obsidian-pdf-overlay-feasibility.md
Normal file
213
reports/04-obsidian-pdf-overlay-feasibility.md
Normal file
|
|
@ -0,0 +1,213 @@
|
|||
# Obsidian PDF Overlay Feasibility Analysis
|
||||
|
||||
> Status: FEASIBLE | Reference: PDF++ (RyotaUshio/obsidian-pdf-plus)
|
||||
|
||||
## Verdict
|
||||
|
||||
**Technically feasible** via monkey-patching Obsidian's private PDF viewer internals. PDF++ (2095 stars, 254 releases) has proven this approach works. However, there is **no public Obsidian API** for PDF viewing — every hook is reverse-engineered.
|
||||
|
||||
## The Alternative: Why Not Build A Custom PDF View?
|
||||
|
||||
| Approach | Pros | Cons |
|
||||
|----------|------|------|
|
||||
| Monkey-patch native viewer | Seamless UX, reuse native features | Fragile, requires version-gating |
|
||||
| Custom view type + loadPdfJs() | Stable API, full control | User loses native PDF features, UX fragmentation |
|
||||
|
||||
**User chose: monkey-patch native viewer.**
|
||||
|
||||
## How PDF++ Patches the PDF Viewer
|
||||
|
||||
### Core Patcher: `monkey-around`
|
||||
|
||||
```typescript
|
||||
import { around } from 'monkey-around';
|
||||
|
||||
export const patchPDFInternals = (plugin: Plugin): boolean => {
|
||||
plugin.register(around(PDFViewerChild.prototype, {
|
||||
load(old) {
|
||||
return function () {
|
||||
// Plugin initialization here
|
||||
old.call(this);
|
||||
};
|
||||
},
|
||||
loadFile(old) {
|
||||
return function (file: TFile) {
|
||||
old.call(this, file);
|
||||
// Create visualizer, register event listeners
|
||||
};
|
||||
}
|
||||
}));
|
||||
};
|
||||
```
|
||||
|
||||
### Patch Chain
|
||||
|
||||
```
|
||||
patchWorkspace() → WorkspaceLeaf openLinkText
|
||||
patchPDFView() → PDFView.prototype (getState, setState, onLoadFile)
|
||||
patchPDFInternals() → PDFViewerComponent, PDFViewerChild, ObsidianViewer
|
||||
patchPDFInternalFromPDFEmbed() → PDF embeds
|
||||
patchBacklink() → backlink pane
|
||||
```
|
||||
|
||||
## DOM Structure of Obsidian's PDF Viewer
|
||||
|
||||
```
|
||||
div.pdf-viewer-container
|
||||
div.pdf-viewer
|
||||
div.page[data-page-number="1"]
|
||||
canvas.canvasWrapper
|
||||
canvas ← rendered PDF canvas
|
||||
div.textLayer
|
||||
span.textLayerNode[data-idx]
|
||||
div.annotationLayer
|
||||
section[data-annotation-id]
|
||||
div.custom-overlay-layer ← PaperForge injects here
|
||||
div.annotation-highlight
|
||||
div.page[data-page-number="2"]...
|
||||
```
|
||||
|
||||
## Key Obsidian Internal Classes
|
||||
|
||||
| Class | How to Access |
|
||||
|-------|---------------|
|
||||
| `PDFView` | `leaf.view` on 'pdf' type leaf |
|
||||
| `PDFViewerComponent` | `pdfView.viewer` |
|
||||
| `PDFViewerChild` | `viewerComponent.child` |
|
||||
| `ObsidianViewer` | `child.pdfViewer` |
|
||||
| `PDFPageView` | `pdfViewer.getPageView(n)` |
|
||||
| `EventBus` | `child.pdfViewer.eventBus` |
|
||||
|
||||
All obtained via:
|
||||
```typescript
|
||||
const pdfLeaves = app.workspace.getLeavesOfType('pdf');
|
||||
const pdfView = pdfLeaves[0].view as any;
|
||||
```
|
||||
|
||||
## Events to Listen To
|
||||
|
||||
| Event | When to Rerender |
|
||||
|-------|------------------|
|
||||
| `textlayerrendered` | Page text layer ready, safe to render text-based annotations |
|
||||
| `annotationlayerrendered` | Page annotation layer ready |
|
||||
| `pagerendered` | Page canvas rendered |
|
||||
| `pagechanging` | Current page changed |
|
||||
| `scalechanged` | Zoom level changed |
|
||||
| `pagesloaded` | All pages loaded |
|
||||
|
||||
```typescript
|
||||
child.pdfViewer.eventBus.on('textlayerrendered', (data) => {
|
||||
const pageView = data.source; // PDFPageView
|
||||
renderAnnotationsForPage(pageView, annotationsForPage(pageView.id));
|
||||
});
|
||||
```
|
||||
|
||||
## Coordinate Transformation
|
||||
|
||||
### PDF → Overlay Position
|
||||
```typescript
|
||||
function placeAnnotationRect(rect: number[], pageView: PDFPageView) {
|
||||
const viewport = pageView.viewport;
|
||||
const [left, bottom, right, top] = rect;
|
||||
|
||||
// Mirror Y axis (PDF origin is bottom-left, screen is top-left)
|
||||
const mirrored = window.pdfjsLib.Util.normalizeRect([
|
||||
left,
|
||||
viewport.viewBox[3] - bottom + viewport.viewBox[1],
|
||||
right,
|
||||
viewport.viewBox[3] - top + viewport.viewBox[1]
|
||||
]);
|
||||
|
||||
// Create overlay layer per page
|
||||
const pageDiv = pageView.div;
|
||||
let layer = pageDiv.querySelector('div.pf-annotation-overlay');
|
||||
if (!layer) {
|
||||
layer = pageDiv.createDiv('pf-annotation-overlay');
|
||||
window.pdfjsLib.setLayerDimensions(layer, pageView.viewport);
|
||||
}
|
||||
|
||||
// Place rect with percentage positioning (zoom-independent)
|
||||
const pageW = viewport.viewBox[2] - viewport.viewBox[0];
|
||||
const pageH = viewport.viewBox[3] - viewport.viewBox[1];
|
||||
const el = createDiv('pf-annotation-rect');
|
||||
el.setCssStyles({
|
||||
left: `${100 * (mirrored[0] - viewport.viewBox[0]) / pageW}%`,
|
||||
top: `${100 * (mirrored[1] - viewport.viewBox[1]) / pageH}%`,
|
||||
width: `${100 * (mirrored[2] - mirrored[0]) / pageW}%`,
|
||||
height: `${100 * (mirrored[3] - mirrored[1]) / pageH}%`,
|
||||
background: `rgba(255, 212, 0, 0.25)`,
|
||||
});
|
||||
layer.appendChild(el);
|
||||
}
|
||||
```
|
||||
|
||||
## Selection → Annotation
|
||||
|
||||
```typescript
|
||||
function getSelectionRect(pageView: PDFPageView): Rect[] | null {
|
||||
const selection = window.getSelection();
|
||||
if (!selection || selection.isCollapsed) return null;
|
||||
|
||||
const textLayer = pageView.textLayer;
|
||||
if (!textLayer) return null;
|
||||
|
||||
// Iterate through text layer nodes to find selected range
|
||||
// Extract PDF coordinates from text content items
|
||||
// Returns array of [left, top, right, bottom] rects
|
||||
}
|
||||
```
|
||||
|
||||
## Dark Mode Handling
|
||||
|
||||
```css
|
||||
.theme-light .pf-annotation-highlight {
|
||||
background: rgba(255, 212, 0, 0.25);
|
||||
}
|
||||
.theme-dark .pf-annotation-highlight {
|
||||
background: rgba(255, 200, 0, 0.2);
|
||||
}
|
||||
|
||||
.theme-light .pf-annotation-underline {
|
||||
border-bottom: 2px solid #ff6666;
|
||||
}
|
||||
.theme-dark .pf-annotation-underline {
|
||||
border-bottom: 2px solid #ff8888;
|
||||
}
|
||||
```
|
||||
|
||||
Listen for theme changes:
|
||||
```typescript
|
||||
app.workspace.on('css-change', () => {
|
||||
const isDark = document.body.classList.contains('theme-dark');
|
||||
// Re-render with appropriate colors
|
||||
});
|
||||
```
|
||||
|
||||
## Getting Current PDF Path
|
||||
|
||||
```typescript
|
||||
const activeLeaf = app.workspace.activeLeaf;
|
||||
if (activeLeaf?.view?.file?.extension === 'pdf') {
|
||||
const pdfFile: TFile = activeLeaf.view.file;
|
||||
const path = pdfFile.path; // e.g. "Resources/Literature/domain/paper.pdf"
|
||||
}
|
||||
```
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Obsidian version breakage | `requireApiVersion()` checks; version-gate patches |
|
||||
| PDF.js version changes | PDF++ handles v1.7.7 vs v1.8.0 diffs (ObsidianViewer class vs factory) |
|
||||
| Conflicts with other plugins | Document known incompatibilities; avoid patching same methods |
|
||||
| Performance with 500+ annotations | Cache rects per annotation ID; debounce rerender; requestAnimationFrame |
|
||||
| Mobile support | Check `Platform.isDesktopApp`; degrade gracefully |
|
||||
|
||||
## Recommended Libraries
|
||||
|
||||
| Library | Purpose |
|
||||
|---------|---------|
|
||||
| `monkey-around` | Prototype patching (PDF++ proven) |
|
||||
| `obsidian` (built-in) | Plugin API, TFile, Workspace, loadPdfJs() |
|
||||
|
||||
No additional PDF.js bundling needed — Obsidian provides it natively.
|
||||
249
reports/05-mvp-implementation-plan.md
Normal file
249
reports/05-mvp-implementation-plan.md
Normal file
|
|
@ -0,0 +1,249 @@
|
|||
# MVP Implementation Plan
|
||||
|
||||
> 2-3 weeks total | Phase 1: DB+CLI (1 week) | Phase 2: Plugin Overlay (1-2 weeks)
|
||||
|
||||
## File Structure
|
||||
|
||||
### New Python Modules
|
||||
|
||||
```
|
||||
paperforge/
|
||||
├── annotation/ ← NEW: annotation package
|
||||
│ ├── __init__.py
|
||||
│ ├── probe.py ← Zotero SQLite read-only parser
|
||||
│ ├── db.py ← annotations.db connection + schema
|
||||
│ ├── schema.py ← annotations.db DDL + migration
|
||||
│ ├── import_annotations.py ← import pipeline (Zotero → annotations.db)
|
||||
│ └── export.py ← JSON / annotated PDF export
|
||||
├── commands/
|
||||
│ ├── annotation.py ← `paperforge annotation` CLI
|
||||
│ └── ...
|
||||
├── memory/
|
||||
│ └── ... ← unchanged (paperforge.db untouched)
|
||||
└── services/
|
||||
└── sync_service.py ← unchanged
|
||||
```
|
||||
|
||||
### New TS Modules (Obsidian Plugin)
|
||||
|
||||
```
|
||||
paperforge/plugin/src/
|
||||
├── main.ts ← patched: add ribbon/cmd for annotation overlay
|
||||
├── pdf-overlay/
|
||||
│ ├── patch-pdf-viewer.ts ← monkey-around patches
|
||||
│ ├── overlay-layer.ts ← overlay DOM management per page
|
||||
│ ├── rect-renderer.ts ← rect placement + color theming
|
||||
│ ├── selection-handler.ts ← text selection → annotation
|
||||
│ ├── popover.ts ← click annotation → edit comment/color
|
||||
│ ├── annotation-fetcher.ts ← execFile → paperforge annotation list
|
||||
│ └── types.ts ← shared interfaces
|
||||
└── testable.js ← extend
|
||||
```
|
||||
|
||||
## New CLI Commands
|
||||
|
||||
```bash
|
||||
# Import annotations from Zotero SQLite
|
||||
paperforge annotation import
|
||||
[--zotero-db PATH] # override auto-detected Zotero data dir
|
||||
[--copy-db] # copy zotero.sqlite to temp before reading
|
||||
[--paper KEY] # specific paper only
|
||||
[--dry-run] # preview without writing
|
||||
[--json]
|
||||
|
||||
# List annotations for a paper
|
||||
paperforge annotation list
|
||||
<paper_key> # zotero_key of the paper
|
||||
[--page N] # filter by page
|
||||
[--type TYPE] # filter by type
|
||||
[--json]
|
||||
[--limit N]
|
||||
|
||||
# Create a local annotation
|
||||
paperforge annotation create
|
||||
--paper KEY
|
||||
--type TYPE
|
||||
--page-index N
|
||||
[--page-label TEXT]
|
||||
[--selected-text TEXT]
|
||||
[--comment TEXT]
|
||||
[--color HEX]
|
||||
[--position JSON]
|
||||
[--sort-index TEXT]
|
||||
[--json]
|
||||
|
||||
# Update annotation
|
||||
paperforge annotation patch
|
||||
<annotation_id>
|
||||
[--comment TEXT]
|
||||
[--color HEX]
|
||||
[--json]
|
||||
|
||||
# Soft delete annotation
|
||||
paperforge annotation delete
|
||||
<annotation_id>
|
||||
[--hard] # permanent delete
|
||||
[--json]
|
||||
|
||||
# Export annotations
|
||||
paperforge annotation export
|
||||
<paper_key>
|
||||
[--format json|markdown] # default: json
|
||||
[--output PATH]
|
||||
[--json]
|
||||
|
||||
# Check annotation DB status
|
||||
paperforge annotation status
|
||||
[--json]
|
||||
```
|
||||
|
||||
## New DB Table (annotations.db)
|
||||
|
||||
See `reports/03-paperforge-schema-design.md` for full schema.
|
||||
|
||||
Key decisions:
|
||||
- `sync_state` includes `pending_push` from day 1, even though v1 won't push
|
||||
- `sync_queue` table is pre-defined for future Web API push
|
||||
- FTS5 on `selected_text` + `comment` + `tags_json`
|
||||
- Soft delete via `deleted_at`
|
||||
|
||||
## API Routes (for Plugin execFile interaction)
|
||||
|
||||
```
|
||||
→ paperforge annotation list <key> --json
|
||||
← {
|
||||
"ok": true,
|
||||
"data": {
|
||||
"paper_id": "ABCD1234",
|
||||
"annotations": [
|
||||
{ id, type, page_index, selected_text, comment, color, position_json, ... }
|
||||
],
|
||||
"count": 15
|
||||
}
|
||||
}
|
||||
|
||||
→ paperforge annotation create --paper KEY --type highlight --page-index 3 --json
|
||||
← {
|
||||
"ok": true,
|
||||
"data": { "id": "uuid-here", ... }
|
||||
}
|
||||
```
|
||||
|
||||
## Plugin ↔ Python Interaction
|
||||
|
||||
```typescript
|
||||
// In annotation-fetcher.ts
|
||||
import { execFile } from 'child_process';
|
||||
|
||||
async function fetchAnnotations(paperKey: string): Promise<Annotation[]> {
|
||||
const result = await runSubprocess(
|
||||
pythonExe,
|
||||
['-m', 'paperforge', 'annotation', 'list', paperKey, '--json'],
|
||||
vaultPath,
|
||||
30000
|
||||
);
|
||||
return JSON.parse(result.stdout).data.annotations;
|
||||
}
|
||||
|
||||
async function createAnnotation(paperKey: string, data: CreateAnnotationPayload): Promise<Annotation> {
|
||||
const args = [
|
||||
'-m', 'paperforge', 'annotation', 'create',
|
||||
'--paper', paperKey,
|
||||
'--type', data.type,
|
||||
'--page-index', String(data.pageIndex),
|
||||
'--selected-text', data.selectedText || '',
|
||||
'--comment', data.comment || '',
|
||||
'--color', data.color || '#ffd400',
|
||||
'--position', JSON.stringify(data.position),
|
||||
'--sort-index', data.sortIndex,
|
||||
'--json'
|
||||
];
|
||||
const result = await runSubprocess(pythonExe, args, vaultPath, 10000);
|
||||
return JSON.parse(result.stdout).data;
|
||||
}
|
||||
```
|
||||
|
||||
## UI Interaction Flow
|
||||
|
||||
```
|
||||
1. User opens PDF in Obsidian (native PDF viewer)
|
||||
2. Plugin patches PDFViewerChild.loadFile() → fetches annotations from annotations.db
|
||||
3. Plugin renders overlay rects on each page
|
||||
4. User sees highlights/underlines/notes on PDF
|
||||
|
||||
User creates annotation:
|
||||
1. Select text in PDF (native text selection works)
|
||||
2. Plugin detects selection, shows "Add highlight" floating button
|
||||
3. User clicks → execFile → paperforge annotation create
|
||||
4. Plugin receives new annotation → adds to overlay immediately (optimistic)
|
||||
|
||||
User views annotation:
|
||||
1. Hover over highlight → tooltip shows comment
|
||||
2. Click → popover shows full annotation details
|
||||
3. If is_readonly=1 → show lock icon, no edit buttons
|
||||
4. If is_readonly=0 → show edit/delete buttons
|
||||
|
||||
User edits annotation:
|
||||
1. Click edit in popover → inline textarea
|
||||
2. Save → execFile → paperforge annotation patch
|
||||
3. Plugin updates overlay
|
||||
|
||||
Zotero annotations:
|
||||
- Imported with is_readonly=1, sync_state='zotero_synced'
|
||||
- Displayed with lock icon
|
||||
- First version: user reads them in PDF, cannot edit
|
||||
```
|
||||
|
||||
## Not In MVP
|
||||
|
||||
| Feature | Reason |
|
||||
|---------|--------|
|
||||
| Zotero Web API write-back | Requires API key management, rate limiting, conflict UI |
|
||||
| PDF file export with embedded annotations | Complex PDF manipulation, post-MVP |
|
||||
| Ink annotation editing | Zotero's ink rendering is complex |
|
||||
| EPUB annotation | Different position model (CSS selector) |
|
||||
| Group library support | Zotero SQLite has different path resolution for groups |
|
||||
| Multi-device sync | Out of scope for first version |
|
||||
| Template-driven annotation notes | ZotFlow's feature, good but not required |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Python Tests
|
||||
```bash
|
||||
# Unit tests for annotation DB
|
||||
python -m pytest tests/unit/test_annotation_db.py -v
|
||||
|
||||
# Integration test: probe Zotero SQLite
|
||||
python experiments/zotero_annotation_probe.py --zotero-db fixtures/test_zotero.sqlite --limit 10
|
||||
|
||||
# CLI tests
|
||||
python -m pytest tests/cli/test_annotation_commands.py -v
|
||||
```
|
||||
|
||||
### JS Tests
|
||||
```bash
|
||||
# Unit tests for overlay components (no Obsidian dependency)
|
||||
cd paperforge/plugin && npx vitest run
|
||||
|
||||
# Test: annotation-fetcher mock responses
|
||||
# Test: coordinate transformation
|
||||
# Test: rect placement logic
|
||||
```
|
||||
|
||||
### Fixtures
|
||||
```bash
|
||||
fixtures/
|
||||
├── test_zotero.sqlite ← minimal Zotero SQLite with 1 paper + annotations
|
||||
├── sample_annotations.json ← known-good annotation JSON
|
||||
└── sample_annotation_probe.json ← expected probe output
|
||||
```
|
||||
|
||||
## Risk & Mitigation
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|------------|
|
||||
| Zotero SQLite schema changes | Low | High | Schema version check; fail gracefully with clear error |
|
||||
| Zotero running locks DB | Medium | Medium | Copy to temp before reading; detect lock errors |
|
||||
| Obsidian PDF viewer internal API changes | High | Medium | version-gate patches; plugin auto-disables on major version mismatch |
|
||||
| Monkey-patching conflicts with PDF++/other plugins | Medium | Low | Document conflicts; PaperForge checks for other patchers |
|
||||
| Large annotations (500+) performance | Low | Medium | Rects merged per page; debounced rerender; cache computed positions |
|
||||
65
reports/06-risk-and-license-review.md
Normal file
65
reports/06-risk-and-license-review.md
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
# Risk & License Review
|
||||
|
||||
## License Compatibility
|
||||
|
||||
| Component | License | Usage | Compatible? |
|
||||
|-----------|---------|-------|-------------|
|
||||
| Zotero | AGPL-3.0 | SQLite schema reference, annotations.js implementation patterns | ✅ Read-only reference, no code copied |
|
||||
| ZotFlow | AGPL-3.0 | Architecture inspiration, not code | ✅ Clean-room design |
|
||||
| PDF++ | MIT | Overlay technique reference | ✅ MIT allows study/implementation |
|
||||
| Obsidian | Proprietary | Plugin development | ✅ Plugin API is public |
|
||||
| PaperForge | (Check LICENSE file) | This project | N/A |
|
||||
|
||||
## Technical Risks
|
||||
|
||||
### Risk 1: Zotero SQLite Schema Instability
|
||||
- **Impact**: Schema changes could break annotation import
|
||||
- **Mitigation**: Version check on Zotero schema; probe script validates detected schema; clear error message if schema is unknown
|
||||
- **Detection**: Compare detected table columns against known schema version
|
||||
|
||||
### Risk 2: Obsidian PDF Viewer Internal API Breakage
|
||||
- **Impact**: Monkey-patching breaks after Obsidian update
|
||||
- **Mitigation**:
|
||||
- `requireApiVersion()` checks before applying patches
|
||||
- Graceful degradation (no crash if patch fails)
|
||||
- Each patch independently try-caught
|
||||
- Alert user if annotation overlay cannot initialize
|
||||
- **Historical context**: PDF++ has survived 254 releases across 2 years, proving the approach is viable
|
||||
|
||||
### Risk 3: Concurrent Zotero SQLite Access
|
||||
- **Impact**: Reading zotero.sqlite while Zotero is running may get stale data or SQLITE_BUSY
|
||||
- **Mitigation**:
|
||||
- Default behavior: copy zotero.sqlite to temp directory before reading
|
||||
- SQLITE_BUSY detection with retry
|
||||
- Clear warning in documentation about closing Zotero before import
|
||||
|
||||
### Risk 4: Performance with Large Annotation Sets
|
||||
- **Impact**: 500+ annotations on a single paper could cause slow overlay rendering
|
||||
- **Mitigation**:
|
||||
- Rects merged per page, not per annotation
|
||||
- Only render annotations for visible pages
|
||||
- Debounce rerender during scroll/zoom
|
||||
- Cache computed overlay DOM elements
|
||||
|
||||
### Risk 5: Plugin Conflict with PDF++
|
||||
- **Impact**: Both plugins patch the same PDF viewer methods
|
||||
- **Detection**: Check for `window.PDFPlus` or `document.querySelector('.pdf-plus-*')` at init
|
||||
- **Mitigation**:
|
||||
- Document that PaperForge's annotation overlay is incompatible with PDF++ highlights
|
||||
- Disable overlay if PDF++ detected, fall back to panel-only mode
|
||||
|
||||
## Operational Risks
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| User accidentaly deletes annotations | Data loss | Soft delete by default; export/backup command |
|
||||
| Corrupted annotations.db | Data loss | WAL mode; integrity_check on startup; backup before migration |
|
||||
| Zotero upgrade changes SQLite path | Import fails | ZOTERO_DATA_DIR env var; auto-detect from registry/macOS paths |
|
||||
| Very large position JSON (>65KB) | Truncation risk | Zotero splits annotations >65KB; handle both split and unsplit cases |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- **No credential exposure**: Local SQLite reading requires no API keys
|
||||
- **No network dependency**: Phase 1 is fully offline
|
||||
- **Future Web API keys**: Will use Obsidian SecretStorage (like ZotFlow), never data.json
|
||||
- **SQL injection**: All SQL queries use parameterized statements
|
||||
60
reports/07-final-recommendation.md
Normal file
60
reports/07-final-recommendation.md
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
# Final Recommendation
|
||||
|
||||
> PaperForge PDF Annotation Layer — Architecture & MVP
|
||||
|
||||
## Summary
|
||||
|
||||
After comprehensive research across ZotFlow, Zotero source code, and Obsidian PDF plugin ecosystem, the recommendation is clear:
|
||||
|
||||
**Build a lightweight annotation layer using local Zotero SQLite read-only parsing, cached in a dedicated annotations.db, displayed via monkey-patched PDF viewer overlays in Obsidian, with write-back architecture designed from day one but implemented in Phase 3.**
|
||||
|
||||
## Why This Approach
|
||||
|
||||
### 1. ZotFlow is too heavy; PaperForge should not replicate it
|
||||
ZotFlow's bidirectional sync engine (42,918 lines), embedded reader iframe (entire `zotero/reader` submodule), CodeMirror 6 extensions, React UI, Comlink/Penpal communication — 80% of ZotFlow's code is specific to being a full Zotero-in-Obsidian replacement. PaperForge needs only annotation display and lightweight sync.
|
||||
|
||||
### 2. Local SQLite reading is sufficient for v1, ideal for v1
|
||||
All annotation data is available in Zotero's `zotero.sqlite`, fully documented. Read access is explicitly permitted by Zotero's documentation. The read path is:
|
||||
- Copy DB → temp → parse → annotations.db
|
||||
|
||||
### 3. Hybrid read/write architecture is the right long-term design
|
||||
- **Read**: Local SQLite (fast, offline, zero config)
|
||||
- **Write**: Zotero Web API (safe, versioned, official)
|
||||
- **schema includes `sync_state = pending_push` from day one**
|
||||
|
||||
### 4. Custom overlay via monkey-patching is the right UX choice
|
||||
Users chose "direct monkey-patch" over alternatives. PDF++ has proven this is sustainable despite private API dependency.
|
||||
|
||||
## Recommended Development Order
|
||||
|
||||
```
|
||||
Phase 1 (1 week): annotations.db + CLI import/list/patch/delete
|
||||
Phase 2 (1-2 weeks): Obsidian PDF overlay (show + create + edit)
|
||||
Phase 3 (post-MVP): Zotero Web API write-back
|
||||
```
|
||||
|
||||
## Key Numbers
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| New Python files | ~6 (annotation/ package) |
|
||||
| New TS files | ~7 (pdf-overlay/ directory) |
|
||||
| New CLI commands | 6 (import, list, create, patch, delete, export, status) |
|
||||
| New DB | 1 (annotations.db, independent schema) |
|
||||
| DB migrations planned | schema v1 (MVP), v2 (Web API write-back) |
|
||||
| Estimated development time | 2-3 weeks for MVP |
|
||||
| Risk level | Medium (Obsidian API stability) |
|
||||
|
||||
## What Was NOT Recommended
|
||||
|
||||
| Approach | Why Rejected |
|
||||
|----------|-------------|
|
||||
| Full ZotFlow bidirectional sync | Over-engineered for PaperForge's needs |
|
||||
| Custom PDF.js reader iframe | Heavy, UX fragmentation, Obsidian already has PDF.js |
|
||||
| Annotator-style markdown storage | Fragile, not AI-queryable, hard to maintain |
|
||||
| Dedicated PDF annotation service | Overkill; local SQLite + CLI is sufficient for v1 |
|
||||
| Read-only only (no write-back design) | User explicitly requested write-back be designed from start |
|
||||
|
||||
## Final Assessment
|
||||
|
||||
PaperForge's Annotation Layer is **well-scoped for MVP delivery in 2-3 weeks**. The architecture is clean, the risks are manageable, and the design accounts for future write-back without over-engineering the first version. The key dependency — Obsidian's private PDF viewer API — is a managed risk that PDF++ has successfully navigated for 2+ years.
|
||||
30
tests/test_dashboard_command.py
Normal file
30
tests/test_dashboard_command.py
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
from pathlib import Path
|
||||
|
||||
from paperforge.commands.dashboard import _dashboard_from_db
|
||||
|
||||
|
||||
def test_dashboard_from_db_uses_configured_system_dir(tmp_path: Path) -> None:
|
||||
(tmp_path / "paperforge.json").write_text(
|
||||
'{"vault_config": {"system_dir": "02_文献管理/System"}}',
|
||||
encoding="utf-8",
|
||||
)
|
||||
db_path = tmp_path / "02_文献管理" / "System" / "PaperForge" / "indexes" / "paperforge.db"
|
||||
db_path.parent.mkdir(parents=True)
|
||||
|
||||
import sqlite3
|
||||
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
conn.execute(
|
||||
"CREATE TABLE papers (domain TEXT, has_pdf INTEGER, ocr_status TEXT)"
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO papers(domain, has_pdf, ocr_status) VALUES (?, ?, ?)",
|
||||
("Sports", 1, "done"),
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
data = _dashboard_from_db(tmp_path)
|
||||
|
||||
assert data is not None
|
||||
assert data["stats"]["papers"] == 1
|
||||
69
tests/test_runtime_path_resolution.py
Normal file
69
tests/test_runtime_path_resolution.py
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from paperforge.commands.annotation import _resolve_paper_key
|
||||
from paperforge.embedding._chroma import get_vector_db_path
|
||||
from paperforge.embedding.build_state import get_vector_build_state_path
|
||||
from paperforge.setup.checker import SetupChecker
|
||||
|
||||
|
||||
def test_resolve_paper_key_uses_configured_memory_db_path(tmp_path: Path) -> None:
|
||||
(tmp_path / "paperforge.json").write_text(
|
||||
'{"vault_config": {"system_dir": "02_文献管理/System"}}',
|
||||
encoding="utf-8",
|
||||
)
|
||||
db_path = tmp_path / "02_文献管理" / "System" / "PaperForge" / "indexes" / "paperforge.db"
|
||||
db_path.parent.mkdir(parents=True)
|
||||
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
conn.execute("CREATE TABLE papers (zotero_key TEXT, pdf_path TEXT)")
|
||||
conn.execute(
|
||||
"INSERT INTO papers(zotero_key, pdf_path) VALUES (?, ?)",
|
||||
("K1", "[[02_文献管理/System/Zotero/storage/ABCD1234/paper.pdf]]"),
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
key = _resolve_paper_key(tmp_path, "02_文献管理/System/Zotero/storage/ABCD1234/paper.pdf")
|
||||
|
||||
assert key == "K1"
|
||||
|
||||
|
||||
def test_setup_checker_uses_configured_system_dir_for_exports(tmp_path: Path) -> None:
|
||||
(tmp_path / "paperforge.json").write_text(
|
||||
'{"vault_config": {"system_dir": "02_文献管理/System"}}',
|
||||
encoding="utf-8",
|
||||
)
|
||||
exports_dir = tmp_path / "02_文献管理" / "System" / "PaperForge" / "exports"
|
||||
exports_dir.mkdir(parents=True)
|
||||
(exports_dir / "demo.json").write_text("[]", encoding="utf-8")
|
||||
|
||||
result = SetupChecker(tmp_path).run()
|
||||
|
||||
assert result.details["bbt_exports_found"] is True
|
||||
|
||||
|
||||
def test_vector_runtime_paths_use_configured_system_dir(tmp_path: Path) -> None:
|
||||
(tmp_path / "paperforge.json").write_text(
|
||||
'{"vault_config": {"system_dir": "02_文献管理/System"}}',
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
vector_db_path = get_vector_db_path(tmp_path)
|
||||
build_state_path = get_vector_build_state_path(tmp_path)
|
||||
|
||||
assert "02_文献管理" in str(vector_db_path)
|
||||
assert "02_文献管理" in str(build_state_path)
|
||||
assert vector_db_path.name == "vectors"
|
||||
assert build_state_path.name == "vector-build-state.json"
|
||||
|
||||
|
||||
def test_preflight_index_path_resolves_correctly(tmp_path: Path) -> None:
|
||||
(tmp_path / "paperforge.json").write_text(
|
||||
'{"vault_config": {"system_dir": "02_文献管理/System"}}',
|
||||
encoding="utf-8",
|
||||
)
|
||||
from paperforge.config import paperforge_paths
|
||||
paths = paperforge_paths(tmp_path)
|
||||
expected = tmp_path / "02_文献管理" / "System" / "PaperForge" / "indexes" / "formal-library.json"
|
||||
assert paths["index"] == expected.resolve()
|
||||
24
tests/test_sync_command.py
Normal file
24
tests/test_sync_command.py
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
from pathlib import Path
|
||||
|
||||
from paperforge.commands.sync import _write_orphan_state
|
||||
from paperforge.core.result import PFResult
|
||||
|
||||
|
||||
def test_write_orphan_state_uses_configured_system_dir(tmp_path: Path) -> None:
|
||||
(tmp_path / "paperforge.json").write_text(
|
||||
'{"vault_config": {"system_dir": "02_文献管理/System"}}',
|
||||
encoding="utf-8",
|
||||
)
|
||||
result = PFResult(
|
||||
ok=True,
|
||||
command="sync",
|
||||
version="1.0.0",
|
||||
data={"prune": {"preview": [{"key": "K1"}]}}
|
||||
)
|
||||
|
||||
_write_orphan_state(tmp_path, result)
|
||||
|
||||
expected = tmp_path / "02_文献管理" / "System" / "PaperForge" / "indexes" / "sync-orphan-state.json"
|
||||
wrong = tmp_path / "System" / "PaperForge" / "indexes" / "sync-orphan-state.json"
|
||||
assert expected.exists()
|
||||
assert not wrong.exists()
|
||||
|
|
@ -46,6 +46,18 @@ class TestSyncServicePruneMethod:
|
|||
mock_read.assert_called_once()
|
||||
mock_fn.assert_called_once()
|
||||
|
||||
def test_load_exports_defaults_to_system_dir_when_config_missing(self, tmp_path: Path) -> None:
|
||||
svc = SyncService(tmp_path)
|
||||
exports_dir = tmp_path / "System" / "PaperForge" / "exports"
|
||||
exports_dir.mkdir(parents=True)
|
||||
(exports_dir / "demo.json").write_text("[]", encoding="utf-8")
|
||||
|
||||
with patch("paperforge.services.sync_service.load_export_rows", return_value=[{"key": "K1"}]) as mock_load:
|
||||
rows = svc.load_exports()
|
||||
|
||||
mock_load.assert_called_once_with(exports_dir / "demo.json")
|
||||
assert rows == [{"key": "K1"}]
|
||||
|
||||
|
||||
class TestSyncServiceRunPrune:
|
||||
"""SyncService.run() with prune/prune_force flags."""
|
||||
|
|
|
|||
Loading…
Reference in a new issue