obsidian-plugin/lllin000_PaperForge
1
0
Fork 0
mirror of https://github.com/lllin000/PaperForge.git synced 2026-07-22 06:50:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs: complete project research

Browse source
This commit is contained in:
Research Assistant 2026-05-03 13:46:11 +08:00
parent 3db87ce120
commit 8fee2b10ef
5 changed files with 1246 additions and 726 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

679
.planning/research/ARCHITECTURE.md
View file

@ -1,193 +1,560 @@
# Architecture Research — v1.5 Obsidian Plugin Settings Tab
# Architecture Patterns
**Domain:** Obsidian plugin architecture — settings tab integration with existing sidebar
**Researched:** 2026-04-29
**Confidence:** HIGH
**Domain:** PaperForge v1.6 literature asset foundation integration
**Researched:** 2026-05-03
## Current Architecture (v1.4 plugin)
## Recommended Architecture
```
paperforge/plugin/main.js (257 lines, CommonJS)
│
├── PaperForgePlugin extends Plugin
│ ├── onload() → registerView, addRibbonIcon, addCommand(sync, ocr)
│ └── onunload() → detachLeavesOfType
│
├── PaperForgeStatusView extends ItemView ← SIDEBAR (unchanged)
│ ├── _buildPanel() → metrics, OCR progress, quick actions
│ ├── _fetchStats() → exec('python -m paperforge status --json')
│ └── _runAction() → exec('python -m paperforge sync/ocr')
│
└── ACTIONS[] → sync, ocr command definitions
Evolve the existing `formal-library.json` into the canonical asset index rather than introducing a second competing index. The current code already has a single write point for the file in `paperforge/worker/sync.py` (`run_index_refresh`) and a stable path contract in `_utils.pipeline_paths()["index"]`. Reusing that artifact preserves the current layout, minimizes migration risk, and avoids a new “which index is truth?” problem.
The target model should be:
```text
paperforge.json / vault_config -> configuration truth
library-records/*.md -> user intent truth
ocr/<key>/meta.json + fulltext/images/json -> OCR asset truth
Literature/<domain>/<key> - <title>.md -> formal note + deep-reading truth
indexes/formal-library.json -> canonical derived read model
plugin dashboard / status / health / maturity UI -> thin-shell views over canonical index
```
Key observations:
- No `loadData`/`saveData` usage — no persistence at all
- No `PluginSettingTab` subclass — no settings tab
- Uses `exec` (not `spawn`) for subprocesses
- Single file, no module splitting
This keeps PaperForge local-first and file-based, but makes one important shift: lifecycle, health, readiness, and maturity are no longer recomputed ad hoc by each surface. They are derived once in Python and published through the canonical index.
## Target Architecture (v1.5)
## Architecture Diagram
```
paperforge/plugin/main.js
│
├── PaperForgePlugin extends Plugin ← MODIFIED (adds settings tab)
│ ├── DEFAULT_SETTINGS (new) ← Settings data shape
│ ├── settings (new) ← Runtime settings state (loadData/saveData)
│ ├── onload() ← MODIFIED: adds addSettingTab()
│ │ ├── registerView (unchanged)
│ │ ├── addRibbonIcon (unchanged)
│ │ ├── addSettingTab(new PaperForgeSettingTab(this.app, this)) ← NEW
│ │ └── addCommand × 2 (unchanged)
│ ├── loadSettings() (new) ← Object.assign(DEFAULTS, await loadData())
│ └── saveSettings() (new) ← await saveData(this.settings)
│
├── PaperForgeSettingTab extends PluginSettingTab ← NEW: settings tab
│ ├── display()
│ │ ├── Section: "基础路径" → vault_path, system_dir, resources_dir, etc.
│ │ ├── Section: "API 密钥" → paddleocr_api_key (password field)
│ │ ├── Section: "Zotero 链接" → zotero_data_dir
│ │ └── Section: "安装" → Install button + status area
│ ├── _validate() ← Field-level validation
│ ├── _runSetup() ← spawn('python -m paperforge setup')
│ └── _formatNotice() ← Polished notice from step results
│
├── PaperForgeStatusView extends ItemView ← SIDEBAR (completely unchanged)
│
└── ACTIONS[] (unchanged)
```text
+----------------------+
| paperforge.json |
| vault_config |
+----------+-----------+
|
v
paperforge.config.py
|
+--------------------+--------------------+
| | |
v v v
selection-sync ocr worker deep-reading sync
(user intent mirror) (asset producer) (agent-result sync)
| | |
+--------------------+--------------------+
|
v
NEW: asset index builder / state resolver
|
v
<system_dir>/PaperForge/indexes/formal-library.json
|
+---------------+----------------+
| |
v v
CLI JSON views/status Obsidian plugin dashboard
doctor/repair summaries Base-support mirror fields
```
## Integration Points
## Decision: Evolve `formal-library.json`, do not add a second canonical index
### 1. Plugin class additions (MODIFIED)
### Recommendation
```js
// Settings data model
const DEFAULT_SETTINGS = {
vault_path: '',
system_dir: '99_System',
resources_dir: '20_Resources',
literature_dir: 'Literature',
control_dir: 'Control',
agent_config_dir: '.opencode',
paddleocr_api_key: '',
zotero_data_dir: '',
};
Keep the path:
// In PaperForgePlugin:
async loadSettings() {
this.settings = Object.assign({}, DEFAULT_SETTINGS, await this.loadData());
}
`<system_dir>/PaperForge/indexes/formal-library.json`
async saveSettings() {
await this.saveData(this.settings);
}
but change its schema from a bare list of note rows to a versioned envelope:
// In onload(), add after existing registrations:
this.addSettingTab(new PaperForgeSettingTab(this.app, this));
```
### 2. SettingsTab class (NEW)
```js
class PaperForgeSettingTab extends PluginSettingTab {
constructor(app, plugin) {
super(app, plugin);
this.plugin = plugin;
}
display() {
const { containerEl } = this;
containerEl.empty();
// ... build form sections ...
```json
{
"schema_version": "2",
"generated_at": "2026-05-03T12:00:00Z",
"config_snapshot": {
"system_dir": "99_System",
"resources_dir": "03_Resources",
"literature_dir": "Literature",
"control_dir": "LiteratureControl",
"base_dir": "05_Bases"
},
"summary": {
"total_assets": 0,
"health": {},
"maturity": {}
},
"items": [
{
"zotero_key": "ABCDEFG",
"domain": "骨科",
"title": "...",
"intent": { "analyze": false, "do_ocr": false },
"assets": { "pdf": {}, "ocr": {}, "note": {}, "figures": {} },
"state": { "lifecycle": "ocr_ready", "ai_ready": false },
"health": { "library": "healthy", "pdf": "ok", "ocr": "pending" },
"maturity": { "score": 35, "level": 2, "next_step": "run_ocr" },
"paths": { "record": "...", "note": "...", "ocr_meta": "..." }
}
]
}
```
### 3. Subprocess interface (NEW pattern: spawn instead of exec)
### Why not a new file?
Settings tab runs setup via `spawn` for non-blocking progress:
| Option | Verdict | Why |
|---|---|---|
| Add `asset-index.json` beside `formal-library.json` | Reject | Duplicates responsibility and creates migration drift immediately |
| Rename `formal-library.json` to a new filename | Reject for v1.6 | Touches path contracts and upgrade logic without enough payoff |
| Evolve `formal-library.json` in place | Accept | Lowest-risk change, existing path already centralized |
```js
_runSetup() {
const { spawn } = require('node:child_process');
const child = spawn('python', ['-m', 'paperforge', 'setup'], {
cwd: this.plugin.settings.vault_path,
});
child.stdout.on('data', (data) => { /* parse step, show notice */ });
child.stderr.on('data', (data) => { /* parse error, show friendly notice */ });
child.on('close', (code) => { /* final notice */ });
### Migration rule
Add one tolerant reader in Python that accepts both:
- legacy list format
- new envelope format with `items`
Writers should emit only the new envelope after v1.6 migration.
## Data Ownership Boundaries
### 1. Configuration truth
**Authoritative source:** `paperforge.json` + `vault_config`
**Owner:** `paperforge/config.py`
**Rule:** plugin settings are not runtime truth. They are only a UI cache/draft until written back through Python.
Implication:
- `paperforge/plugin/main.js` should stop owning independent defaults like `System`, `Resources`, `Notes`, `Index_Cards`, `Base`.
- plugin should load resolved config from Python or from `paperforge.json`, then persist only as a mirror.
- any config-changing UI should write via setup/config command flow, not invent JS-only state.
### 2. User intent truth
**Authoritative source:** `library-records/<domain>/<key>.md` frontmatter
**User-owned fields:**
- `analyze`
- `do_ocr`
- future manual workflow flags
**Machine-mirrored fields:**
- `has_pdf`
- `pdf_path`
- `ocr_status`
- `deep_reading_status`
- `fulltext_md_path`
- future `asset_state`, `library_health`, `maturity_score`, `next_step`
Rule: users may edit intent fields; workers may overwrite mirrored fields.
### 3. OCR asset truth
**Authoritative source:**
- `ocr/<key>/meta.json`
- `ocr/<key>/fulltext.md`
- `ocr/<key>/json/result.json`
- `ocr/<key>/images/*`
**Owner:** `paperforge/worker/ocr.py`
Rule: no other component should invent OCR completion state without validating these files.
### 4. Deep-reading truth
**Authoritative source:** formal note content under `## 🔍 精读`
**Owner:** agent output + `has_deep_reading_content()` / `run_deep_reading()`
Rule: `deep_reading_status` is derived from actual note content, not from the plugin.
### 5. Canonical derived truth
**Authoritative source for lifecycle/health/maturity/dashboard:** `formal-library.json`
**Owner:** new asset-index builder in Python
Rule: plugin, `status`, health UI, and future context pack features read from this index instead of recomputing from filesystem independently.
## Recommended New/Modified Artifacts
### New Python modules
| Artifact | Type | Responsibility |
|---|---|---|
| `paperforge/worker/asset_index.py` | New | Read all source artifacts, derive canonical item rows, write `formal-library.json` |
| `paperforge/worker/asset_state.py` | New | Pure lifecycle/readiness/health/maturity derivation functions |
| `paperforge/worker/context_pack.py` | New | Build per-paper/per-collection AI context packs from canonical items |
| `paperforge/commands/config.py` | New | JSON get/set surface for plugin config sync |
| `paperforge/commands/context_pack.py` | New | CLI wrapper for ask-this-paper / ask-this-collection / copy-context-pack |
### Modified Python modules
| Artifact | Change |
|---|---|
| `paperforge/config.py` | Make resolved config the single runtime truth; expose stable JSON-serializable config payload |
| `paperforge/worker/sync.py` | Split note-writing from index-building; call asset index refresh after selection/index changes |
| `paperforge/worker/ocr.py` | After each touched key, refresh canonical index rows for those keys |
| `paperforge/worker/deep_reading.py` | Refresh canonical index after syncing deep-reading status |
| `paperforge/worker/status.py` | Read summary counts from canonical index instead of recounting raw files where possible |
| `paperforge/worker/repair.py` | Repair source artifacts first, then refresh canonical index |
| `paperforge/worker/base_views.py` | Add derived display columns/filters fed by mirrored fields from canonical index |
| `paperforge/setup_wizard.py` | Stop writing duplicated top-level path fields unless needed for backward compatibility; prefer `vault_config` |
| `paperforge/plugin/main.js` | Read config/dashboard data from Python, never own lifecycle logic |
### Modified on-disk artifacts
| Artifact | Status | Notes |
|---|---|---|
| `<system_dir>/PaperForge/indexes/formal-library.json` | Evolve | Canonical index envelope |
| `library-records/*.md` | Extend | Add derived mirror fields for Base/UI filtering |
| `paperforge.json` | Tighten | Single config truth; preserve legacy top-level keys only as compatibility shim |
### New on-disk artifacts
| Artifact | Purpose |
|---|---|
| `<system_dir>/PaperForge/indexes/context-packs/<key>.md` | Cached per-paper AI context pack |
| `<system_dir>/PaperForge/indexes/context-packs/collections/<slug>.md` | Cached per-collection AI context pack |
| `<system_dir>/PaperForge/indexes/health-report.json` | Optional summarized health snapshot for dashboard and diagnostics |
## State Model
The new model should explicitly separate intent, assets, and derived state.
### Intent layer
From `library-record` frontmatter:
```text
analyze
do_ocr
```
This is what the user wants.
### Asset layer
Observed from files:
```text
export present?
pdf resolvable?
ocr meta present?
ocr payload complete?
formal note present?
deep-reading content present?
figure assets present?
```
This is what actually exists.
### Derived state layer
Computed in Python only:
```text
imported
indexed
pdf_ready
ocr_requested
ocr_processing
fulltext_ready
figure_ready
deep_read_ready
deep_read_done
ai_context_ready
```
Recommended item schema section:
```json
"state": {
"lifecycle": "fulltext_ready",
"readiness": {
"pdf": true,
"ocr": true,
"deep_read": false,
"ai_context": false
},
"next_actions": ["generate_context_pack", "run_pf_deep"]
}
```
### 4. Notice formatting (NEW)
## Read/Write Flow
Current code: `new Notice('[!!] sync failed: ' + stderr)` — raw terminal output.
### Worker write flow
New pattern:
```js
_formatNotice(type, step, detail) {
const messages = {
success: { prefix: '[OK]', style: 'notice-success' },
error: { prefix: '[!!]', style: 'notice-error' },
progress: { prefix: '[...]', style: '' },
};
// Map raw stderr to friendly Chinese messages
// Log full details to console, show summary in Notice
}
```text
Zotero export change
-> run_selection_sync()
-> update library-records (intent + mirrors)
-> refresh_asset_index(keys=changed)
OCR job change
-> run_ocr()
-> update meta.json/fulltext/assets
-> refresh_asset_index(keys=touched)
Deep-reading state change
-> run_deep_reading()
-> sync deep_reading_status mirror
-> refresh_asset_index(keys=touched)
Repair change
-> run_repair()
-> fix source artifacts
-> refresh_asset_index(keys=fixed)
```
## Data Flow
### Plugin read/write flow
```
1. User opens Obsidian → Plugin Settings → PaperForge
2. display() renders form fields from this.plugin.settings
3. User fills in paths, API key
4. Each text input onChange → debounced saveSettings()
5. User clicks "安装配置" button
6. _runSetup():
a. _validate() all fields → show specific errors if any
b. Disable button, show "[...] 正在配置..."
c. spawn('python -m paperforge setup') with env vars from settings
d. Parse stdout line-by-line for step progress
e. Show Notice per step ("创建目录... ✓", "写入配置... ✓")
f. On completion: Enable button, show "[OK] 配置完成" or "[!!] 配置失败: reason"
7. Sidebar continues working independently (still reads paperforge.json via exec)
```text
Plugin action button
-> spawn `python -m paperforge <command>`
-> command mutates source artifacts
-> command refreshes canonical index
-> plugin re-reads JSON summary/query output
Plugin dashboard load
-> spawn `python -m paperforge status --json` for summary
-> spawn `python -m paperforge context-pack/index query --json` for lists/detail
-> render only; no lifecycle recomputation in JS
```
## Component Responsibilities
### Critical rule
| Component | Responsibility | Status |
|-----------|---------------|--------|
| `DEFAULT_SETTINGS` | Data shape, defaults | NEW |
| `PaperForgePlugin.loadSettings()` | Persistence load with defaults merge | NEW |
| `PaperForgePlugin.saveSettings()` | Debounced persistence save | NEW |
| `PaperForgeSettingTab.display()` | UI rendering | NEW |
| `PaperForgeSettingTab._validate()` | Pre-submit field validation | NEW |
| `PaperForgeSettingTab._runSetup()` | Setup subprocess orchestration | NEW |
| `PaperForgeSettingTab._formatNotice()` | Human-readable notice output | NEW |
| `PaperForgeStatusView` | Sidebar (unchanged) | EXISTING |
| `ACTIONS[]` | Quick action definitions (unchanged) | EXISTING |
The plugin should never determine:
- whether OCR is truly complete
- whether a paper is AI-ready
- whether health is red/yellow/green
- what the next step should be
## Build Order
Those are Python-derived fields.
```
Phase 1: Settings Shell + Persistence
├── Add DEFAULT_SETTINGS
├── Add loadSettings() / saveSettings() to Plugin
├── Create PaperForgeSettingTab with display()
├── Register via addSettingTab() in onload()
└── Verify: settings tab appears, fields persist across restart, sidebar still works
## Integration Points in Existing Code
Phase 2: Install Button + Setup Orchestration
├── Add _validate() for field-level checks
├── Add _runSetup() with spawn integration
├── Add _formatNotice() for polished output
└── Verify: install button works, notices are human-readable, sidebar unaffected
### `paperforge/config.py`
Current strength: already centralizes precedence. Current problem: plugin defaults drift from Python defaults and setup writes both top-level keys and `vault_config`.
Recommendation:
- keep `load_vault_config()` as the only resolver
- add a JSON export helper used by plugin
- prefer nested `vault_config` as canonical
- keep top-level path keys only as compatibility read support, not required write support
### `paperforge/worker/sync.py`
Current role is overloaded:
1. ingest export rows
2. write library-records
3. write formal notes
4. write `formal-library.json`
Recommendation:
- leave `run_selection_sync()` focused on control records
- leave `run_index_refresh()` focused on formal notes
- move canonical index assembly into `asset_index.py`
- have both call `refresh_asset_index()` rather than hand-building `index_rows`
This is the cleanest way to integrate new lifecycle/health fields without making `sync.py` larger.
### `paperforge/worker/ocr.py`
Current role is already the source of OCR truth via `meta.json` and validation. Keep it that way.
Recommendation:
- do not push health logic into the plugin
- after status transitions, refresh canonical rows for touched keys
- expose validated OCR health in index as derived fields, not ad hoc status strings scattered across surfaces
### `paperforge/worker/deep_reading.py`
Current role is status synchronization, which fits v1.6 well.
Recommendation:
- continue deriving deep-read completion from note content
- publish the result into canonical index and mirrored record fields
- let maturity scoring depend on this derived state
### `paperforge/worker/status.py`
Current role manually counts files and scans frontmatter.
Recommendation:
- keep doctor-style filesystem checks in place
- move dashboard/status summaries to canonical index summary where possible
- use raw scans only as fallback if index missing/corrupt
This gives one consistent count across CLI and plugin.
### `paperforge/worker/repair.py`
Current role already handles state divergence.
Recommendation:
- extend it to compare source artifacts vs canonical index freshness
- never repair the index directly; repair sources, then rebuild index
### `paperforge/worker/base_views.py`
Current Base strategy is still good. Do not replace Bases with a new system.
Recommendation:
- add columns like `asset_state`, `library_health`, `maturity_level`, `next_step`
- continue filtering on `analyze`, `do_ocr`, `ocr_status`, `deep_reading_status`
- derive these extra display fields from canonical index and mirror them back into library-record frontmatter during refresh
This preserves current user workflow while making the dashboard and Bases agree.
### `paperforge/plugin/main.js`
Current plugin is correctly thin in command execution, but too independent in configuration and too limited in dashboard data.
Recommendation:
- keep CommonJS and the current shell approach; do not do a front-end rewrite
- add a config fetch path from Python
- add dashboard JSON fetches backed by canonical index
- keep all writes as CLI subprocesses
- keep settings as UI cache only, not truth
## Patterns to Follow
### Pattern 1: Canonical read model over file truths
**What:** derive one authoritative JSON read model from multiple file-backed truths.
**When:** whenever multiple surfaces need the same counts, health, lifecycle, or readiness logic.
**Example:**
```python
item = build_asset_item(
export_row=export_row,
record=parse_library_record(record_path),
ocr_meta=load_meta(meta_path),
note_state=inspect_note(note_path),
)
```
---
### Pattern 2: User-intent fields stay editable, derived fields stay overwriteable
**What:** keep manual controls and machine mirrors separate in the same markdown record.
**When:** preserving current Base workflow without losing canonical derivation.
**Example:**
*Architecture research for: PaperForge v1.5 Obsidian Plugin Setup Integration*
*Researched: 2026-04-29*
```yaml
analyze: true # user-owned
do_ocr: true # user-owned
asset_state: "ocr_ready" # machine-owned mirror
library_health: "warning" # machine-owned mirror
maturity_score: 55 # machine-owned mirror
```
### Pattern 3: Thin-shell plugin, thick Python semantics
**What:** JS renders and triggers commands; Python decides meaning.
**When:** any dashboard, health, maturity, or AI-ready feature.
**Example:**
```text
plugin -> python -m paperforge status --json -> render cards
plugin -> python -m paperforge context-pack PAPER123 --json -> render/copy
```
## Anti-Patterns to Avoid
### Anti-Pattern 1: Second lifecycle implementation in the plugin
**What:** JS infers OCR/deep-read/health from file presence or cached settings.
**Why bad:** guaranteed drift from CLI and workers.
**Instead:** plugin reads Python-produced JSON only.
### Anti-Pattern 2: Treating `formal-library.json` as user-editable data
**What:** manual edits or plugin patch writes into canonical index.
**Why bad:** index becomes a source instead of a read model.
**Instead:** source artifacts change first; index is regenerated.
### Anti-Pattern 3: Making `library-record` the source of all truth
**What:** stuffing every derived health/lifecycle decision into frontmatter and reading only that.
**Why bad:** repeats current divergence problems.
**Instead:** frontmatter stores intent plus mirrors; source truths remain separate.
### Anti-Pattern 4: Creating a greenfield asset database layer
**What:** SQLite/daemon/service rewrite for v1.6.
**Why bad:** violates local-first simplicity and existing file-based contract.
**Instead:** improve the current JSON + markdown architecture.
## Scalability Considerations
| Concern | At 100 users/papers | At 10K papers | At 1M papers |
|---|---|---|---|
| Index rebuild cost | Full rebuild acceptable | Prefer keyed/incremental refresh | Would need sharding or DB, out of scope |
| Plugin dashboard load | Read whole index fine | Add filtered JSON queries and summary endpoints | Full file read impractical |
| Base mirrors | Direct frontmatter update fine | Batch writes should be keyed and minimal | Too many markdown rewrites |
| Context packs | On-demand generation fine | Cache per paper/collection | Need streaming/chunking |
For v1.6, optimize for incremental per-key refresh, not a new database.
## Suggested Build Order
### Phase 1: Configuration truth hardening
- Normalize plugin defaults to Python defaults
- Add config JSON read surface
- Make `vault_config` the canonical write target
- Verify plugin, CLI, setup, and workers resolve identical paths
### Phase 2: Canonical index builder
- Add `asset_index.py`
- Evolve `formal-library.json` schema
- Add tolerant legacy reader
- Make `run_index_refresh()` delegate index writing here
### Phase 3: Unified state + health derivation
- Add lifecycle/readiness/health/maturity pure functions in `asset_state.py`
- Feed from library-records, OCR meta, formal notes
- Mirror selected display fields back into library-record frontmatter
### Phase 4: Status/repair/dashboard convergence
- Make `status --json` read canonical summary
- Make `repair` rebuild index after fixes
- Update plugin dashboard to read canonical summaries/query endpoints
### Phase 5: Maturity scoring + next-step recommendations
- Add scoring rules to canonical items
- Surface next-step guidance in plugin and Base views
### Phase 6: AI context packs
- Generate per-paper and per-collection context packs from canonical items
- Store pack paths/readiness in index
- Keep pack bodies outside the main index to avoid bloat
## Practical Field Additions
Recommended mirrored `library-record` fields for Bases/UI:
```yaml
asset_state: "fulltext_ready"
pdf_health: "ok"
ocr_health: "warning"
template_health: "ok"
library_health: "warning"
maturity_score: 62
maturity_level: 3
next_step: "run_pf_deep"
context_pack_ready: false
```
Recommended canonical index sections per item:
```json
"intent": {},
"assets": {},
"state": {},
"health": {},
"maturity": {},
"recommendations": {},
"paths": {}
```
## Sources
- Internal code inspection: `paperforge/config.py` — config precedence and path resolution
- Internal code inspection: `paperforge/worker/_utils.py` — existing `formal-library.json` path contract
- Internal code inspection: `paperforge/worker/sync.py` — current library-record/note/index write flow
- Internal code inspection: `paperforge/worker/ocr.py` — OCR truth and validation flow
- Internal code inspection: `paperforge/worker/deep_reading.py` — deep-reading state sync
- Internal code inspection: `paperforge/worker/status.py` — current summary/dashboard JSON producer
- Internal code inspection: `paperforge/worker/repair.py` — divergence repair model
- Internal code inspection: `paperforge/plugin/main.js` — current plugin thin-shell pattern and config drift risk

169
.planning/research/FEATURES.md
View file

@ -1,62 +1,143 @@
# Feature Research — v1.5 Obsidian Plugin Settings Tab
# Feature Landscape
**Domain:** Obsidian plugin UX — settings tab with setup wizard integration
**Researched:** 2026-04-29
**Domain:** AI-ready literature asset management on top of Obsidian + Zotero
**Researched:** 2026-05-03
**Confidence:** HIGH
## Feature Landscape
## Framing
### Table Stakes (Must Have)
This milestone should treat PaperForge as a durable literature asset platform, not a bundle of one-off reading prompts. In this ecosystem, users already expect Zotero to remain the bibliographic source of truth, Obsidian to remain the knowledge workspace, and any AI layer to be grounded in traceable local assets rather than opaque summaries.
| Feature | Why | Complexity |
|---------|-----|------------|
| Settings tab opens from plugin settings | Standard Obsidian UX — every plugin with config has this | LOW — `PluginSettingTab` is 1 class |
| All wizard fields rendered as form inputs | Users expect to see every config option they'd enter in CLI wizard | LOW — `Setting.addText()` per field |
| Field labels are clear and beginner-friendly | Target user is a novice researcher | LOW — Chinese labels with English tooltips |
| Settings persist across Obsidian restarts | Users shouldn't re-enter config every session | LOW — `loadData/saveData` |
| Default values pre-filled | Reduces friction; matches setup_wizard.py defaults | LOW — `DEFAULT_SETTINGS` object |
| "Install" button triggers setup | Core value prop — one click replaces terminal wizard | MEDIUM — subprocess orchestration |
| Install progress feedback | Users need to know the setup isn't frozen | MEDIUM — step-by-step notices |
For v1.6, the highest-value features are the ones that make the library legible, repairable, standardized, and reusable by both humans and AI. The strongest differentiators are not "more extraction buttons"; they are reliable lifecycle state, health visibility, and reusable context packaging.
### Differentiators (Valuable Polish)
## Table Stakes
| Feature | Value | Complexity |
|---------|-------|------------|
| Human-readable notices, never raw terminal output | User explicitly requested this; major UX differentiator vs CLI | MEDIUM — error pattern matching + friendly messages |
| Step-by-step progress ("Creating directories... ✓", "Writing config... ✓") | Makes complex setup feel manageable | LOW — notice per setup step |
| Field validation before install (path exists, API key format) | Prevents cryptic failures mid-install | MEDIUM — validation logic |
| Chinese-language UI | Target audience is Chinese medical researchers | LOW — all labels/notices in Chinese |
| Settings tab doesn't break existing sidebar | Sidebar must work exactly as before | MEDIUM — careful scoping of SettingTab class |
Features users will expect from the new milestone direction. Missing these makes the product feel like a collection of scripts rather than a literature asset manager.
### Anti-Features (Explicitly Avoid)
| Category | Feature | Why Expected | Complexity | Existing Dependencies | Notes |
|---------|---------|--------------|------------|------------------------|-------|
| Lifecycle management | **Canonical asset index** | Long-lived libraries need one place to answer: what exists, what is missing, what is usable | High | `paperforge sync`, library-records, formal notes, OCR `meta.json`, `paperforge.json` | This is the foundation. Dashboard and AI entry points should read this, not recompute state separately. |
| Lifecycle management | **Explicit lifecycle state model** | Researchers need stable states like imported, indexed, OCR-ready, fulltext-ready, deep-read, AI-ready | Medium | Existing queue fields, OCR status, deep-reading status | Separate user intent from machine facts and derived readiness. |
| Health diagnostics | **Library health surfaces** | Large libraries inevitably drift; users expect to see broken PDFs, OCR failures, path drift, template drift | Medium | `doctor`, `repair`, path normalization, OCR pipeline | Should be visible per paper and aggregated per collection/library. |
| Asset standardization | **Stable per-asset schema** | AI workflows only scale when metadata, links, fulltext, and notes are predictably shaped | Medium | library-record frontmatter, formal note frontmatter, path normalization | Standardize identifiers, asset paths, provenance fields, and readiness flags. |
| Workflow progression | **Queue and next-step views driven by derived state** | Users expect actionable progression, not raw status fields | Medium | Existing Base views, canonical index, lifecycle model | "What should I do next?" is a table-stakes question for a maintained library. |
| Workflow progression | **Idempotent rebuild / refresh behavior** | Asset managers must safely recompute views after fixes without corrupting notes | Medium | current workers, repair flows, config single source of truth | Essential for trust in long-term use. |
| Feature | Why Avoid |
|---------|-----------|
| Settings tab shows pipeline status/metrics | Sidebar already does this; duplicating creates confusion |
| Settings tab has "Run Sync" / "Run OCR" buttons | Sidebar action cards already do this |
| Real-time settings sync between tabs | Unnecessary complexity for a single-user tool |
| Multi-language i18n system | Chinese-only is sufficient for target audience; i18n can be added later |
## Differentiators
These are the features that can make PaperForge notably better than a generic Zotero-to-Obsidian sync.
| Category | Feature | Value Proposition | Complexity | Existing Dependencies | Notes |
|---------|---------|-------------------|------------|------------------------|-------|
| AI context entry points | **Ask-this-paper context pack** | Turns one paper into a reusable, grounded AI entry point with metadata, fulltext, figures, note links, and provenance | Medium | canonical index, OCR fulltext, figure-map, formal note, PDF link | Strong differentiator because it packages existing assets instead of inventing new extraction schemas. |
| AI context entry points | **Ask-this-collection / copy-context-pack** | Lets users move from single-paper workflows to collection-scale synthesis while staying source-grounded | Medium | collection metadata from Zotero, canonical index, note links | Aligns with NotebookLM-style source-grounded workflows, but stays local-first and reusable. |
| Workflow progression | **Library maturity / workflow level scoring** | Gives researchers a simple, motivating signal: how usable is this paper or collection for AI and downstream writing? | Medium | lifecycle model, health diagnostics, canonical index | Valuable if transparent and rule-based; bad if gamified or opaque. |
| Health diagnostics | **Actionable diagnostics with fix paths** | Better than just saying "broken"; tells the user exactly whether to sync, OCR, repair paths, or regenerate a note | Medium | `doctor`, `repair`, queue logic | This converts diagnostics from reporting into workflow guidance. |
| Asset standardization | **Traceable provenance bundle** | Every AI-facing output can point back to PDF, OCR text, notes, and source identifiers | Medium | normalized paths, formal notes, OCR artifacts | Important differentiator for research trust and reuse. |
| Lifecycle management | **Thin-shell plugin dashboard over canonical index** | Keeps UX polished without creating a second business-logic implementation | High | plugin shell, CLI, canonical index, `paperforge.json` | This is product-quality differentiation, not just engineering cleanliness. |
## Anti-Features
These are attractive distractions for this milestone. Avoid them.
| Anti-Feature | Why Avoid | What to Do Instead |
|--------------|-----------|-------------------|
| **Productizing discipline-specific extraction outputs** (PICO tables, mechanism tables, parameter tables, etc.) | High maintenance, domain-specific, and fights the platform direction | Ship reusable context packs and templates that specialized prompts can consume |
| **Replacing Zotero as the primary reference manager** | Zotero already owns collections, tags, duplicates, attachments, and citation workflows well | Treat Zotero metadata and attachment relationships as upstream truth |
| **Auto-running deep-reading agents from workers** | Breaks the worker/agent boundary and creates opaque automation | Keep agents user-invoked; improve readiness and context packaging instead |
| **A second lifecycle engine inside the Obsidian plugin** | Creates config drift and inconsistent state calculations | Plugin should read canonical index produced by Python logic |
| **"AI organizes everything" black-box features** | Researchers need traceability and repairability, not magic states they cannot audit | Use explicit states, rule-based maturity, and provenance-preserving outputs |
| **Feature explosion of per-prompt buttons** | Hard to maintain, quickly becomes UX clutter, and turns platform capabilities into brittle one-offs | Support a few generic entry points: ask-paper, ask-collection, copy-context-pack |
| **Cloud-hosted collaboration / sync for this milestone** | Expands scope far beyond the asset-foundation problem and conflicts with local-first constraints | Keep single-user, local-first architecture |
| **Full literature discovery graph product** (Litmaps/ResearchRabbit clone) | Discovery maps are valuable, but they are adjacent to asset reliability and curation, not the current foundation gap | Integrate with external discovery workflows through clean collections/tags/context packs |
## Feature Dependencies
```
PluginSettingTab class (settings UI shell)
└──requires──> DEFAULT_SETTINGS model (data shape)
└──requires──> loadData/saveData (persistence)
```text
Single configuration truth (`paperforge.json`)
-> enables canonical asset index
Form validation
└──enhances──> Install button (prevents wasted subprocess calls)
Canonical asset index
-> enables lifecycle state model
-> enables health dashboards
-> enables queue views
-> enables maturity scoring
-> enables AI context packs
Install button (setup orchestration)
└──requires──> Settings persistence (reads current values)
└──requires──> Subprocess runner (calls paperforge setup)
└──requires──> Notice formatter (polished output)
Stable per-asset schema
-> enables trustworthy index derivation
-> enables reusable AI entry points
Sidebar preservation
└──independent──> (settings tab is additive, sidebar code unchanged)
Health diagnostics
-> feeds workflow progression
-> feeds maturity scoring
AI context packs
-> require canonical index
-> require provenance fields
-> benefit from OCR/fulltext/figure readiness
```
---
## Milestone Scoping by Requested Category
*Feature research for: PaperForge v1.5 Obsidian Plugin Setup Integration*
*Researched: 2026-04-29*
### 1. Lifecycle management
- Must include: canonical index, explicit state model, recomputation rules
- Should defer: complicated branching workflows or multi-user orchestration
### 2. Health diagnostics
- Must include: PDF health, OCR health, path health, note/template/base health
- Should defer: speculative ML-based anomaly detection
### 3. Asset standardization
- Must include: canonical identifiers, normalized paths, provenance fields, consistent frontmatter contract
- Should defer: discipline-specific metadata taxonomies as built-ins
### 4. AI context entry points
- Must include: ask-this-paper, ask-this-collection, copy-context-pack
- Should defer: fully embedded chat platform, vector DB, or provider-specific lock-in as milestone-defining work
### 5. Workflow progression
- Must include: readiness views, next-step recommendations, maturity score/rules
- Should defer: complex automation that removes user control
### 6. What not to productize
- Do not turn every successful prompt into a first-class feature
- Do not hardcode scholarly extraction schemas into core product logic
- Do not duplicate Zotero or NotebookLM product surfaces
## Reusable Platform Capabilities vs One-Off Workflows
### Reusable platform capabilities to build
- Canonical index
- Lifecycle and health derivation
- Provenance-preserving asset schema
- Context pack generation
- Dashboard and queue views
- Rule-based maturity / next-step guidance
### One-off scholarly workflows to keep optional
- Medical evidence extraction tables
- Domain-specific summary formats
- Prompt-specific synthesis buttons
- Special-case manuscript scaffolds tied to one discipline
## MVP Recommendation
Prioritize:
1. **Canonical asset index + lifecycle state model**
2. **Library health surfaces + actionable next-step guidance**
3. **Generic AI context packs for paper and collection scopes**
Defer: **Domain-specific extraction products** — they should consume the platform, not define it.
## Sources
- Project direction and scope guardrails: `.planning/PROJECT.md` — HIGH confidence
- Zotero official docs: Collections and Tags (updated 2025-07-29) — https://www.zotero.org/support/collections_and_tags — HIGH confidence
- Zotero official docs: Adding Files to your Zotero Library (updated 2025-09-11) — https://www.zotero.org/support/attaching_files — HIGH confidence
- Zotero official docs: Duplicate Detection — https://www.zotero.org/support/duplicate_detection — MEDIUM confidence (older doc, still aligned with current product model)
- Obsidian official help: Bases / Bases syntax / Properties — via Context7 official help mirror — HIGH confidence
- Google official help: NotebookLM grounded chat with inline citations — https://support.google.com/notebooklm/answer/16164461?hl=en — HIGH confidence
- ResearchRabbit product positioning (discovery/visualization/organization) — https://www.researchrabbit.ai/ — MEDIUM confidence
- Litmaps product positioning (discover/visualize/monitor) — https://www.litmaps.com/ — MEDIUM confidence
- SciSpace product positioning (chat with PDF / literature review / extract data) — https://www.scispace.com/ — MEDIUM confidence

662
.planning/research/PITFALLS.md
View file

@ -1,439 +1,255 @@
# Pitfalls Research
# Domain Pitfalls
**Domain:** Obsidian Plugin Settings Tab (Retrofit to Existing CommonJS Plugin)
**Researched:** 2026-04-29
**Confidence:** HIGH
**Domain:** Brownfield evolution of PaperForge into an AI-ready local-first literature asset manager
**Researched:** 2026-05-03
> Source material: Official Obsidian API type definitions (`obsidian.d.ts`, v1.12+), existing PaperForge `main.js` (CommonJS, sidebar + ribbon + commands, zero settings), Obsidian `Setting` / `PluginSettingTab` / `Plugin.loadData/saveData` APIs, `Notice` / `ButtonComponent` / `exec` from `node:child_process`.
## Recommended v1.6 mitigation phases
---
1. **Phase 1 — Truth model and migration contract**
- Define source-of-truth ownership for config, user intent, machine facts, and derived state.
2. **Phase 2 — Canonical asset index and rebuild pipeline**
- Build the index as a deterministic projection with repair/rebuild support.
3. **Phase 3 — Health engine and thin-shell dashboard**
- Centralize health/readiness logic in Python; plugin renders only.
4. **Phase 4 — AI context packaging and maturity guidance**
- Add reusable context packs and next-step guidance without hardcoding discipline schemas.
5. **Phase 5 — Brownfield rollout, migration, and verification**
- Ship compatibility checks, doctor/repair upgrades, and safe rollout gates.
## Critical Pitfalls
### Pitfall 1: `loadData()` / `saveData()` Data Corruption Due to Timing and Partial Writes
**What goes wrong:**
Calling `this.saveData(this.settings)` inside an `onChange` handler fires on every keystroke. Each keystroke triggers a full `data.json` write (file overwrite). On slow filesystems or with rapid typing, the Obsidian Sync service may pick up intermediate partial writes, or two sequential writes may collide. The result: truncated or corrupted `data.json`, plugin settings reset to defaults on next load.
**Why it happens:**
`Plugin.saveData()` writes to `data.json` in `{vault}/.obsidian/plugins/paperforge/` synchronously from the caller's perspective (it returns a Promise but fires immediately). Obsidian's `Setting.addText(cb => cb.onChange(...))` fires on every `input` event. There is no built-in debounce. Developers naively add `this.saveData()` inside the onChange callback, creating a write-per-keystroke pattern.
**How to avoid:**
- **Debounce saveData**: Wrap `this.saveData()` in a debounce function (250-500ms). Clear it on component unload.
- **Atomic writes**: Obsidian's `saveData` is already atomic (write-to-temp then rename), but debouncing still prevents rapid-fire writes.
- **Schema versioning**: Always include `version: 2` in settings. On `loadData()`, check `settings.version` and run migration if missing or outdated.
```javascript
// Pattern — debounced save with version guard
let _saveTimeout = null;
const debouncedSave = () => {
clearTimeout(_saveTimeout);
_saveTimeout = setTimeout(() => this.saveData(this.settings), 300);
};
// In onunload: clearTimeout(_saveTimeout)
```
**Warning signs:**
- `data.json` shows intermediate/incomplete JSON after typing in a text field
- User reports "my settings reset" after Obsidian restart
- Obsidian Sync shows "conflict" on `data.json`
**Phase to address:**
Phase 1 (Settings tab shell + persistence layer). Build debounced save + versioning before ANY settings controls.
---
### Pitfall 2: Settings Tab `display()` Breaking the Existing Sidebar View
**What goes wrong:**
The existing plugin has a `PaperForgeStatusView` (sidebar `ItemView`). Adding a `PluginSettingTab` that mutates global state, overrides `this.app.workspace` internals, or triggers workspace layout changes will break the existing sidebar. Specifically: if the settings tab code uses `this.app.workspace.getLeaf()` or triggers a workspace layout save/restore, it can steal focus from the sidebar, close the sidebar pane, or cause layout jank.
**Why it happens:**
`PluginSettingTab` extends `SettingTab` which has a `display()` method called whenever the user switches to the settings tab. This method is called **fresh each time** — the `containerEl` is reused but emptied each time the tab gains focus. If `display()` opens/closes leaves, registers views, or calls `workspace.revealLeaf()`, it disrupts the workspace layout that the sidebar view depends on.
**How to avoid:**
- **Settings tab is self-contained**: Only mutate `this.containerEl` in `display()`. Never touch `this.app.workspace` from settings.
- **Never register views in `display()`**: View registration (`registerView`) belongs in `onload()`, not in settings display.
- **Do not call `PaperForgeStatusView.open()` from settings**: The settings tab should not trigger sidebar actions.
- **Settings actions that need refresh**: Use `Notice` to tell user "Restart required" or "Click Refresh in sidebar" rather than programmatically opening views.
**Warning signs:**
- Sidebar closes/disappears when switching to Settings tab
- "PaperForge Dashboard" ribbon icon stops working after visiting settings
- Workspace layout resets to default after opening settings
**Phase to address:**
Phase 1 (Settings tab shell). Day 1 test: open sidebar, open settings, verify sidebar still visible and functional.
---
### Pitfall 3: `exec` Blocking the Obsidian UI Thread During Long Subprocesses
**What goes wrong:**
The existing plugin uses `exec('python -m paperforge ...', { cwd, timeout }, callback)`. While `exec` itself is async-callback (non-blocking), a settings "Setup Wizard" button that runs `paperforge doctor` or `paperforge sync --setup` can take 10-60 seconds. The user sees a frozen settings tab with no progress indicator. Worse: if the callback throws and isn't caught, the plugin crashes silently. If the timeout fires, the child process may become a zombie (orphaned Python process keeping file handles open).
**Why it happens:**
`exec` buffers all stdout/stderr in memory and delivers them once the process exits. There is no streaming feedback to the settings UI. The `ButtonComponent.onClick(callback)` pattern passes an async callback — but `exec`'s callback-based API doesn't integrate with async/await naturally. The result: the button looks "stuck" with no visual feedback for 10-60 seconds.
**How to avoid:**
- **Use `execFile` or `spawn` for streaming**: `spawn` provides `stdout.on('data')` for live progress. Pipe it to a progress display in the settings tab.
- **Button state management**: Immediately disable the button (`.setDisabled(true)` + `.setButtonText('Running...')`) after click. Re-enable in callback.
- **Wrap exec in Promise with proper cleanup**:
```javascript
const runCommand = (cmd, args, cwd, timeoutMs) => new Promise((resolve, reject) => {
const child = spawn(cmd, args, { cwd, timeout: timeoutMs, shell: true });
let stdout = '', stderr = '';
child.stdout.on('data', d => stdout += d);
child.stderr.on('data', d => stderr += d);
child.on('close', code => {
if (code === 0) resolve(stdout);
else reject(new Error(stderr || `Exit code ${code}`));
});
child.on('error', reject);
});
```
- **Show incremental progress**: For long commands, use `--verbose` or `--progress` flags on the Python side and parse streaming output for progress updates.
- **Timeout + cleanup**: Explicitly `child.kill('SIGTERM')` on timeout. Do not rely solely on the `timeout` option (which can fail on Windows).
**Warning signs:**
- Clicking "Run Setup" freezes Obsidian for 20+ seconds
- Multiple "Run" clicks spawn multiple subprocesses (because button wasn't disabled)
- Zombie Python processes after Obsidian close (check Task Manager)
- Settings tab becomes unresponsive with no loading indicator
**Phase to address:**
Phase 2 (Setup wizard controls). This is where subprocess UX matters most.
---
### Pitfall 4: Windows Path Encoding — `basePath` with Spaces and Unicode in `exec` cwd
**What goes wrong:**
The existing code uses `this.app.vault.adapter.basePath` as `cwd` for `exec`. On Windows, `basePath` returns a native path like `C:\Users\Lin\My Vault` (with spaces) or `D:\L\Med\Research\99_System\LiteraturePipeline` (with non-ASCII-like directory names). When passed to `exec('python -m paperforge ...', { cwd: basePath })`, the Windows command interpreter may fail to parse the path due to spaces, or the Python process may mishandle Unicode in working directory paths.
Additionally, `basePath` may have different case than the actual filesystem path (Windows is case-insensitive but Node's `path` comparisons are case-sensitive). If any code does `pathA === pathB` comparisons, it will fail.
**Why it happens:**
- `exec` uses `cmd.exe` on Windows for `shell: true` (default). Spaces in paths require quoting.
- `basePath` is a native OS path. Inside Obsidian's Electron runtime, this is fine for Vault API calls. But passing it directly to a Python subprocess as `cwd` may expose encoding mismatches (Electron's `basePath` may be UTF-16 on Windows while Python expects UTF-8 or system locale encoding).
- `path.sep` on Windows is `\`, but the vault adapter uses `/` internally. Mixed separators in path construction can cause "file not found" errors.
**How to avoid:**
- **Quote the cwd**: `{ cwd: basePath }` is fine — Node's `child_process` handles quoting internally. But verify by testing with a path like `C:\Users\Test User\My Vault`.
- **Normalize paths before using in Python commands**: Use `path.normalize()` for any path argument passed as a CLI argument, and wrap in double quotes.
- **Test with a vault path containing**: spaces, Chinese characters, em-dashes, and trailing spaces.
- **`spawn` over `exec`**: `spawn` accepts an explicit `cwd` option with proper quoting. It's safer for Windows subprocess handling because it doesn't go through `cmd.exe` by default (though `shell: true` restores cmd.exe behavior).
- **Path arguments passed to Python**: Always wrap in double quotes: `python -m paperforge status --vault "C:\My Vault\"`
**Warning signs:**
- `exec` returns "The system cannot find the path specified" despite path existing
- Works on one Windows machine but fails on another (encoding/locale difference)
- Python process starts but can't find vault files (wrong working directory)
- Error: `'C:\Users\Lin' is not recognized as an internal or external command` (space in path not quoted)
**Phase to address:**
Phase 2 (Setup wizard / path configuration). Must test on a vault with spaces and Unicode in path.
---
### Pitfall 5: Raw stderr Dumped Into `Notice` — User-Unfriendly Output
**What goes wrong:**
The existing plugin code does:
```javascript
new Notice(`[!!] ${a.cmd} failed: ${(stderr || err.message).slice(0, 120)}`, 8000);
```
This shows raw Python tracebacks, CLI errors, or JSON blobs in the Obsidian notification. The user sees technical gobbledygook like `[!!] sync failed: Traceback (most recent call last):\n File "paperforge", line 12, in <module>\nModuleNotFoundError: No module named 'paperforge'`.
**Why it happens:**
`exec` delivers unfiltered stderr. The naive approach is to show it directly. Python's tracebacks are multi-line, include absolute paths, and are not user-actionable for non-technical users.
**How to avoid:**
- **Parse stderr for known error patterns, return user-friendly messages**:
```javascript
const parseError = (stderr) => {
if (stderr.includes('ModuleNotFoundError'))
return 'PaperForge is not installed. Open Settings and run the Setup Wizard.';
if (stderr.includes('FileNotFoundError') && stderr.includes('library.json'))
return 'No Zotero export found. Check that Better BibTeX is configured.';
if (stderr.includes('ConnectionError') || stderr.includes('timeout'))
return 'Cannot reach PaddleOCR API. Check your internet connection and API key.';
// Fallback: show first line only, stripped of path info
const firstLine = stderr.split('\n').find(l => l.trim() && !l.includes('File "'));
return `Error: ${firstLine || 'Unknown error'}. See console for details.`;
};
```
- **Log full stderr to console**: `console.error('[PaperForge]', stderr)` so developers can still access the raw error.
- **Use `Notice` duration**: Short messages 4000ms, errors 8000ms, success 3000ms.
- **Never show `[!!]` or `[OK]` in user-facing notices**: These are terminal artifacts.
- **Use Obsidian's `createFragment` for rich notices**: `new Notice(createFragment(frag => frag.createEl('b', { text: 'Setup Complete' })))`.
**Warning signs:**
- Notices show full Python tracebacks
- Error messages contain absolute file paths from the developer's machine
- User reports "I got an error but I don't understand what to do"
**Phase to address:**
Phase 2 (Setup wizard error handling). All subprocess calls from settings must produce polished notices.
---
### Pitfall 6: Form State Loss on Tab Switch — `display()` is Called Fresh, Rebuilding Destroys Unsaved Changes
**What goes wrong:**
`SettingTab.display()` is called **every time the user switches to the settings tab**. If `display()` rebuilds the entire form from `this.settings`, any unsaved changes in text fields are lost when the user switches to another tab and comes back. Worse: if the user typed something invalid, the validation state is lost and the field resets to the last saved value.
**Why it happens:**
`SettingTab.hide()` is called when user switches away. `display()` is called when they return. The `containerEl` is managed by Obsidian's settings system — it gets emptied or hidden. The standard pattern is `display()` reads from `plugin.settings` and rebuilds the UI. But if the user typed a new value and hasn't triggered `saveData()` yet (debounce hasn't fired), the in-memory `this.settings` may or may not be up to date depending on whether `onChange` updated it synchronously.
**How to avoid:**
- **Update `this.settings` immediately on change, debounce only the `saveData()` call**:
```javascript
new Setting(containerEl)
.addText(cb => cb
.setValue(this.plugin.settings.someKey)
.onChange(value => {
this.plugin.settings.someKey = value; // immediate in-memory update
this.debouncedSave(); // debounced disk write
}));
```
- **Do NOT read from DOM on save**: Always read from `this.plugin.settings` when saving, not from `component.getValue()`.
- **Clean up in `hide()`**: If you have validation state, persist it before hiding. Obsidian calls `hide()` before removing the tab content.
- **Never reset the form from `display()` without checking if unsaved changes exist**.
**Warning signs:**
- User types a value, switches to "Appearance" tab, comes back — field is empty again
- Validation errors disappear when switching tabs (they shouldn't)
- Settings file shows old value despite user seeing new value in the field
**Phase to address:**
Phase 1 (Settings persistence). This is fundamental to the `PluginSettingTab` lifecycle.
---
### Pitfall 7: `data.json` Missing Keys Crash on First Load (No Migration Path)
**What goes wrong:**
On first load (or after adding a new setting field), `loadData()` returns `null` or `{}`. If the code does `this.settings = await this.loadData()` and then accesses `this.settings.someNewKey` without checking existence, it gets `undefined`. If the code later does `this.settings.someNewKey.toLowerCase()` or similar, it throws a TypeError that crashes the plugin silently (no `Notice`, only console error).
**Why it happens:**
`loadData()` returns `null` when `data.json` doesn't exist (first run). Returns `{}` when file exists but is empty. Returns parsed JSON otherwise. Developers forget to handle the `null` and `undefined` cases for newly added settings keys.
**How to avoid:**
- **Always merge with defaults after load**:
```javascript
const DEFAULT_SETTINGS = {
version: 2,
resourcesDir: '20_Resources',
controlDir: '21_Library',
autoAnalyzeAfterOcr: false,
pythonPath: 'python',
};
async onload() {
const data = await this.loadData();
this.settings = Object.assign({}, DEFAULT_SETTINGS, data || {});
// Migrate from older versions:
if (this.settings.version < 2) {
this.settings.autoAnalyzeAfterOcr = false; // new field
this.settings.version = 2;
await this.saveData(this.settings);
}
}
```
- **Never access `this.settings.X` without default fallback**: Use `this.settings.someKey ?? defaultValue`.
- **Test first-launch scenario**: Delete `data.json`, reload Obsidian, verify settings tab works without crash.
**Warning signs:**
- Plugin silently fails after update (new setting field added without migration)
- `Cannot read property 'toLowerCase' of undefined` in console
- Settings tab blank after first install (data.json missing)
**Phase to address:**
Phase 1 (Settings persistence layer). Write DEFAULT_SETTINGS + merge + migration before any settings controls.
---
### Pitfall 8: Button Double-Click Spawning Multiple Subprocesses
**What goes wrong:**
A "Run Setup" button in settings that uses `onClick` without disabling itself will allow the user to click repeatedly. Each click spawns a new `exec`/`spawn`. Two Python processes now compete for resources, write to the same files, and the second invocation may fail with "file locked" errors.
**Why it happens:**
`ButtonComponent.onClick()` does not auto-disable the button. The developer forgets to add `setDisabled(true)` at the start of the handler and `setDisabled(false)` in the callback. Network latency or slow Python startup means the user sees no immediate feedback and clicks again.
**How to avoid:**
```javascript
new Setting(containerEl)
.setName('Setup')
.addButton(btn => btn
.setButtonText('Run Setup')
.setCta()
.onClick(async (evt) => {
btn.setDisabled(true);
btn.setButtonText('Running...');
try {
const result = await runSetupCommand();
new Notice('Setup complete. Restart Obsidian to apply.', 5000);
} catch (e) {
new Notice(`Setup failed: ${parseError(e.stderr)}`, 8000);
} finally {
btn.setDisabled(false);
btn.setButtonText('Run Setup');
}
}));
```
- **Use a guard flag**: `if (this._setupRunning) return; this._setupRunning = true;` for additional safety.
- **Never keep button enabled during async operations**.
**Warning signs:**
- Two "Python" processes in Task Manager
- "File is locked by another process" errors
- User reports "I clicked it twice and it broke"
**Phase to address:**
Phase 2 (Setup wizard). Every action button must disable itself on click.
---
## Technical Debt Patterns
Shortcuts that seem reasonable but create long-term problems.
| Shortcut | Immediate Benefit | Long-term Cost | When Acceptable |
|----------|-------------------|----------------|-----------------|
| Skip `DEFAULT_SETTINGS` merge — assume `loadData()` always returns full object | Saves 3 lines of code | Plugin crashes on first install; every new setting requires migration logic later | Never |
| Hardcode `exec('python ...')` without configurable Python path | Simple, works for most users | Breaks for users with `python3`, conda envs, or pyenv; requires code change + release to fix | Never — make it a setting from day 1 |
| Use `Notice` for all output, including long success messages | Quick to implement | Long messages clip; no way to show structured output (lists, links); no scroll | MVP only if all messages < 80 chars |
| Skip debounce — save on every `onChange` | Works for low setting count | `data.json` write per keystroke; sync conflicts; disk wear on SSDs over years | Never for text inputs (toggle is ok) |
| Use `setTimeout` for debounce without cleanup | Simple | Memory leak; saveData fires after plugin unload; "Cannot read property of undefined" on closed plugin | Never — always store timer handle and clear |
| Store paths as Windows backslash format in settings | "It works on my machine" | Breaks on macOS/Linux; broken wikilinks; path comparisons fail across OS | Never — always store forward-slash paths |
---
## Integration Gotchas
Common mistakes when connecting to external services.
| Integration | Common Mistake | Correct Approach |
|-------------|----------------|------------------|
| `exec` / `spawn` (Python CLI) | Not quoting `cwd` with spaces on Windows | `spawn` with explicit `{ cwd, shell: true }` — Node handles quoting; test on path with spaces |
| `exec` / `spawn` (Python CLI) | Assuming `python` is the correct binary name | Make Python path a setting (`pythonPath`); default to `python` on Windows, `python3` on macOS/Linux per `process.platform` |
| `exec` / `spawn` (Python CLI) | Using `child_process.exec` which buffers all output (max 1MB) | Use `spawn` for long-running commands (`sync`, `ocr`); use `exec` only for quick commands (`status`, `doctor`) |
| `exec` / `spawn` (Python CLI) | Not handling process.umask or env inheritance | Pass `env: { ...process.env, PYTHONIOENCODING: 'utf-8' }` to force UTF-8 output from Python |
| `Plugin.saveData()` | Calling it outside onload/display contexts where `this.app` may be unavailable | Only call `saveData` in methods with access to `this.app` (settings tab has `this.app` injected); never from standalone functions |
| `Plugin.loadData()` | Not awaiting it | `loadData()` returns `Promise<any>`; must be awaited or `.then()`'d. Synchronous access returns a Promise, not data. |
| Vault adapter `basePath` | Passing it directly to shell commands without sanitization | `basePath` is native OS path; use `path.resolve(basePath)` to normalize, then pass as `cwd` to `spawn` |
| `Notice` message formatting | Using `\n` for multi-line (strips whitespace in Obsidian's notice CSS) | Use `createFragment` for rich text; or keep messages single-line, ~60-80 chars max |
| `Button.onClick` async | Not handling promise rejection | Wrap in try/catch; show Notice on error; always re-enable button in finally block |
---
## Performance Traps
Patterns that work at small scale but fail as usage grows.
| Trap | Symptoms | Prevention | When It Breaks |
|------|----------|------------|----------------|
| Rebuilding entire settings form on every `display()` call | Slow tab switching (500ms+ perceptible lag) when settings tab has 20+ fields | Cache DOM structure; only update values in `display()`, don't recreate; use `Setting.clear()` strategically | 15+ settings fields |
| `saveData()` on every keystroke | Disk I/O spikes; Obsidian Sync uploads; SSDs degrade over years with 100k+ writes/day | Debounce 250-500ms; use `saveLocalStorage` for transient UI state instead of `saveData` | Any text input setting |
| Loading entire `library.json` in settings display | Settings tab freezes for 5+ seconds when Zotero export is large | Never load external data files in settings `display()`; show counts only; use a "View Details" button that opens a separate view | Zotero library > 1000 entries |
| `exec` with 300s timeout for OCR jobs | Settings tab appears frozen; no cancel button; user force-quits Obsidian | Never run long subprocesses from settings UI; dispatch a command that opens the sidebar view with progress tracking instead | Any subprocess > 10 seconds |
| Re-registering event handlers every `display()` call | Memory leak: each tab switch adds new listeners without removing old ones | Use `SettingTab.hide()` to clean up; register events through `this.plugin.registerEvent()` for auto-cleanup on unload | After 10+ tab switches |
---
## Security Mistakes
Domain-specific security issues beyond general web security.
| Mistake | Risk | Prevention |
|---------|------|------------|
| Storing API keys (PaddleOCR) in `data.json` unencrypted | API key readable by any plugin; leaked via Obsidian Sync; exposed in git if vault is versioned | Use `SecretStorage` API (Obsidian v1.11.4+) for API keys: `await this.app.secretStorage.set('paddleocr_key', key)` |
| Passing user-provided paths to `exec` without validation | Command injection if user types `; rm -rf /` or `&& evil_command` in a path setting | Validate paths: `path.resolve(userPath); if (!resolved.startsWith(basePath)) reject;` — use `spawn` with args array, never string interpolation |
| Exposing `data.json` to git via vault versioning | API keys, internal paths, and settings leaked to public repo | Add `.obsidian/plugins/paperforge/data.json` to `.gitignore` (or use SecretStorage for secrets) |
| Running Python subprocess with inherited `PATH` | Malicious `python` binary in PATH earlier than expected; `paperforge` could be a different package | Use absolute path to Python if configured; verify `python -c "import paperforge"` before running commands |
| Settings import/export without validation | Malformed JSON import can crash plugin or execute arbitrary code if eval'd | Validate imported JSON against schema; never `eval()`; use `JSON.parse()` with try/catch; verify all keys match expected schema |
---
## UX Pitfalls
Common user experience mistakes in this domain.
| Pitfall | User Impact | Better Approach |
|---------|-------------|-----------------|
| Showing raw Python errors in Notice | User sees cryptic traceback, doesn't know what action to take | Parse error; map to human-readable message with action hint ("Check your Python installation in Settings > PaperForge") |
| No feedback during long setup operations | User clicks "Run Setup", nothing happens for 30s, assumes plugin is broken | Show "Running..." on button; add a progress bar or spinner; show incremental status messages |
| `Notice` text too long (clips at ~80 chars) | Critical information hidden | Split into multiple short notices or use a modal for detailed output |
| Toggle/checkbox with no immediate save indication | User changes toggle, doesn't know if it's saved | Show brief "Saved" notice after debounce fires; or show a green checkmark icon next to saved settings |
| Settings tab with no "Reset to Defaults" | User makes a mess, can't undo | Add "Reset to Defaults" button per section; confirm with Notice; merge defaults without overwriting API keys |
| Path inputs requiring exact format without browse button | User guesses wrong path format (backslash vs forward slash, trailing slash) | Provide a "Browse" button using Obsidian's file/folder suggest API; auto-normalize path on blur |
| Setup wizard that doesn't check prerequisites | User goes through setup, fails at the end because Python is missing | Check prerequisites FIRST (Python exists, paperforge importable, Zotero configured) before showing the main settings form |
---
## "Looks Done But Isn't" Checklist
Things that appear complete but are missing critical pieces.
- [ ] **Persistence:** Settings survive Obsidian restart — verify by changing a toggle, restarting, checking it persisted
- [ ] **Persistence:** New setting defaults populate on first load — delete `data.json`, reload, verify all fields show defaults
- [ ] **Validation:** Empty required fields show red error text — test empty Python path, empty resources directory
- [ ] **Validation:** Invalid paths are caught before subprocess runs — test with non-existent path, UNC path, path with `..`
- [ ] **Subprocess:** Button disables during execution — rapid double-click test
- [ ] **Subprocess:** Timeout doesn't orphan the Python process — run a command that hangs, check Task Manager after timeout
- [ ] **Sidebar:** Existing sidebar (PaperForgeStatusView) still works after settings tab added — open sidebar, open settings, switch tabs, verify sidebar still renders
- [ ] **Ribbon:** Ribbon icon still opens sidebar after settings tab added — click ribbon, verify sidebar opens
- [ ] **Commands:** All existing commands (`PaperForge: Sync Library`, `PaperForge: Run OCR`) still work — invoke from command palette
- [ ] **Windows path:** Works with vault path containing spaces and Unicode — test with `C:\Users\Test User\My 测试 Vault\`
- [ ] **Notice quality:** Error messages are user-readable, not raw Python tracebacks — trigger each error case, check Notice text
- [ ] **Unload:** Settings tab closes cleanly when plugin is disabled — disable plugin, verify no errors in console, no orphaned processes
- [ ] **Migration:** Upgrading from v1.4.9 (no settings) to v1.5.0 (with settings) doesn't lose data — install old plugin, add data, upgrade, verify
### Pitfall 1: Creating multiple truths for the same paper
**What goes wrong:** `paperforge.json`, plugin `data.json`, library-record frontmatter, OCR `meta.json`, formal notes, and the new asset index all start carrying overlapping state. Users then see contradictory answers to simple questions like “is this paper OCR-ready?” or “which path is canonical?”
---
**Why it happens:** Brownfield systems accrete state by convenience. PaperForge already has proven drift history across queue/status surfaces, and v1.6 adds more derived surfaces unless ownership is locked down first.
## Recovery Strategies
**Consequences:** State drift, repair complexity, mistrust in dashboard output, and future migrations that require special-case logic.
When pitfalls occur despite prevention, how to recover.
| Pitfall | Recovery Cost | Recovery Steps |
|---------|---------------|----------------|
| Corrupted `data.json` | LOW | Delete `data.json`, reload Obsidian — plugin creates fresh defaults. User reconfigures settings (5 fields max). |
| Orphaned Python subprocess | LOW | Kill via Task Manager (Windows) or `pkill -f paperforge` (macOS/Linux). No data loss — just process leak. |
| Settings tab crashes sidebar | MEDIUM | Disable plugin, restart Obsidian, delete `data.json`, re-enable. Sidebar view is re-registered in `onload()`. |
| `loadData()` returns `null` with no merge | LOW | Add `DEFAULT_SETTINGS` merge in `onload()` — no data lost, just settings reset to defaults on next load. |
| Double-click spawning duplicate subprocess | LOW | Kill extra processes. Add button disable pattern — fix is code-only, no data migration needed. |
| Broken path config (spaces, encoding) | MEDIUM | User edits `data.json` manually to fix path. Or reset settings and reconfigure. Provide a "Repair" button in settings that validates all paths. |
| `saveData` without debounce causing sync conflicts | MEDIUM | Delete `data.json` on all synced devices, reconfigure on one device. Add debounce in next release. |
**Prevention:**
- Publish a strict ownership matrix before implementation:
- `paperforge.json` = runtime configuration truth
- plugin `data.json` = UI draft/cache only, never workflow truth
- library-record = user intent + stable imported metadata
- OCR/meta outputs = machine facts
- asset index = derived projection only
- Ban duplicate writable fields across layers.
- Add a `source_of_truth` note to ADR/docs for every new field.
---
**Detection:**
- Same concept appears writable in both plugin and CLI outputs
- “Fix” commands need to choose which file wins
- Status differs depending on command/UI surface
## Pitfall-to-Phase Mapping
**Mitigation phase:** Phase 1
How roadmap phases should address these pitfalls.
| Pitfall | Prevention Phase | Verification |
|---------|------------------|--------------|
| #1: loadData/saveData corruption | Phase 1 (Settings shell) | Delete data.json, restart, verify defaults; toggle setting, restart, verify persistence |
| #2: Settings tab breaks sidebar | Phase 1 (Settings shell) | Open sidebar + settings simultaneously, switch tabs, verify sidebar stays open and renders correctly |
| #3: exec blocks UI | Phase 2 (Setup wizard) | Run setup with slow network; verify button shows "Running..." and doesn't freeze Obsidian |
| #4: Windows path encoding | Phase 1 (Settings shell) | Test with vault path: `C:\Users\Test User\My Vault 测试\` |
| #5: Raw stderr in Notice | Phase 2 (Setup wizard) | Trigger Python not found error; verify Notice shows "PaperForge not installed. Open Settings to configure." |
| #6: Form state loss on tab switch | Phase 1 (Settings shell) | Type in text field, switch to Appearance tab, switch back, verify text preserved |
| #7: Missing keys crash on first load | Phase 1 (Settings shell) | Delete data.json (or fresh install), open settings, verify all controls render without console errors |
| #8: Button double-click spawns duplicates | Phase 2 (Setup wizard) | Rapid click "Run Setup" 5 times; verify only 1 Python process in Task Manager |
### Pitfall 2: Treating the canonical asset index like a hand-maintained database
**What goes wrong:** The new index becomes a semi-manual store that mixes imported facts, repair flags, health summaries, and UI-only annotations. Once users or multiple commands edit it directly, rebuilds stop being trustworthy.
---
**Why it happens:** “Canonical index” sounds like “put everything there.” In local-first systems, durable tables often become junk drawers unless they are clearly defined as projections.
**Consequences:** Non-idempotent rebuilds, hard-to-debug corruption, impossible provenance, and brittle plugin/CLI dependencies.
**Prevention:**
- Make the index a deterministic projection from owned sources, not a primary authoring surface.
- Store per-record provenance: input files, mtimes/hashes, generated_at, schema_version.
- Support `rebuild-index` from source artifacts with no data loss.
- Keep manual overrides in a separate override layer if truly needed.
**Detection:**
- Rebuilding the index changes meaning, not just freshness
- Developers hesitate to delete/regenerate the index
- User edits to the index are considered normal workflow
**Mitigation phase:** Phase 2
### Pitfall 3: Mixing user intent, machine facts, and derived readiness in one state machine
**What goes wrong:** Fields like `analyze`, `do_ocr`, `ocr_status`, deep-reading completion, figure readiness, and AI readiness get collapsed into one monolithic lifecycle enum. A user toggle then accidentally overwrites machine facts, or a failed OCR run blocks unrelated workflows forever.
**Why it happens:** A single “state” field feels tidy, but PaperForge already has different actors: user, worker, agent, and derived health logic.
**Consequences:** Sticky bad states, confusing retries, complicated repair code, and unreadable business logic.
**Prevention:**
- Separate three planes explicitly:
- **Intent:** what the user wants next
- **Asset facts:** what files/processes actually exist
- **Derived readiness/health:** computed conclusions
- Model readiness as computed predicates, not stored toggles.
- Only persist state transitions owned by a real actor.
**Detection:**
- One command both changes user intent and marks derived readiness
- Retry flows require manually editing multiple unrelated fields
- “Why is this blocked?” cannot be answered from one explanation chain
**Mitigation phase:** Phase 1
### Pitfall 4: Re-implementing health and lifecycle logic inside the Obsidian plugin
**What goes wrong:** The plugin starts recomputing health, maturity, and readiness in JavaScript because it needs fast dashboard cards. Soon the Python CLI and plugin disagree about counts, warnings, and next steps.
**Why it happens:** UI teams want instant rendering, and dashboard work invites “just compute it here.” But PaperForge already established a thin-shell direction precisely to avoid this split.
**Consequences:** Plugin/CLI duplication, inconsistent support burden, doubled test surface, and brownfield regressions whenever state rules change.
**Prevention:**
- Expose health/index data from Python as stable JSON contracts.
- Plugin consumes rendered summaries or structured check results; it does not infer workflow truth.
- Put contract fixtures in tests used by both CLI and plugin.
- Refuse UI-only derived fields unless they are clearly cosmetic.
**Detection:**
- Same rule exists in JS and Python
- Dashboard numbers differ from `paperforge status`
- Plugin release requires business-rule changes without CLI changes
**Mitigation phase:** Phase 3
### Pitfall 5: Building maturity scores that look smart but are operationally empty
**What goes wrong:** “Library maturity” becomes a vanity number with no evidence trail. Users see 62/100 but cannot tell what to fix next, or why the score changed after a sync.
**Why it happens:** Product scoring is tempting, especially for AI readiness, but scoring before explainability creates opaque UX.
**Consequences:** User distrust, noisy support requests, and pressure to game the score instead of improving library quality.
**Prevention:**
- Every score must decompose into named checks with weights, evidence, and recommended next action.
- Prefer level-based maturity bands at first (`Foundational`, `Usable`, `AI-ready`) over fake precision.
- Show “what improved / what regressed” deltas after worker runs.
**Detection:**
- Score exists without per-check explanations
- Two libraries with different failure modes get the same score
- Users ask “what does this number mean?”
**Mitigation phase:** Phase 4
### Pitfall 6: Over-productizing extraction schemas into the core asset model
**What goes wrong:** v1.6 turns “AI-ready” into built-in PICO tables, mechanism schemas, parameter schemas, or discipline-specific extraction templates embedded in the canonical index.
**Why it happens:** Once context packaging exists, the fastest demo is a hardcoded schema. But the project explicitly wants a framework, not a stack of medical extraction products.
**Consequences:** Schema lock-in, brittle migrations, domain overfitting, poor reuse outside one workflow, and a permanently bloated index contract.
**Prevention:**
- Keep the core asset model generic: provenance, text assets, figures, notes, readiness, health.
- Implement context packs as pluggable packagers/templates over the core assets.
- Allow optional schema manifests outside the canonical index.
- Treat missing structured extraction as a pack-level capability gap, not a library corruption.
**Detection:**
- Core index fields start naming domain-specific extraction slots
- New workflows require core schema migration instead of a new packager
- “AI-ready” is defined as “matches one extraction template”
**Mitigation phase:** Phase 4
### Pitfall 7: Shipping v1.6 without a brownfield migration and rollback story
**What goes wrong:** Existing vaults upgrade into new index/state logic with partial migrations, stale Bases, old commands, and no clean recovery path.
**Why it happens:** Teams focus on the new model, not on the installed base. PaperForge already has real-world path quirks, metadata drift, and existing plugin persistence.
**Consequences:** Broken dashboards, false health alarms, user-edited records rendered invalid, and high-maintenance emergency fixes.
**Prevention:**
- Version every generated artifact and contract.
- Add `doctor` checks for config drift, index schema version, stale Base templates, and orphaned OCR/meta assets.
- Add one-step repair/rebuild commands before turning on new dashboard surfaces.
- Keep rollout reversible: regenerate index, restore previous Bases, ignore unsupported plugin fields gracefully.
**Detection:**
- Upgrade requires manual file surgery
- Users must delete unknown files to recover
- Docs say “reinstall if weird things happen”
**Mitigation phase:** Phase 5
## Moderate Pitfalls
### Pitfall 8: Health checks that are too shallow to be trusted
**What goes wrong:** “PDF Health” only checks path existence, “OCR Health” only checks folder presence, and “Base Health” only checks file existence. Users get green badges on unusable assets.
**Prevention:**
- Validate at the failure mode level: file exists, readable, non-empty, expected sidecars present, schema/version matches, referenced note exists, figure-map parseable.
- Distinguish `missing`, `broken`, `stale`, and `incomplete`.
- Return evidence paths, not just booleans.
**Mitigation phase:** Phase 3
### Pitfall 9: Making AI context packs too heavy, too eager, or too opaque
**What goes wrong:** `ask-this-paper` or `copy-context-pack` assembles huge prompt blobs every time, includes duplicated OCR/note content, and hides where statements came from.
**Prevention:**
- Build packs from explicit sections with token budgets.
- Prefer manifest-first packaging: summary + citations + selected assets + provenance links.
- Cache pack manifests; assemble full text lazily.
- Include source paths for every included block.
**Mitigation phase:** Phase 4
### Pitfall 10: Dashboard-first implementation order
**What goes wrong:** The plugin dashboard is built before the index and health contracts stabilize, so UI demands freeze bad backend assumptions.
**Prevention:**
- Finish JSON contracts and fixture snapshots before dashboard polish.
- Use a temporary debug/JSON view during backend stabilization.
- Only add cards/recommendations after health semantics are verified.
**Mitigation phase:** Phase 3
### Pitfall 11: Ignoring stale generated surfaces beyond the new index
**What goes wrong:** The index updates, but generated Bases, note templates, cached plugin views, and old dashboard expectations still reflect pre-v1.6 fields.
**Prevention:**
- Track generator version on every generated artifact.
- Add stale-template/Base detection to doctor.
- Make regeneration explicit and safe.
**Mitigation phase:** Phase 5
## Minor Pitfalls
### Pitfall 12: Vocabulary drift across health, readiness, and maturity terms
**What goes wrong:** “ready,” “healthy,” “complete,” “indexed,” and “AI-ready” are used inconsistently across CLI, plugin, docs, and Bases.
**Prevention:**
- Publish a term glossary with exact meanings.
- Reuse the same labels in CLI JSON, plugin cards, docs, and Base columns.
**Mitigation phase:** Phase 1
### Pitfall 13: Hiding actionable repair paths behind aggregate status
**What goes wrong:** Users see “12 broken assets” but not the exact records or command to fix them.
**Prevention:**
- Every aggregate status should drill down to paper-level evidence and a fix path.
- CLI and plugin should share the same remediation text.
**Mitigation phase:** Phase 3
## Phase-Specific Warnings
| Phase Topic | Likely Pitfall | Mitigation |
|-------------|---------------|------------|
| Truth model | Field ownership ambiguity between config, plugin cache, library-record, OCR meta, and index | Publish ownership matrix and forbid duplicate writable fields |
| Canonical index | Index becomes a second database instead of a rebuildable projection | Deterministic projection + provenance + rebuild command |
| Lifecycle model | Intent/facts/readiness collapsed into one enum | Separate planes and compute readiness |
| Health engine | Green badges based on existence-only checks | Failure-mode-level checks with evidence |
| Plugin dashboard | JS duplicates Python business rules | Python emits JSON contracts; plugin renders only |
| Maturity scoring | Opaque score with no next action | Explainable checks, weighted bands, remediation text |
| AI context packaging | Core model polluted by discipline-specific schemas | Keep packagers optional and outside core index |
| Brownfield rollout | Old vaults, Bases, and caches break on upgrade | Versioned artifacts, doctor/repair, reversible rebuild |
## Sources
- **Official Obsidian API types** (`obsidian.d.ts`, retrieved from `https://raw.githubusercontent.com/obsidianmd/obsidian-api/master/obsidian.d.ts`): Plugin, PluginSettingTab, Setting, SettingTab, Notice, ButtonComponent, TextComponent, DropdownComponent, ToggleComponent, DataAdapter, Vault, Workspace, loadData, saveData, addSettingTab. HIGH confidence.
- **Official Obsidian API docs** (`https://docs.obsidian.md/Reference/TypeScript+API/Plugin/loadData`): Confirms `loadData()` returns `Promise<any>`, data stored in `data.json`. HIGH confidence.
- **Existing PaperForge plugin code** (`paperforge/plugin/main.js`): Confirms CommonJS pattern, `exec` usage, `Notice` patterns, `basePath` usage, sidebar `ItemView`, ribbon icon, commands. HIGH confidence.
- **Node.js `child_process` documentation** (`https://nodejs.org/api/child_process.html`): `exec` vs `spawn` behavior, `shell` option on Windows, timeout behavior, quoting requirements. HIGH confidence.
- **Training data knowledge**: Common Obsidian plugin development patterns, `SettingTab.display()` lifecycle, `hide()` behavior, debounce patterns in Electron apps, Windows path handling in Electron. MEDIUM confidence.
---
*Pitfalls research for: Obsidian Plugin Settings Tab (Retrofit to PaperForge CommonJS Plugin)*
*Researched: 2026-04-29*
- `.planning/PROJECT.md` — current v1.6 goals, constraints, and explicit anti-goals. Confidence: HIGH.
- `docs/ARCHITECTURE.md` — worker/agent split, local-first file boundaries, command centralization. Confidence: HIGH.
- `.planning/phases/15-deep-reading-queue-merge/15-LEARNINGS.md` — prior move toward canonical pure data acquisition functions. Confidence: HIGH.
- `.planning/phases/20-plugin-settings-shell-persistence/20-LEARNINGS.md` — proven risk of tracking/implementation drift and plugin thin-shell lessons. Confidence: HIGH.
- Context7 Obsidian developer docs: Plugin settings persistence and ItemView/View registration patterns. Confidence: HIGH.
- https://github.com/obsidianmd/obsidian-developer-docs/blob/main/en/Plugins/User%20interface/Settings.md
- https://github.com/obsidianmd/obsidian-developer-docs/blob/main/en/Plugins/User%20interface/Views.md
- Ink & Switch, “Local-first software: You own your data, in spite of the cloud” — local-first principle that local data is primary and derived surfaces should preserve ownership and rebuildability. Confidence: HIGH.
- https://www.inkandswitch.com/local-first/
- Exa-discovered AI knowledge/RAG articles used only for product-shaping heuristics around explainability, retrieval governance, and avoiding hardcoded schemas. Confidence: LOW.
- https://redwerk.com/blog/rag-best-practices/
- https://oleno.ai/blog/design-a-knowledge-base-that-makes-ai-content-writing-accurate-and-repeatable/

248
.planning/research/STACK.md
View file

@ -1,93 +1,201 @@
# Stack Research — v1.5 Obsidian Plugin Settings Tab
# Technology Stack
**Domain:** Obsidian plugin development (pure JS/CommonJS)
**Researched:** 2026-04-29
**Confidence:** HIGH (verified against existing plugin code and Obsidian API)
**Project:** PaperForge v1.6 literature asset foundation
**Researched:** 2026-05-03
## Stack: Zero New Dependencies
## Recommended Stack
This milestone requires NO new npm packages or tooling. Everything is built into Obsidian's plugin API and Node.js stdlib.
This milestone should stay **Python-first for business logic** and **CommonJS Obsidian-shell only for UI**. The plugin should render the canonical index and invoke CLI commands; it should not recompute lifecycle, health, maturity, or AI-context state on the JavaScript side.
### Obsidian Plugin API (already available)
### Core Framework
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| Python | 3.10+ (existing) | Single owner of config, lifecycle, health, maturity, context-pack generation | Already owns workers/CLI and is the only place that can safely unify business rules without duplicating them in the plugin |
| Pydantic | 2.13.3 | Typed models for `paperforge.json`, canonical asset index, health records, maturity scoring payloads, AI context manifests | Best fit for strict runtime validation plus `model_json_schema()` generation from one source of truth; keeps Python authoritative and produces machine-readable contracts for the plugin/tests |
| Obsidian plugin (`main.js`, CommonJS) | existing thin shell | Dashboard/settings/commands only | Current plugin already shells out to `python -m paperforge status --json`; extend that pattern instead of adding a second domain layer |
| API Class/Method | Purpose | Notes |
|-----------------|---------|-------|
| `PluginSettingTab` | Settings tab base class | `constructor(app, plugin)`, `display()`, `hide()` |
| `Setting` (from `obsidian`) | Form field builder | `.setName()`, `.setDesc()`, `.addText()`, `.addButton()`, `.addToggle()` |
| `Plugin.loadData()` | Load persisted settings | Returns `Promise<object | null>` |
| `Plugin.saveData(data)` | Persist settings to `data.json` | Returns `Promise<void>` |
| `Notice` (from `obsidian`) | Toast notifications | `new Notice(message, duration)`, supports `DocumentFragment` for rich content |
| `SettingTab.containerEl` | DOM container for settings UI | Use `this.containerEl.empty()` then `.createEl()` |
### Database
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| Filesystem JSON snapshot | UTF-8 JSON, schema-versioned | Canonical derived literature asset index | Local-first, Git/backup friendly, easy for Python to emit and plugin to read, no daemon/service required |
| Optional JSONL append log | UTF-8 JSON Lines | Append-only lifecycle/diagnostic history if needed later | Good long-lived audit primitive without committing to SQLite/event sourcing in v1.6 |
### Node.js Built-ins (already available in Electron/Obsidian)
### Infrastructure
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| filelock | 3.29.0 | Cross-process locking around index/context-pack writes | Verified cross-platform, and on Windows uses OS-level locking; prevents sync/OCR/plugin-triggered commands from corrupting shared JSON files |
| `tempfile` + `os.replace` | stdlib | Atomic file writes | Windows-safe write pattern for canonical JSON artifacts; avoids partial writes when commands are interrupted |
| `pathlib`, `hashlib`, `json` | stdlib | Path normalization, content fingerprints, serialization | Already aligned with current codebase and enough for local asset indexing |
| Module | Purpose | Notes |
|--------|---------|-------|
| `node:child_process` (`exec`, `spawn`) | Run Python subprocesses | Already used via `exec` for sidebar actions |
| `node:path` | Cross-platform path handling | `path.sep`, `path.join()`, `path.normalize()` |
### Supporting Libraries
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| jsonschema | 4.26.0 | Validate emitted JSON artifacts against generated schemas in tests and `doctor`-style diagnostics | Use in Python tests/tooling, not as plugin runtime logic |
### What NOT to Add
## File-Format Standards
| Avoid | Why |
|-------|-----|
| TypeScript | Existing plugin is pure JS CommonJS; converting is out of scope |
| Build system (esbuild, webpack) | Plugin ships as single `main.js`; adding build step breaks current release flow |
| `obsidian-daily-notes-interface` or similar wrappers | Not needed for settings tab |
| React/Vue/Svelte | Obsidian settings use vanilla DOM; framework adds unnecessary complexity |
| `electron` APIs directly | Use Obsidian Plugin API wrappers to stay compatible |
### 1. `paperforge.json` stays the human-edited source of truth
- Keep it as the canonical config file.
- Add `schema_version`.
- Validate it in Python with a `PaperForgeConfig` Pydantic model.
- Keep secrets out of it; API keys remain in `.env` / environment.
## Settings Persistence Pattern
Recommended structure for new v1.6 fields:
```js
// DEFAULT settings — merged on every load to prevent crashes on new keys
const DEFAULT_SETTINGS = {
vault_path: '',
system_dir: '99_System',
resources_dir: '20_Resources',
literature_dir: 'Literature',
control_dir: 'Control',
agent_config_dir: '.opencode',
paddleocr_api_key: '',
zotero_data_dir: '',
};
// In Plugin class:
async loadSettings() {
this.settings = Object.assign({}, DEFAULT_SETTINGS, await this.loadData());
}
async saveSettings() {
await this.saveData(this.settings);
```json
{
"schema_version": "1",
"vault_config": { ...existing paths... },
"index": {
"output": "<system_dir>/PaperForge/indexes/library-assets.v1.json"
},
"health": {
"enable_maturity_scoring": true
},
"context": {
"output_dir": "<system_dir>/PaperForge/context",
"default_max_chars": 24000
}
}
```
## Subprocess Pattern for Setup
### 2. Canonical asset index = one derived snapshot file
Recommended path:
```js
const { spawn } = require('node:child_process');
`<system_dir>/PaperForge/indexes/library-assets.v1.json`
// For progress-streaming setup steps (NOT exec — spawn avoids blocking + enables streaming)
function runSetup(settings, onStep, onDone) {
const child = spawn('python', ['-m', 'paperforge', 'setup', '--json'], {
cwd: settings.vault_path,
env: { ...process.env, PADDLEOCR_API_TOKEN: settings.paddleocr_api_key },
});
// ... stream parsing ...
Recommended top-level shape:
```json
{
"schema_version": "1",
"generated_at": "2026-05-03T12:00:00Z",
"paperforge_version": "1.6.x",
"vault": { "...": "..." },
"summary": { "...": "..." },
"assets": [
{
"zotero_key": "ABCDEFG",
"intent": { "analyze": true, "do_ocr": true },
"artifacts": { "pdf": {...}, "ocr": {...}, "formal_note": {...}, "figures": {...} },
"derived": {
"lifecycle_state": "fulltext_ready",
"health": { "library": "healthy", "pdf": "healthy", "ocr": "warning" },
"maturity": { "level": 3, "score": 0.72, "next_steps": ["run_deep_reading"] },
"ai_ready": true
}
}
]
}
```
## Debounce Pattern for SaveData
Rules:
- `library-record` frontmatter remains **user intent**.
- `meta.json`, formal notes, OCR files, figure-map stay **machine artifacts**.
- `library-assets.v1.json` is **derived state only**.
- Plugin reads this file; it does not infer the state itself.
```js
// Prevent data.json corruption from every-keystroke writes
let saveTimeout;
function debouncedSave(plugin) {
clearTimeout(saveTimeout);
saveTimeout = setTimeout(() => plugin.saveSettings(), 500);
}
### 3. Generated schema file
Recommended path:
`<system_dir>/PaperForge/indexes/library-assets.schema.json`
Generate from Pydantic via `model_json_schema()`. This gives:
- a contract for future migrations,
- validation in tests/doctor,
- a stable interface for the plugin without duplicating Python rules.
### 4. AI context pack format
Use a folder, not a database blob:
`<system_dir>/PaperForge/context/<scope>/<id>/`
Contents:
- `manifest.json` — typed metadata, provenance, hashes, included files
- `context.md` — user/LLM-ready packaged text
This is Obsidian-compatible, inspectable, and easy to copy/share locally.
## Implementation Primitives
1. **Python domain models**
- `PaperForgeConfig`
- `AssetIndex`
- `AssetRecord`
- `AssetHealth`
- `LifecycleState`
- `MaturityReport`
- `ContextPackManifest`
2. **One index builder module**
- Example boundary: `paperforge/indexing/`
- Reads library-records, OCR `meta.json`, formal notes, figure-map, exports
- Produces the canonical index snapshot
3. **One health engine**
- Example boundary: `paperforge/health/`
- Pure functions: PDF health, OCR health, Base/template health, overall library health
- Reused by `status`, `doctor`, plugin dashboard, context pack generation
4. **One context-pack generator**
- Example boundary: `paperforge/context/`
- Builds `ask-this-paper`, `ask-this-collection`, `copy-context-pack` payloads from canonical index + source artifacts
5. **One write discipline**
- `FileLock(...).acquire()` around derived-artifact writes
- write temp file
- `os.replace()` into final path
6. **One plugin integration contract**
- Plugin invokes CLI commands such as `paperforge index refresh`, `paperforge status --json`, `paperforge context pack ...`
- Plugin reads `library-assets.v1.json`
- Plugin never re-implements scoring/health/lifecycle logic
## What Should Stay in Stdlib / Current Stack
| Keep | Why |
|------|-----|
| `json` | Canonical index is not large enough to justify a binary or DB format yet |
| `pathlib` | Existing code is already path-centric and Windows-aware |
| `hashlib` | Enough for PDF/fulltext/context-pack fingerprints and stale-cache detection |
| Existing CLI/worker module pattern | Already proven in v1.2-v1.5 and matches the thin-shell plugin goal |
| Existing Obsidian CommonJS plugin style | No build-system migration needed for this milestone |
## What NOT to Add
| Category | Recommended | Alternative | Why Not |
|----------|-------------|-------------|---------|
| Index storage | JSON snapshot + optional JSONL history | SQLite | Adds migration/query complexity and hides state from the vault filesystem without solving a proven scale bottleneck |
| Config typing | Pydantic model on top of current loader | `pydantic-settings` | Helpful for env-heavy apps, but PaperForge's real source of truth is `paperforge.json`; adding another settings layer is unnecessary in v1.6 |
| Plugin validation | Python-generated schema + version check | AJV / Zod in plugin | Creates a second contract surface in JS and pushes business rules toward the plugin |
| File watching | Explicit refresh after worker commands | watchdog / daemon | Violates the local-simple architecture and is unnecessary for user-triggered workflows |
| Search/index engine | Current filesystem artifacts + canonical JSON | Elasticsearch / LanceDB / vector DB | Overkill for milestone scope; AI packaging needs traceable bundles first, not semantic infra |
| YAML parser | Current narrow frontmatter handling + Python indexer | PyYAML / round-trip YAML libs | Extra dependency and formatting churn risk; users do not need full YAML mutation for v1.6 |
## Installation
```bash
# Core additions
pip install "pydantic>=2.13,<3" "filelock>=3.29,<4"
# Dev / contract validation
pip install -D "jsonschema>=4.26,<5"
```
---
## Integration Notes for Existing Architecture
*Stack research for: PaperForge v1.5 Obsidian Plugin Setup Integration*
*Researched: 2026-04-29*
- Extend `paperforge.config` to return validated Pydantic-backed config objects, but keep current precedence semantics.
- Add a dedicated `index refresh` command that all relevant workers can call after successful mutations.
- Refactor `status --json` to consume the canonical index instead of recounting raw files independently.
- Have the plugin dashboard read index summary fields first, and shell out only for refresh/actions.
- Keep OCR/fulltext/figure outputs where they already live; the new index should reference them, not relocate them.
## Sources
- Pydantic docs via Context7 — typed models, validation, `model_dump()`, `model_json_schema()` — HIGH confidence — https://context7.com/pydantic/pydantic
- Pydantic PyPI JSON — current version `2.13.3` — HIGH confidence — https://pypi.org/pypi/pydantic/json
- filelock docs via Context7 — platform-aware locking, Windows OS-level locking — HIGH confidence — https://context7.com/tox-dev/filelock
- filelock PyPI JSON — current version `3.29.0` — HIGH confidence — https://pypi.org/pypi/filelock/json
- jsonschema docs via Context7 — Draft 2020-12 validator support — HIGH confidence — https://context7.com/python-jsonschema/jsonschema/v4.25.1
- jsonschema PyPI JSON — current version `4.26.0` — HIGH confidence — https://pypi.org/pypi/jsonschema/json
- Existing repo context: `pyproject.toml`, `paperforge/config.py`, `paperforge/worker/status.py`, `paperforge/plugin/main.js` — HIGH confidence

214
.planning/research/SUMMARY.md
View file

@ -1,46 +1,194 @@
# Research Summary — v1.5 Obsidian Plugin Settings Tab
# Project Research Summary
**Synthesized:** 2026-04-29
**Sources:** STACK.md, FEATURES.md, ARCHITECTURE.md, PITFALLS.md
**Project:** PaperForge v1.6 literature asset foundation
**Domain:** 基于 Zotero + Obsidian 的本地优先文献资产管理与 AI 上下文基础设施
**Researched:** 2026-05-03
**Confidence:** HIGH
## Stack Additions
## Executive Summary
**Zero new dependencies.** All work uses Obsidian Plugin API + Node.js stdlib:
- `PluginSettingTab` — settings tab base class
- `Plugin.loadData()` / `Plugin.saveData()` — persistence to `data.json`
- `Setting` — form field builder (`.addText()`, `.addButton()`, etc.)
- `Notice` — toast notifications (`.setName()`, `.setDesc()`)
- `node:child_process.spawn` — non-blocking subprocess execution (upgrade from `exec`)
PaperForge v1.6 不应被做成“又一个提示词按钮集合”，而应被收敛为一个建立在现有 Worker/Agent 双层架构之上的“文献资产底座”。研究结论非常一致：Zotero 继续做书目与附件真相源，Obsidian 继续做知识工作台，Python 继续做业务语义与派生状态的唯一所有者；新增能力的核心不是更多提取功能，而是把每篇文献当前“有什么、缺什么、下一步做什么”稳定地表示出来。
No TypeScript, no build system, no new npm packages.
推荐路径是：围绕现有 `paperforge.json`、`library-records`、OCR 产物、正式文献笔记，演进出一个由 Python 统一生成的规范资产索引，并让 plugin 只做 thin shell 展示与命令触发。这样可以把 lifecycle、health、maturity、AI-ready 等判断从当前分散的命令/界面里收敛到一个可重建的派生读模型中，再在此基础上提供 ask-this-paper、ask-this-collection、copy-context-pack 等通用 AI 入口。
## Feature Table Stakes
最高风险不在“技术做不做得出”，而在 brownfield 演进中制造多重真相源、把 canonical index 用成手工数据库、以及在 plugin 中重复实现 Python 业务规则。v1.6 的里程碑设计必须先锁定字段归属、重建契约与迁移/回滚路径，再做 dashboard、成熟度评分和 AI context packaging；否则表面功能越多，状态漂移和维护成本只会越严重。
1. Settings tab with all setup_wizard.py fields (vault path, 5 dirs, API key, Zotero junction)
2. Settings persist across Obsidian restarts
3. One-click "Install" button runs full setup pipeline
4. Human-readable Chinese notices — never raw terminal output
5. Existing sidebar preserved unchanged
## Key Findings
## Architecture
### Recommended Stack
Two-phase build order:
1. **Phase 1 — Settings Shell + Persistence:** Add `DEFAULT_SETTINGS`, `loadSettings()`/`saveSettings()`, `PaperForgeSettingTab` class, register via `addSettingTab()`. Verify sidebar still works.
2. **Phase 2 — Install Button + Setup UX:** Field validation, `spawn`-based subprocess orchestration, polished notice formatting.
v1.6 的推荐栈是“Python-first + CommonJS thin-shell plugin”，不是引入新前端层，也不是把业务逻辑迁到 JS。核心新增依赖很克制：用 Pydantic 建立 `paperforge.json`、canonical index、health/maturity/context manifest 的强类型契约；用 filelock 与 `tempfile` + `os.replace()` 保证 Windows 友好的跨进程锁与原子写入；用 jsonschema 做测试和 `doctor` 级别的契约校验。
Settings tab is purely additive — zero changes to existing `PaperForgeStatusView` or `ACTIONS[]`.
最重要的栈结论不是“加什么”，而是“不要加什么”：不要上 SQLite、不要上 watchdog/daemon、不要在 plugin 里引入 AJV/Zod 作为第二套契约、不要为 v1.6 引入向量库/搜索引擎。当前 PaperForge 的问题是资产状态不可统一解释，不是检索基础设施不足。
## Watch Out For (Critical Pitfalls)
**Core technologies:**
- **Python 3.10+**：继续作为 config、lifecycle、health、maturity、context-pack 的唯一业务所有者，避免 JS/Python 双实现漂移。
- **Pydantic 2.13.3**：为 `paperforge.json`、`formal-library.json` 演进后的 envelope、context pack manifest 提供单一类型真相与 schema 输出。
- **filelock 3.29.0**：为 index/context pack 写入提供跨进程锁，防止 sync、ocr、plugin 同时写文件造成损坏。
- **`tempfile` + `os.replace()`**：保证 canonical JSON/缓存产物原子落盘，避免中断时出现半写文件。
- **CommonJS Obsidian plugin（现有）**：仅负责 dashboard、settings、命令触发与结果展示，不承担状态推导。
- **jsonschema 4.26.0（测试/诊断）**：用于验证生成的 schema 与导出产物，提升 doctor/CI 的可验证性。
| # | Pitfall | Prevention |
|---|---------|------------|
| 1 | `saveData` on every keystroke corrupts `data.json` | Debounce saves (500ms timeout) |
| 2 | `display()` destroys form state on tab switch | Immediate in-memory update on change, debounce only disk write |
| 3 | Raw Python tracebacks in Notice | Parse stderr, map to friendly messages, log full details to console |
| 4 | Windows paths with spaces/Unicode break `exec` | Use `spawn` with proper quoting; test on `C:\Users\Test User\Test 测试\` |
| 5 | Button double-click spawns duplicate processes | `setDisabled(true)` + `setButtonText('Running...')` at start |
| 6 | `loadData()` returns `null` → TypeError | Always merge: `Object.assign({}, DEFAULTS, data || {})` |
**关键版本/格式要求：**
- `paperforge.json` 增加 `schema_version`
- 继续以文件系统 JSON snapshot 为主，不引入数据库
- canonical index 采用版本化 envelope，而不是裸列表
### Expected Features
研究对“必须做什么”和“不要做什么”划分很清楚。v1.6 的 table stakes 不是炫技式 AI，而是把文献库做成一个可读、可修、可重建、可复用的资产系统；真正的 differentiator 也不是医学专用提取表，而是可追溯的 context pack、统一 health/lifecycle 语义与可执行的 next-step guidance。
**Must have（table stakes）:**
- **Canonical asset index**：统一回答每篇文献有哪些资产、缺哪些资产、当前可用于什么。
- **显式 lifecycle state model**：区分 imported、indexed、pdf_ready、fulltext_ready、deep_read_done、ai_context_ready 等派生状态。
- **Library health surfaces**：能看到 PDF、路径、OCR、note/template/base 的健康状况，并支持聚合到 collection/library。
- **Stable per-asset schema**：统一标识符、路径、provenance、readiness 字段，保证长期可维护。
- **Derived queue / next-step views**：从派生状态给出“下一步该 sync / ocr / repair / pf-deep 什么”，而不是只暴露原始 frontmatter。
- **Idempotent rebuild / refresh**：任何修复后都能安全重建索引和视图，不污染正式笔记。
**Should have（differentiators）:**
- **Ask-this-paper context pack**：把单篇论文打包为可追溯 AI 输入，包含 metadata、fulltext、figures、note links、provenance。
- **Ask-this-collection / copy-context-pack**：把 collection 级资产打包给 NotebookLM 风格、但本地优先的综合工作流。
- **Maturity / workflow level scoring**：用透明、可解释的等级/评分告诉用户这篇论文距离“AI-ready”还有多远。
- **Actionable diagnostics with fix paths**：不仅报错，还明确推荐 `sync`、`ocr`、`repair`、重建 note 或重建 index。
- **Thin-shell plugin dashboard**：基于 canonical index 的产品化界面，而不是第二套业务引擎。
**Defer（v2+ 或保持为可选能力）:**
- 学科专用 extraction outputs（PICO、机制表、参数表等）
- 自动从 worker 触发 deep-reading agent
- 大量 prompt-specific 按钮与“功能爆炸”式 AI surface
- 替代 Zotero 的参考文献管理能力
- 云协作/远程同步
- Litmaps/ResearchRabbit 风格的完整发现图谱产品
**Anti-features（本里程碑明确不应产品化）:**
- 不把每个成功 prompt 升级为核心功能
- 不把 domain-specific extraction schema 烙进 core index
- 不在 plugin 里做第二套 lifecycle/health 逻辑
- 不做黑箱式“AI 自动整理一切”
### Architecture Approach
架构结论非常明确：不要新造第二个 canonical index，也不要重写现有本地文件架构；应当直接演进现有 `<system_dir>/PaperForge/indexes/formal-library.json`，把它从“笔记列表”升级为“版本化 canonical derived read model”。它读取 `paperforge.json` 作为配置真相、`library-record` 作为用户意图真相、OCR 目录作为机器事实真相、正式笔记中的 `## 🔍 精读` 作为 deep-reading 真相，再由 Python 一次性推导 lifecycle/health/maturity/context readiness，供 CLI、plugin、Bases 复用。
**Major components:**
1. **配置真相层（`paperforge.json` / `paperforge.config`）** — 统一路径解析与运行时配置，plugin 只做镜像缓存，不做独立默认值真相源。
2. **资产索引构建层（建议 `asset_index.py`）** — 汇总 library-record、OCR meta、formal note、figure-map 等，生成版本化 `formal-library.json`。
3. **状态/健康推导层（建议 `asset_state.py`）** — 以纯函数形式统一 lifecycle、readiness、health、maturity 与 next-step 规则。
4. **上下文打包层（建议 `context_pack.py`）** — 基于 canonical item 生成 per-paper / per-collection AI context packs。
5. **thin-shell plugin + Base 镜像层** — 只读 canonical index 或 CLI JSON 输出，展示 dashboard、queue、settings 和执行入口。
**架构立场（里程碑必须坚持）：**
- 演进 `formal-library.json`，不要新增并行索引文件
- `library-record` 保存用户意图 + machine mirror，不承载所有真相
- plugin 不决定 OCR 是否完成、paper 是否 AI-ready、health 是否红黄绿
- repair 永远修 source artifacts，再重建 index，不直接修 index
### Critical Pitfalls
1. **多重真相源并存** — 必须先发布字段归属矩阵：`paperforge.json` 管配置，plugin `data.json` 仅作 UI cache，`library-record` 管用户意图，OCR/meta 管机器事实，canonical index 只做派生投影。
2. **把 canonical index 用成手工数据库** — index 必须可删除、可重建、可追溯；任何手工覆写都应放在独立 override 层，而不是直接改 index。
3. **把意图、事实、派生 readiness 混成一个状态机** — `analyze/do_ocr`、OCR 完成、deep-reading 完成、AI-ready 必须分层表示；readiness 应计算得出，不应靠用户手改。
4. **在 Obsidian plugin 里重写 health/lifecycle 逻辑** — 所有 dashboard 数字和 next-step 都应来自 Python 输出的 JSON 契约，否则 CLI/plugin 很快会分叉。
5. **做出“看起来智能、实际上不可解释”的成熟度分数** — 若要评分，必须能拆成检查项、权重、证据与修复建议，优先做 level/band 而非伪精确百分制。
6. **没有 brownfield 迁移与回滚故事就上线 v1.6** — 必须做 schema version、doctor/repair 升级、legacy reader、stale Base/template 检测与可逆重建。
## Implications for Roadmap
基于四份研究，v1.6 最合适的 milestone 结构不是“先做 dashboard 再补后端”，而是“先收敛真相模型，再构建 canonical index，再统一 health/status，最后再把 AI 入口挂上去”。下面的阶段划分最贴合当前 PaperForge 架构与 brownfield 风险控制。
### Phase 1: 真相模型与配置收口
**Rationale:** 这是后续所有工作最强依赖；如果配置、意图、机器事实、派生状态的边界不先锁定，后面每个功能都会放大漂移。
**Delivers:** `paperforge.json` 的 schema_version、`vault_config` 作为规范写入目标、plugin 配置只作镜像缓存、字段 ownership matrix、术语表。
**Addresses:** 单一配置真相、stable per-asset schema 的前置部分、idempotent rebuild 的制度基础。
**Avoids:** 多重真相源、状态机混层、术语漂移。
### Phase 2: Canonical asset index 演进与重建管线
**Rationale:** canonical index 是 table stakes 中最核心的底座，也是 dashboard、queue、maturity、context pack 的共同依赖。
**Delivers:** 演进版 `formal-library.json` envelope、legacy tolerant reader、`asset_index.py`、按 key 增量刷新、原子写入与文件锁。
**Uses:** Pydantic、filelock、`tempfile` + `os.replace()`、现有 sync/ocr/deep-reading/repair 流程。
**Implements:** 从 source artifacts 到 derived read model 的唯一投影层。
**Avoids:** index 被当数据库、非幂等 rebuild、并发写入损坏。
### Phase 3: 统一 lifecycle / health / next-step 引擎
**Rationale:** 没有统一状态与健康规则，queue、doctor、status、plugin 仍会各自解释同一篇论文。
**Delivers:** `asset_state.py`、显式 lifecycle/readiness 模型、PDF/OCR/path/note/template/base 健康检查、evidence-aware diagnostics、镜像回写 `library-record` 的展示字段。
**Addresses:** health surfaces、derived queue views、actionable diagnostics、maturity 的规则基础。
**Avoids:** 浅层 health 检查、聚合状态不指向具体修复路径、CLI/plugin 语义不一致。
### Phase 4: Status / Repair / Plugin / Bases 收敛
**Rationale:** 只有在 canonical contract 稳定后，前端展示与修复入口才能避免锁死错误后端假设。
**Delivers:** `status --json` 读取 canonical summary、`repair` 修源后重建 index、plugin dashboard 只消费 Python JSON、Bases 增加 `asset_state` / `library_health` / `maturity_level` / `next_step` 等列。
**Addresses:** thin-shell dashboard、workflow progression、可操作 queue 与库级总览。
**Avoids:** dashboard-first 反向冻结后端、plugin 重算业务规则、stale generated surfaces 被忽略。
### Phase 5: Maturity guidance 与通用 AI context packs
**Rationale:** 这是真正的 differentiator，但只能建立在索引、健康与 provenance 已可靠之后；否则 AI 入口只会包装不可信资产。
**Delivers:** ask-this-paper、ask-this-collection、copy-context-pack、manifest + context.md、level/score + next-step guidance、token budget 与 provenance 显示。
**Addresses:** AI context entry points、traceable provenance bundle、rule-based maturity guidance。
**Avoids:** 黑箱 AI feature、过重 context pack、学科专用 schema 侵入 core model。
### Phase 6: Brownfield rollout、迁移验证与回滚保障
**Rationale:** v1.6 面向已有 vault 与已有用户，必须把升级安全性当作交付物，而不是上线后补救项。
**Delivers:** doctor/repair 升级、schema/version 检测、旧格式兼容、stale Base/template 检测、可逆 rebuild 流程、发布验证清单。
**Addresses:** 长期可维护性、升级可恢复性、对既有数据的兼容性保障。
**Avoids:** 旧库升级后 dashboard 损坏、用户需要手工删文件恢复、文档只能建议“重装试试”。
### Phase Ordering Rationale
- **先真相模型、后界面层**：因为当前 PaperForge 已经有 worker/plugin 多表面并存，若不先统一字段归属，dashboard 只会把旧分歧可视化。
- **先 canonical index、后 maturity/context**：因为 features 研究明确指出 queue、health、AI pack 都依赖 canonical index 和 stable schema。
- **先 health/repair、后 AI 入口**：因为“AI-ready”必须建立在可解释、可修复、可追溯的资产状态上，而不是包装坏资产。
- **把 plugin 放在 Phase 4 而不是更早**：这是直接响应 architecture 与 pitfalls 的共同结论，避免 UI 反向冻结错误契约。
- **单独留 rollout phase**：因为 v1.6 是 brownfield 演进，不是 greenfield 新产品，迁移安全本身就是 milestone 范围的一部分。
### Research Flags
Phases likely needing deeper research during planning:
- **Phase 5（Maturity + Context Packs）:** 需要进一步约束 token budget、pack 结构、collection 聚合粒度，以及与 `/pf-deep`、NotebookLM 风格工作流的衔接细节。
- **Phase 6（Brownfield rollout）:** 需要针对真实旧 vault 样本验证 legacy `formal-library.json`、旧 Base 模板、旧 plugin data 的兼容策略与回滚步骤。
Phases with standard patterns（可少做专项 research-phase，直接进入需求拆解）:
- **Phase 1（真相模型与配置收口）:** 主要基于现有代码结构与已识别漂移问题，模式清晰。
- **Phase 2（Canonical index）:** 文件投影 + schema version + 原子写入 + 文件锁属于成熟工程模式，研究结论一致。
- **Phase 3（统一 health/status 规则）:** 规则虽需细化，但总体边界已充分明确，重点在实现而非再探索方向。
- **Phase 4（Status/Plugin/Bases 收敛）:** 在 thin-shell 原则已明确的前提下，可按契约驱动开发，无需重新定义产品方向。
## Confidence Assessment
| Area | Confidence | Notes |
|------|------------|-------|
| Stack | HIGH | 以官方文档、PyPI 版本信息与现有仓库结构为主，结论集中且依赖收敛。 |
| Features | HIGH | 直接对齐 `.planning/PROJECT.md`、Zotero/Obsidian 官方产品边界与本项目 local-first 方向。 |
| Architecture | HIGH | 基于现有 `formal-library.json` 路径契约、sync/ocr/status/plugin 真实代码边界，建议非常贴合当前仓库。 |
| Pitfalls | HIGH | 与本项目既有漂移经验、plugin thin-shell 历史教训和 brownfield 风险高度一致，且可操作性强。 |
**Overall confidence:** HIGH
### Gaps to Address
- **`formal-library.json` 现网内容规模与刷新性能阈值**：规划时需要确认全量重建与按 key 增量刷新在真实库规模下的成本，以决定 Phase 2/3 是否同时做 query endpoint。
- **library-record 镜像字段的最小集合**：需要在需求阶段明确哪些字段必须回写 frontmatter 供 Bases 使用，哪些只保留在 canonical index 中，避免 frontmatter 膨胀。
- **context pack 的缓存策略**：需确认 per-paper/per-collection pack 何时惰性生成、何时缓存刷新，以避免 Phase 5 出现过重/过 eager 产物。
- **成熟度评分展示形式**：研究更偏向 level/band，但具体是否保留 numeric score、如何解释 delta，仍需在 milestone 需求中定型。
- **升级覆盖面验证样本**：需要至少准备几类真实旧 vault（路径异常、OCR 残缺、旧 Base 模板、旧 plugin data）作为 Phase 6 验证基线。
## Sources
### Primary (HIGH confidence)
- `STACK.md` — Pydantic、filelock、jsonschema、原子写入策略与 Python-first 栈建议
- `FEATURES.md` — v1.6 table stakes、differentiators、anti-features、依赖关系
- `ARCHITECTURE.md` — `formal-library.json` 演进方案、数据归属边界、模块拆分、阶段顺序
- `PITFALLS.md` — brownfield 风险、迁移约束、health/maturity/plugin 反模式
- 现有仓库代码检查：`paperforge/config.py`、`paperforge/worker/sync.py`、`paperforge/worker/ocr.py`、`paperforge/worker/status.py`、`paperforge/plugin/main.js`
### Secondary (MEDIUM confidence)
- Zotero 官方文档（collections/tags、attachments、duplicates）— 用于确认上游真相源边界与“不替代 Zotero”的产品定位
- NotebookLM、ResearchRabbit、Litmaps、SciSpace 的产品定位信息 — 用于界定 differentiator 与 anti-feature 边界
### Tertiary (LOW confidence)
- AI knowledge/RAG 一般性文章 — 仅作为 context pack explainability 与 retrieval governance 的启发，不作为核心架构依据
---
*Research complete: 2026-04-29*
*Research completed: 2026-05-03*
*Ready for roadmap: yes*