diff --git a/.planning/phases/02-code-review-command/02-REVIEW.md b/.planning/phases/02-code-review-command/02-REVIEW.md new file mode 100644 index 00000000..480ff6f5 --- /dev/null +++ b/.planning/phases/02-code-review-command/02-REVIEW.md @@ -0,0 +1,157 @@ +--- +phase: 02-code-review +reviewed: 2026-05-14T12:00:00Z +depth: deep +files_reviewed: 1 +files_reviewed_list: + - paperforge/memory/schema.py +findings: + critical: 0 + warning: 3 + info: 2 + total: 5 +status: issues_found +--- + +# Phase 02: Code Review Report + +**Reviewed:** 2026-05-14T12:00:00Z +**Depth:** deep +**Files Reviewed:** 1 +**Status:** issues_found + +## Summary + +Reviewed Task 1 implementation: addition of `CREATE_READING_LOG` and `CREATE_PROJECT_LOG` SQL table definitions to `paperforge/memory/schema.py`. The implementation faithfully follows the plan specification with correct SQL syntax, proper registration in `ensure_schema()`, correct `ALL_TABLES` inclusion, and `CURRENT_SCHEMA_VERSION` bumped from 1 to 2. All 5 existing schema unit tests pass. No critical defects were introduced. + +However, three warnings were identified: (1) the foreign key from `reading_log` to `papers` will create a deletion-ordering conflict in `builder.py` once the table is populated (future Task 6 concern), (2) the column name `date` in `project_log` shadows a SQLite built-in function name, and (3) the foreign key column `reading_log.paper_id` lacks an index, which will cause full table scans on lookup queries. + +## Warnings + +### WR-01: Foreign Key Will Block `DELETE FROM papers` During Rebuild + +**File:** `paperforge/memory/schema.py:153` (FK definition), `paperforge/memory/builder.py:84` (DELETE caller) + +**Issue:** The `reading_log` table defines `FOREIGN KEY (paper_id) REFERENCES papers(zotero_key)` without `ON DELETE CASCADE`. During non-schema-change rebuilds in `builder.py`, the code executes `DELETE FROM papers` (line 84) WITHOUT first clearing `reading_log`. Once Task 6 populates `reading_log` from JSONL, this DELETE will fail with an `IntegrityError` because SQLite enforces foreign key constraints on DML operations when `PRAGMA foreign_keys=ON` (which `db.py:34` explicitly enables). + +Currently latent because `reading_log` is never populated (Task 6 has not been implemented). Will surface during the Task 6 rebuild flow. + +**Fix:** + +In `builder.py`, clear `reading_log` (and `project_log`) before deleting `papers`: + +```python +# In builder.py, before the DELETE FROM papers (line 84), add: +conn.execute("DELETE FROM reading_log;") +conn.execute("DELETE FROM project_log;") +``` + +Alternatively, add `ON DELETE CASCADE` to the foreign key definition in `schema.py`: +```python +FOREIGN KEY (paper_id) REFERENCES papers(zotero_key) ON DELETE CASCADE +``` + +The first approach (explicit DELETE) is preferred because it makes the rebuild flow explicit and doesn't silently cascade deletions that could surprise maintainers. + +--- + +### WR-02: Column Name `date` Shadows SQLite Built-in Function + +**File:** `paperforge/memory/schema.py:161` + +**Issue:** The `project_log` table uses `date` as a column name (line 161: `date TEXT NOT NULL`). While syntactically valid in SQLite (which allows function names as unquoted identifiers), `date` is a built-in SQL function. This creates ambiguity when reading queries — `SELECT date FROM project_log` works but `SELECT date(created_at) FROM project_log` is ambiguous. More importantly, if this schema is ever ported to another SQL dialect (PostgreSQL, MySQL), `date` is a reserved word and will require quoting. + +**Fix:** + +Rename the column to `log_date`, `entry_date`, or `recorded_date`: + +```sql +-- In CREATE_PROJECT_LOG: +log_date TEXT NOT NULL, +``` + +Note: This would also require updating the plan's Task 2 (`permanent.py`) and Task 6 (`builder.py`) where the column is referenced. If changing the schema column name is too invasive for this phase, consider adding a comment noting the potential ambiguity. + +--- + +### WR-03: Missing Index on Foreign Key Column `reading_log.paper_id` + +**File:** `paperforge/memory/schema.py:142` + +**Issue:** The `reading_log` table has a foreign key on `paper_id` (line 142), but no index is created for this column. All queries filtering by `paper_id` (e.g., `get_reading_notes_for_paper()` in `permanent.py`, paper-context lookup) will require full table scans. As the reading log grows, this will degrade query performance. + +**Fix:** + +Add an index alongside the existing `EVENT_INDEX_SQL` block (following the established pattern in the file): + +```python +# After EVENT_INDEX_SQL (line 137), add: +READING_LOG_INDEX_SQL = [ + "CREATE INDEX IF NOT EXISTS idx_reading_log_paper ON reading_log(paper_id);", + "CREATE INDEX IF NOT EXISTS idx_reading_log_project ON reading_log(project);", + "CREATE INDEX IF NOT EXISTS idx_reading_log_created ON reading_log(created_at);", +] +``` + +And register in `ensure_schema()` after the `EVENT_INDEX_SQL` loop: + +```python +for idx_sql in READING_LOG_INDEX_SQL: + conn.execute(idx_sql) +``` + +Also add a project-level index for `project_log`: + +```python +PROJECT_LOG_INDEX_SQL = [ + "CREATE INDEX IF NOT EXISTS idx_project_log_project ON project_log(project);", + "CREATE INDEX IF NOT EXISTS idx_project_log_created ON project_log(created_at);", +] +``` + +--- + +## Info + +### IN-01: Test Hardcodes Stale Schema Version + +**File:** `tests/unit/memory/test_schema.py:73` + +**Issue:** The test `test_get_schema_version_returns_stored_value` inserts `schema_version = '1'` and asserts the returned value equals 1. Since `CURRENT_SCHEMA_VERSION` was bumped to 2, the hardcoded '1' no longer matches the current version. While the test is verifying read-back behavior (not version equality), the stale value could confuse future maintainers who assume the test reflects the current schema version. + +**Fix:** + +Either update to a version-independent test or clarify with a comment: + +```python +# Test uses arbitrary version '1' for read-back verification (not tied to CURRENT_SCHEMA_VERSION) +conn.execute( + "INSERT INTO meta (key, value) VALUES ('schema_version', '1')" +) +conn.commit() +assert get_schema_version(conn) == 1 +``` + +--- + +### IN-02: `ALL_TABLES` Ordering — Child Tables After Parent (Latent DDL Risk) + +**File:** `paperforge/memory/schema.py:175` + +**Issue:** The `ALL_TABLES` list orders `papers` (index 1) before its child tables `paper_assets`, `paper_aliases`, `paper_events`, and now `reading_log` (indices 2-7). In `drop_all_tables()`, SQLite with `PRAGMA foreign_keys=ON` will reject `DROP TABLE papers` if any child table contains rows referencing papers. This is currently benign because `drop_all_tables()` is only called when schema versions mismatch — at which point the new `reading_log` table hasn't been populated yet. However, if `drop_all_tables` is ever called on a populated database, the ordering would cause a failure. + +**Fix (if desired — pre-existing, not introduced by this change):** + +Reorder `ALL_TABLES` so child tables (FK-referencing) appear before parent tables (FK-referenced): + +```python +ALL_TABLES = ["paper_fts", "reading_log", "project_log", "paper_events", "paper_aliases", "paper_assets", "papers", "meta"] +``` + +Note: `paper_fts` is a virtual table and cannot have FK constraints, so its position is flexible. + +--- + +_Reviewed: 2026-05-14T12:00:00Z_ +_Reviewer: the agent (gsd-code-reviewer)_ +_Depth: deep_ diff --git a/.planning/phases/permanent-jsonl-review/REVIEW.md b/.planning/phases/permanent-jsonl-review/REVIEW.md new file mode 100644 index 00000000..c5681bb6 --- /dev/null +++ b/.planning/phases/permanent-jsonl-review/REVIEW.md @@ -0,0 +1,183 @@ +--- +phase: permanent-jsonl-review +reviewed: 2026-05-14T00:00:00Z +depth: standard +files_reviewed: 1 +files_reviewed_list: + - paperforge/memory/permanent.py +findings: + critical: 0 + warning: 4 + info: 4 + total: 8 +status: issues_found +--- + +# Phase permanent-jsonl-review: Code Review Report + +**Reviewed:** 2026-05-14 +**Depth:** standard +**Files Reviewed:** 1 +**Status:** issues_found + +## Summary + +Reviewed `paperforge/memory/permanent.py` — a new 154-line JSONL permanent storage module implementing two append-only logs (`reading-log.jsonl` and `project-log.jsonl`) with 10 functions for append, read-all, and filtered-read operations. The file correctly uses `secrets.token_hex` for collision-resistant ID generation, handles write-side OSError, and gracefully skips malformed JSON lines on read. + +Four WARNING-level issues were found, primarily around schema inconsistency with the corresponding SQL tables defined in `paperforge/memory/schema.py` (added in the immediately prior commit 5493e28). The JSONL field names for `project_log` diverge from the SQL column names (`entry_type` vs `type`, `content` vs `title`), the `reading_log` is missing the `verified` field present in SQL, and `append_project_entry` lacks input validation. The module also has no tests and no consumers — it is not imported anywhere in the codebase. + +No BLOCKER issues were found: the code is free of crashes, data loss, and security vulnerabilities. + +## Warnings + +### WR-01: Schema field name mismatch — project_log JSONL vs SQL + +**File:** `paperforge/memory/permanent.py:114-133` +**Issue:** The JSONL `append_project_entry` records use field names that differ from the SQL `project_log` table defined in `paperforge/memory/schema.py` (lines 158-173): + +| JSONL (permanent.py) | SQL (schema.py) | +|-----------------------|-----------------| +| `entry_type` | `type` | +| `content` | `title` | +| *(none)* | `date` (NOT NULL) | + +`append_project_entry` also lacks a `date` field entirely — the SQL schema requires `date TEXT NOT NULL`. The `reading_log` tables match more closely but still diverge in one column (see WR-02). + +These two commits (5493e28 for the SQL tables, 0d1ca3a for JSONL) appear related. Inconsistent field names will cause confusion and potential data corruption when any bridge/migration code is written between the two storage backends. + +**Fix:** Align the JSONL field names with the SQL schema. Either: +1. Rename JSONL fields to match SQL (`entry_type` → `type`, `content` → `title`, add `date` field), or +2. Rename the SQL columns to match JSONL (if the SQL schema has not yet been deployed). + +Example aligned implementation for `append_project_entry`: +```python +record: dict[str, object] = { + "id": entry_id, + "created_at": now, + "project": entry.get("project", ""), + "date": date_str, # ADDED — matches SQL NOT NULL + "type": entry.get("type", ""), # RENAMED from entry_type + "title": entry.get("title", ""), # RENAMED from content + "content": entry.get("content", ""), # KEEP for backward compat, or remove + "status": entry.get("status", ""), + ... +} +``` + +### WR-02: Missing `verified` field in reading_log JSONL entries + +**File:** `paperforge/memory/permanent.py:32-76` +**Issue:** The SQL `reading_log` table (schema.py line 140-155) includes a `verified INTEGER DEFAULT 0` column, but the JSONL `append_reading_note` function (lines 53-65) never emits this field. If data is ever migrated between JSONL and SQL, verification status will be silently dropped. + +**Fix:** Add `verified` to the entry dict with a default of `0` (or make it a parameter): +```python +entry: dict[str, object] = { + ... + "verified": 0, # ADDED — matches SQL schema +} +``` + +### WR-03: Missing input validation on `append_project_entry` + +**File:** `paperforge/memory/permanent.py:114-144` +**Issue:** `append_reading_note` validates that `paper_id` and `excerpt` are non-empty before writing (lines 44-47). However, `append_project_entry` performs zero validation — it accepts an empty dict, producing a record where all fields default to empty strings/lists. By contrast, the parallel SQL `project_log` table enforces `NOT NULL` constraints on `project`, `date`, `type`, and `title`. + +This inconsistency means the JSONL can accumulate garbage rows that would be rejected by the SQL backend. It also silently swallows caller errors where a required field was omitted. + +**Fix:** Add validation for the fields that are semantically required: +```python +def append_project_entry(vault: Path, entry: dict) -> dict: + project = entry.get("project", "") + entry_type = entry.get("entry_type", "") + if not project: + return {"ok": False, "error": "project is required"} + if not entry_type: + return {"ok": False, "error": "entry_type is required"} + # ... rest of function +``` + +### WR-04: No tests for the new module + +**File:** `paperforge/memory/permanent.py` (entire file) +**Issue:** Zero tests exist for this module. The `tests/unit/memory/` directory has 6 test files (test_builder.py, test_context.py, test_query.py, test_refresh.py, test_schema.py, __init__.py) but none cover `permanent.py`. No entry in the integration or chaos test suites either. + +With 10 functions covering two distinct log types, validation logic, error handling, and JSONL line format, the absence of tests means: +- ID uniqueness (`secrets.token_hex`) is untested +- Unicode handling (`ensure_ascii=False`) is untested +- Malformed JSON skipping is untested +- Empty-string validation is untested +- OSError recovery on write is untested +- Filtering (`get_reading_notes_for_paper`, `get_project_entries`) correctness is untested + +**Fix:** Create `tests/unit/memory/test_permanent.py` covering at minimum: +- `append_reading_note` / `append_project_entry` happy path +- Validation rejection (empty paper_id, empty excerpt) +- Round-trip: append then read-all +- Filtered reads (`get_reading_notes_for_paper`, `get_project_entries`) +- Malformed JSONL line skipping +- Unicode content preservation + +## Info + +### IN-01: Inline filepath construction bypasses `get_*_log_path` helpers + +**File:** `paperforge/memory/permanent.py:68, 136` +**Issue:** Both `append_reading_note` and `append_project_entry` construct the filepath inline (`log_dir / "reading-log.jsonl"`) instead of calling `get_reading_log_path(vault)` and `get_project_log_path(vault)` which are defined directly above them. This duplicates the path logic and creates a risk of divergence if the path is ever changed in only one place. + +**Fix:** Replace: +```python +log_dir = _ensure_logs_dir(vault) +filepath = log_dir / "reading-log.jsonl" +``` +with: +```python +filepath = _ensure_logs_dir(vault) / "reading-log.jsonl" +``` +Or preferably, use the dedicated helpers: +```python +filepath = get_reading_log_path(vault) +_ensure_logs_dir(vault) # ensure directory exists +``` + +### IN-02: No module-level docstring + +**File:** `paperforge/memory/permanent.py:1-11` +**Issue:** The file lacks a module docstring. Other modules in the `paperforge/memory/` package describe their purpose and usage. A docstring would help future developers understand that this is a JSONL append-only log (not a queryable database) and why it exists alongside the SQL tables. + +**Fix:** Add a module docstring: +```python +"""Permanent JSONL storage layer for reading-log and project-log. + +Provides append-only JSONL persistence as an append-only archival backend +alongside the query-oriented SQL tables. Each log entry is written as a +single JSON line with a unique, collision-resistant ID generated via +``secrets.token_hex``. + +All write functions return ``{"ok": True/False, ...}`` dicts. +Read functions return ``list[dict]`` with malformed lines silently skipped. +""" +``` + +### IN-03: `append_reading_note` does not validate `section` for emptiness + +**File:** `paperforge/memory/permanent.py:32-76` +**Issue:** `paper_id` and `excerpt` are validated as non-empty (lines 44-47), but `section` — a required positional parameter with no default — is not validated. The SQL `reading_log` schema also marks `section` as `TEXT NOT NULL` without a default. An empty `section` value would pass through silently and create an entry that's difficult to query or filter by section. + +**Fix:** Add a validation check: +```python +if not section: + return {"ok": False, "error": "section is required"} +``` + +### IN-04: Module not exported from `paperforge/memory/__init__.py` + +**File:** `paperforge/memory/__init__.py` +**Issue:** The package `__init__.py` exports `get_connection`, `get_memory_db_path`, `ensure_schema`, and `drop_all_tables` — all SQL/db-related. The new `permanent` module is not re-exported, and no other module in the codebase imports it (`grep` returned zero results for `from paperforge.memory.*permanent`). This means the 154 lines of code are dead code until a consumer is wired in. + +**Fix:** Add the key functions to the package's public API, or document in a follow-up task which command/worker will consume this module. + +--- + +_Reviewed: 2026-05-14_ +_Reviewer: the agent (gsd-code-reviewer)_ +_Depth: standard_ diff --git a/docs/superpowers/plans/2026-05-14-research-memory-core-REVIEW.md b/docs/superpowers/plans/2026-05-14-research-memory-core-REVIEW.md new file mode 100644 index 00000000..c6475fbe --- /dev/null +++ b/docs/superpowers/plans/2026-05-14-research-memory-core-REVIEW.md @@ -0,0 +1,216 @@ +--- +phase: research-memory-core +reviewed: 2026-05-14T12:00:00Z +depth: deep +files_reviewed: 2 +files_reviewed_list: + - docs/superpowers/plans/2026-05-14-research-memory-core.md + - docs/superpowers/plans/2026-05-14-research-memory-core-REVIEW.md +findings: + critical: 0 + warning: 3 + info: 2 + total: 5 +status: issues_found +--- + +# Re-Review: Research Memory Core (post-critical-fix) + +**Reviewed:** 2026-05-14 (second pass) +**Depth:** deep (cross-file analysis against paperforge/core/, paperforge/memory/) +**Prior Review:** `2026-05-14-research-memory-core-REVIEW.md` (5 critical, 6 warning, 4 info) +**Status:** issues_found — 3 new warnings, 0 critical; all 11 prior issues verified fixed + +--- + +## Summary + +The updated plan has successfully addressed all 5 BLOCKER-level (critical) issues and all 6 WARNING-level issues from the prior review. The fixes are substantive: `ErrorCode.INVALID_INPUT` replaced with `VALIDATION_ERROR`, `--json`/`--payload` argument split corrected, hardcoded paths replaced with `paperforge_paths()`, schema version bumped to 2, stale `reading_log_v2.py` reference removed, and all six warning fixes applied (correction-note write path, input validation, JSON error logging, paper_events clearing, methodology placement documented, secure ID generation). + +However, three NEW warnings were discovered — two are consistency gaps where the CR-02 fix was applied incompletely (stale documentation), and one is a data-loss design gap (correction notes lost on rebuild). None are blockers but should be addressed before implementation. + +--- + +## Prior Fix Verification (PASS/FAIL per issue) + +| Issue ID | Description | Status | Evidence | +|----------|-------------|--------|----------| +| **CR-01** | `ErrorCode.INVALID_INPUT` → `VALIDATION_ERROR` | **PASS** | Plan uses `ErrorCode.VALIDATION_ERROR` at lines 901, 908, 924, 936, 957; zero `INVALID_INPUT` references | +| **CR-02** | `--json` rename → `--payload`, boolean `--json` flag | **PASS*** | Code blocks fixed (lines 788-795); see NEW-W01 for doc gaps | +| **CR-03** | Hardcoded paths → `paperforge_paths()` | **PASS** | Both `_render_reading_log_md` (lines 533-534) and `_render_project_log_md` (lines 880-881) use config-derived paths | +| **CR-04** | `CURRENT_SCHEMA_VERSION` bump | **PASS** | Line 148: `CURRENT_SCHEMA_VERSION = 2` with comment | +| **CR-05** | Stale `reading_log_v2.py` removed | **PASS** | "New files" section (lines 61-64) lists only 3 files; zero `reading_log_v2` references | +| **WR-01** | No `correction_note` write function | **PASS** | Task 3 Step 6 (lines 550-588) adds `--correct`/`--correction`/`--reason` CLI args + INSERT logic | +| **WR-02** | No data validation in `append_reading_note` | **PASS** | Lines 230-233: `paper_id` and `excerpt` empty checks with error dict return | +| **WR-03** | Silent JSON decode errors | **PASS** | Lines 277-280, 340-343: `logger.warning()` added in catch blocks | +| **WR-04** | `paper_events` not cleared on rebuild | **PASS** | Task 6 Step 2 line 1078: `conn.execute("DELETE FROM paper_events;")` added | +| **WR-05** | `METHODOLOGY_COMPACT.md` not discoverable | **PASS** | Task 7 (line 1099): documents it's intentionally outside `archive/`, with rationale | +| **WR-06** | Weak UUID truncation | **PASS** | Lines 237, 304: `secrets.token_hex(4)` replaces `str(uuid.uuid4())[:6]` | + +### CR-02 Detail (* qualified PASS) + +The code blocks in Task 5 are correctly fixed — `--payload` accepts the JSON string and `--json` is a proper `store_true` boolean. However, two documentation sections were NOT updated (see NEW-W01 below). The code is correct; the surrounding specification text is stale. + +--- + +## New Issues + +### NEW-W01: Interface contract and smoke test retain old `--json ''` syntax + +**File:** `docs/superpowers/plans/2026-05-14-research-memory-core.md` +**Lines:** 46, 1170 +**Severity:** WARNING +**Issue:** The CR-02 fix renamed the payload argument from `--json` to `--payload` in the Task 5 code blocks, but two documentation sections still reference the old syntax: + +1. **Line 46** — Interface contract section: + ``` + paperforge project-log --write --json '' --vault $VAULT + ``` + +2. **Line 1170** — Smoke test command: + ```bash + --json '{"date":"2026-05-14","type":"note","title":"test"}' \ + ``` + +An implementer following the interface contract or smoke test would use `--json '<...>'` which now means "output as JSON" (boolean flag), not "JSON payload". The command would either error (boolean flag receiving a string value) or behave unexpectedly. + +**Fix:** +```markdown +# Line 46: Change to: +paperforge project-log --write --payload '' --vault $VAULT + +# Line 1170: Change to: + --payload '{"date":"2026-05-14","type":"note","title":"test"}' \ +``` + +--- + +### NEW-W02: `correction_note` data has no JSONL backing — permanently lost on DB rebuild + +**File:** `docs/superpowers/plans/2026-05-14-research-memory-core.md` +**Lines:** 552-581 (correction write path), 1078 (DELETE FROM paper_events) +**Severity:** WARNING +**Issue:** Task 3 Step 6 writes `correction_note` events directly to `paper_events` (line 573-576). Task 6 clears `paper_events` on rebuild (`DELETE FROM paper_events;`, line 1078). However, unlike reading notes (which are dual-written to JSONL + paper_events), correction notes have **no JSONL equivalent** — they exist only in `paper_events`. On rebuild, all correction history is permanently lost. + +Furthermore, the existing `drop_all_tables(conn)` in `builder.py:79` already drops `paper_events` when the schema version changes (since `paper_events` is in `ALL_TABLES`). The plan's added `DELETE FROM paper_events` handles same-version rebuilds, making the data-loss path even broader. + +**Fix (Option A — preferred):** Mirror the dual-write pattern used for reading notes. When a correction is written, also append to a `corrections.jsonl` file. Add an `_import_corrections()` function in `builder.py` that restores them on rebuild. + +**Fix (Option B — simpler):** Scope the `DELETE FROM paper_events` to exclude `correction_note` rows: +```python +conn.execute("DELETE FROM paper_events WHERE event_type != 'correction_note';") +``` + +**Fix (Option C):** Document that correction notes are ephemeral and lost on rebuild, and ensure the `paper-context` command's `recheck_targets` warning makes this acceptable. + +--- + +### NEW-W03: `append_reading_note()` failure silently ignored — PFResult always reports success + +**File:** `docs/superpowers/plans/2026-05-14-research-memory-core.md` +**Lines:** 432-466 (Task 3 Step 3) +**Severity:** WARNING +**Issue:** In the `reading_log.py` write section, the code calls `append_reading_note()` which returns `{ok, id, path}` or `{ok: False, error: str}`. However, the surrounding logic hardcodes `ok = True` (line 460) without checking the return value: + +```python +result = append_reading_note(vault, args.paper_id, ...) +# ... dual-write to paper_events ... + +ok = True # Always True — ignores result["ok"] +result_obj = PFResult( + ok=ok, + command="reading-log", + version=PF_VERSION, + data={"written": ok, "id": result.get("id")}, # "id" is None on failure +) +``` + +If JSONL write fails (disk full, permission error), `result["ok"]` is `False`, `result["id"]` is `None`, but the PFResult contract reports `{ok: true, data: {written: true, id: null}}`. The CLI consumer has no way to detect the failure. + +**Fix:** +```python +result = append_reading_note(vault, args.paper_id, ...) +if not result.get("ok"): + result_obj = PFResult( + ok=False, command="reading-log", version=PF_VERSION, + error=PFError(code=ErrorCode.INTERNAL_ERROR, + message=result.get("error", "Failed to write reading note")), + ) + # ... output and return +# Continue with paper_events write only if JSONL succeeded +write_reading_note(...) +``` + +--- + +## Info (Non-blocking observations) + +### NEW-IN01: Duplicate step numbering in Task 3 + +**File:** `docs/superpowers/plans/2026-05-14-research-memory-core.md` +**Lines:** 378, 389, 412, 470, 550, 590, 607 +**Severity:** INFO +**Issue:** Task 3 step numbering jumps: Step 1 (l.378), Step 2 (l.389), Step 3 (l.412), Step 4 (l.470), Step 6 (l.550), Step 7 (l.590), Step 6 (l.607, again). There is no Step 5, and two different sections are both labeled "Step 6". This could confuse implementers tracking progress. +**Fix:** Renumber sequentially: Step 5 for correction write support (l.550), Step 6 for render dispatch (l.590), Step 7 for commit (l.607). + +### NEW-IN02: Misleading error code in `paper_context.py` + +**File:** `docs/superpowers/plans/2026-05-14-research-memory-core.md` +**Line:** 736 +**Severity:** INFO +**Issue:** The `paper_context.py` `run()` function uses `ErrorCode.PATH_NOT_FOUND` when no paper is found in the database for a given zotero_key. `PATH_NOT_FOUND` (defined in `errors.py:24` under "Config / Vault") is semantically meant for filesystem or vault path lookup failures, not database record lookups. Using the wrong error code category could mislead downstream consumers (plugin UI generating fix-it guidance for a "missing path" when the real issue is a missing paper key). +**Fix:** Use `ErrorCode.VALIDATION_ERROR` or add a purpose-specific code like `PAPER_NOT_FOUND`: +```python +error=PFError(code=ErrorCode.VALIDATION_ERROR, + message=f"No paper found for key: {key}") +``` + +--- + +## Task-by-Task Re-Assessment + +| Task | Prior | Current | Notes | +|------|-------|---------|-------| +| 1: DB Schema | PASS* | **PASS** | Schema version bumped, new tables registered | +| 2: JSONL Storage | PASS* | **PASS** | Validation added, JSON errors logged, secure IDs | +| 3: Reading-Log CLI | PASS* | **PASS** | All prior issues fixed; NEW-W03 (unchecked result) to address | +| 4: Paper-Context CLI | PASS* | **PASS** | Correction read path now has matching write path; NEW-IN02 minor | +| 5: Project-Log CLI | **FAIL** → **PASS** | CR-01/CR-02/CR-03 all fixed; NEW-W01 (stale spec docs) to address | +| 6: DB Builder | PASS* | **PASS** | paper_events cleared; NEW-W02 (correction loss on rebuild) to address | +| 7: METHODOLOGY_COMPACT | PASS* | **PASS** | Placement documented as intentional | +| 8: Integration | PASS | **PASS** | Smoke test needs --payload fix (NEW-W01) | + +--- + +## Cross-File Consistency Check + +| Check | Result | +|-------|--------| +| `paperforge_paths()` used consistently across render functions | PASS | +| `ErrorCode` values all exist in `errors.py` enum | PASS (verified PATH_NOT_FOUND, VALIDATION_ERROR both exist) | +| `ALL_TABLES` includes `reading_log` and `project_log` | PASS (line 142) | +| `ensure_schema()` creates both new tables | PASS (lines 135-136) | +| `builder.py` delete/add order correct (DELETE before INSERT) | PASS | +| Interface contract matches CLI implementation | FAIL — line 46 uses `--json` not `--payload` | +| Smoke test commands match CLI implementation | FAIL — line 1170 uses `--json` not `--payload` | +| `paper_events` lifecycle documented | PARTIAL — cleared on rebuild but correction notes not restored | + +--- + +## Dependency Graph: Still Verified + +``` +Task 2 (JSONL) → Tasks 3, 4, 5, 6 +Task 7 is independent +Task 8 is integration (depends on all) +``` + +No circular dependencies introduced. + +--- + +_Reviewed: 2026-05-14_ +_Reviewer: the agent (gsd-code-reviewer)_ +_Depth: deep_ +_Prior fixes verified: 11/11 PASS (CR-01 through CR-05, WR-01 through WR-06)_ +_New issues found: 3 warnings, 2 info_ diff --git a/docs/superpowers/plans/2026-05-14-research-memory-core.md b/docs/superpowers/plans/2026-05-14-research-memory-core.md new file mode 100644 index 00000000..b85de8c0 --- /dev/null +++ b/docs/superpowers/plans/2026-05-14-research-memory-core.md @@ -0,0 +1,1206 @@ +# Research Memory Core — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the permanent JSONL-based research memory foundation: paper-context (reading-log safety loop), reading-log/project-log JSONL storage + rendering, and methodology compact injection. + +**Architecture:** Permanent JSONL files (`logs/reading-log.jsonl`, `logs/project-log.jsonl`) serve as source of truth. `paperforge.db` imports from them on rebuild (reading_log and project_log tables). Markdown rendering is a side effect. A new `paperforge paper-context ` command combines paper metadata + prior reading notes + corrections into one call. A new `paperforge project-log` command handles project-level work logging. + +**Tech Stack:** Python 3.11+, SQLite (paperforge.db), JSONL files, PFResult contract + +--- + +## Current State (pre-plan) + +### What exists +- `paperforge.db` with `papers`, `paper_fts`, `paper_events` tables +- `memory/events.py`: `write_reading_note()` and `export_reading_log()` — both write/read from `paper_events` table in DB (not permanent JSON) +- `commands/reading_log.py`: full CLI for reading-log (write/validate/import/lookup/export), 317 lines +- `commands/agent_context.py`: returns library overview + collections + commands + rules +- `commands/search.py`: FTS5 search with filters +- `skills/paperforge/` (newly restructured): 6 workflow files that reference new CLI commands that don't exist yet +- `memory/schema.py`: `paper_events` table, schema version 1 + +### What's broken / missing +1. **P0 (resolved)**: pf_search.py deleted, skill_deploy.py fixed, ld_deep renamed to pf_deep +2. **P0 (remaining)**: None — P0 bugs resolved by skill restructure +3. **paper-context CLI**: Does not exist. Is the key safety-loop command. +4. **reading-log stores in DB only**: `write_reading_note()` writes to `paper_events` table. DB rebuild nukes all reading notes. No permanent JSONL storage. +5. **project-log CLI**: Does not exist at all. +6. **METHODOLOGY_COMPACT.md**: Does not exist. +7. **reading-log and project-log rendering**: Does not exist. + +### What the workflow files expect (interface contract) +The workflow files in `skills/paperforge/workflows/` reference these CLI atoms that must exist: + +``` +paperforge paper-context --json --vault $VAULT + → {ok, data: {paper: {...}, prior_notes: [...], corrections: [...]}} + +paperforge reading-log --write --section "..." --excerpt "..." --context "..." --usage "..." --note "..." --project "..." --tags "..." --vault $VAULT + → {ok, data: {written: true}} + +paperforge reading-log --render --project

--vault $VAULT + → renders Project/

/reading-log.md + +paperforge project-log --write --project

--payload '' --vault $VAULT + → {ok, data: {written: true, id: "plog_..."}} + +paperforge project-log --render --project

--vault $VAULT + → renders Project/

/project-log.md + +paperforge project-log --list --json --vault $VAULT + → {ok, data: {entries: [...]}} +``` + +--- + +## File Structure + +### New files +``` +paperforge/memory/permanent.py ← JSONL read/write for reading-log and project-log +paperforge/commands/project_log.py ← New project-log CLI +paperforge/commands/paper_context.py ← New paper-context command +``` + +### Modified files +``` +paperforge/cli.py:288-300 ← Update reading-log subparser, add project-log + paper-context +paperforge/commands/reading_log.py ← Add --context + --tags + --project + --render + JSONL write +paperforge/memory/events.py:9-35 ← Add context/tags/project fields to write_reading_note +paperforge/memory/schema.py ← Add reading_log + project_log tables + correction_note event_type +paperforge/memory/builder.py ← Rebuild: import JSONL → DB +paperforge/skills/paperforge/SKILL.md ← Update CLI references if needed +``` + +### Deleted files +(none) + +--- + +## Task 1: DB Schema — Add reading_log and project_log tables + +**Files:** +- Modify: `paperforge/memory/schema.py` + +SQLite tables for import from JSONL. These are derived tables, rebuilt from JSONL on `memory build`. + +- [ ] **Step 1: Add CREATE statements** + +In `schema.py`, after `EVENT_INDEX_SQL` (line 137), add: + +```python +CREATE_READING_LOG = """ +CREATE TABLE IF NOT EXISTS reading_log ( + id TEXT PRIMARY KEY, + paper_id TEXT NOT NULL, + project TEXT DEFAULT '', + section TEXT NOT NULL, + excerpt TEXT NOT NULL, + context TEXT DEFAULT '', + usage TEXT NOT NULL, + note TEXT DEFAULT '', + tags_json TEXT DEFAULT '[]', + created_at TEXT NOT NULL, + agent TEXT DEFAULT '', + verified INTEGER DEFAULT 0, + FOREIGN KEY (paper_id) REFERENCES papers(zotero_key) +); +""" + +CREATE_PROJECT_LOG = """ +CREATE TABLE IF NOT EXISTS project_log ( + id TEXT PRIMARY KEY, + project TEXT NOT NULL, + date TEXT NOT NULL, + type TEXT NOT NULL, + title TEXT NOT NULL, + decisions_json TEXT DEFAULT '[]', + detours_json TEXT DEFAULT '[]', + reusable_json TEXT DEFAULT '[]', + todos_json TEXT DEFAULT '[]', + related_papers_json TEXT DEFAULT '[]', + tags_json TEXT DEFAULT '[]', + created_at TEXT NOT NULL, + agent TEXT DEFAULT '' +); +""" +``` + +- [ ] **Step 2: Register in ensure_schema()** + +In `ensure_schema()`, add after `conn.execute(CREATE_EVENTS)`: +```python +conn.execute(CREATE_READING_LOG) +conn.execute(CREATE_PROJECT_LOG) +``` + +- [ ] **Step 3: Add to ALL_TABLES** + +```python +ALL_TABLES = ["paper_fts", "papers", "paper_assets", "paper_aliases", "meta", "paper_events", "reading_log", "project_log"] +``` + +- [ ] **Step 4: Bump CURRENT_SCHEMA_VERSION** + +```python +CURRENT_SCHEMA_VERSION = 2 # Bump from 1 for reading_log + project_log tables +``` +This ensures existing vaults rebuild to get the new tables. + +- [ ] **Step 5: Verify schema compiles** + +Run: `python -c "from paperforge.memory.schema import ensure_schema, ALL_TABLES; print(ALL_TABLES)"` +Expected: List includes `reading_log` and `project_log` + +- [ ] **Step 6: Commit** + +```bash +git add paperforge/memory/schema.py +git commit -m "feat: add reading_log and project_log tables to memory schema" +``` + +--- + +## Task 2: JSONL Permanent Storage Layer + +**Files:** +- Create: `paperforge/memory/permanent.py` + +New module for reading from and appending to JSONL files. No dependencies on DB. + +- [ ] **Step 1: Write failing test** + +Create file with tests (tests will be added later). For now, just write the module. + +- [ ] **Step 2: Write permanent.py** + +```python +"""Permanent JSONL storage for reading-log and project-log. + +JSONL = one JSON object per line. Append-only. Source of truth. +Never deleted by DB rebuild. +""" + +from __future__ import annotations + +import json +import datetime +import logging +from pathlib import Path + +logger = logging.getLogger(__name__) + + +def _logs_dir(vault: Path) -> Path: + from paperforge.config import paperforge_paths + paths = paperforge_paths(vault) + return paths["paperforge"] / "logs" + + +def _ensure_logs_dir(vault: Path) -> Path: + d = _logs_dir(vault) + d.mkdir(parents=True, exist_ok=True) + return d + + +# ── Reading Log ───────────────────────────────────────────── + +def get_reading_log_path(vault: Path) -> Path: + return _logs_dir(vault) / "reading-log.jsonl" + + +def append_reading_note( + vault: Path, + paper_id: str, + section: str, + excerpt: str, + usage: str, + context: str = "", + note: str = "", + project: str = "", + tags: list[str] | None = None, + agent: str = "", +) -> dict: + """Append a single reading note to reading-log.jsonl. + + Returns dict with {id, path, ok} or {ok: false, error: str}. + """ + if not paper_id or not paper_id.strip(): + return {"ok": False, "error": "paper_id is required"} + if not excerpt or not excerpt.strip(): + return {"ok": False, "error": "excerpt is required"} + + import secrets + date_str = datetime.datetime.now().strftime("%Y%m%d") + seq = secrets.token_hex(4) # 8 hex chars + entry_id = f"rln_{date_str}_{seq}" + now_iso = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + entry = { + "id": entry_id, + "paper_id": paper_id, + "project": project, + "section": section, + "excerpt": excerpt, + "context": context, + "usage": usage, + "note": note, + "tags": tags or [], + "created_at": now_iso, + "agent": agent, + "verified": False, + } + + filepath = get_reading_log_path(vault) + _ensure_logs_dir(vault) + with open(filepath, "a", encoding="utf-8") as f: + f.write(json.dumps(entry, ensure_ascii=False) + "\n") + + return {"ok": True, "id": entry_id, "path": str(filepath)} + + +def read_all_reading_notes(vault: Path) -> list[dict]: + """Read all reading notes from reading-log.jsonl.""" + filepath = get_reading_log_path(vault) + if not filepath.exists(): + return [] + entries = [] + with open(filepath, encoding="utf-8") as f: + for line in f: + line = line.strip() + if line: + try: + entries.append(json.loads(line)) + except json.JSONDecodeError: + logger.warning( + "Skipping unparseable line in %s: %s", filepath, line[:100] + ) + continue + return entries + + +def get_reading_notes_for_paper(vault: Path, paper_id: str) -> list[dict]: + """Get all reading notes for a specific paper.""" + all_notes = read_all_reading_notes(vault) + return [n for n in all_notes if n.get("paper_id") == paper_id] + + +# ── Project Log ────────────────────────────────────────────── + +def get_project_log_path(vault: Path) -> Path: + return _logs_dir(vault) / "project-log.jsonl" + + +def append_project_entry(vault: Path, entry: dict) -> dict: + """Append a project log entry to project-log.jsonl. + + entry must have: project, date, type, title. + Auto-generates id and created_at if missing. + """ + import secrets + date_str = entry.get("date", datetime.datetime.now().strftime("%Y-%m-%d")) + seq = secrets.token_hex(4) # 8 hex chars + entry_id = f"plog_{date_str}_{seq}" + now_iso = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + entry["id"] = entry.get("id", entry_id) + entry["created_at"] = entry.get("created_at", now_iso) + # Ensure JSON-serializable fields + entry.setdefault("decisions", []) + entry.setdefault("detours", []) + entry.setdefault("reusable", []) + entry.setdefault("todos", []) + entry.setdefault("related_papers", []) + entry.setdefault("tags", []) + entry.setdefault("agent", "") + + filepath = get_project_log_path(vault) + _ensure_logs_dir(vault) + with open(filepath, "a", encoding="utf-8") as f: + f.write(json.dumps(entry, ensure_ascii=False) + "\n") + + return {"ok": True, "id": entry["id"], "path": str(filepath)} + + +def read_all_project_entries(vault: Path) -> list[dict]: + """Read all project log entries.""" + filepath = get_project_log_path(vault) + if not filepath.exists(): + return [] + entries = [] + with open(filepath, encoding="utf-8") as f: + for line in f: + line = line.strip() + if line: + try: + entries.append(json.loads(line)) + except json.JSONDecodeError: + logger.warning( + "Skipping unparseable line in %s: %s", filepath, line[:100] + ) + continue + return entries + + +def get_project_entries(vault: Path, project: str) -> list[dict]: + """Get entries for a specific project.""" + all_entries = read_all_project_entries(vault) + return [e for e in all_entries if e.get("project") == project] +``` + +- [ ] **Step 3: Verify imports work** + +Run: `python -c "from paperforge.memory.permanent import append_reading_note; print('OK')"` + +- [ ] **Step 4: Commit** + +```bash +git add paperforge/memory/permanent.py +git commit -m "feat: add permanent JSONL storage layer for reading-log and project-log" +``` + +--- + +## Task 3: Reading-Log CLI Upgrade — JSONL + context/tags/project + render + +**Files:** +- Modify: `paperforge/commands/reading_log.py` +- Modify: `paperforge/cli.py:288-300` +- Modify: `paperforge/memory/events.py` (add context/tags fields to write_reading_note) + +Update the existing `reading-log` CLI to: +1. Write to JSONL (permanent) AND paper_events (for FTS search in DB) +2. Support new fields: `--context`, `--tags`, `--project` +3. Add `--render` subcommand to generate markdown per project + +- [ ] **Step 1: Add new CLI arguments** + +In `cli.py`, find the `reading-log` subparser (line 288) and add: + +```python +p_rl.add_argument("--context", help="Full paragraph containing excerpt") +p_rl.add_argument("--tags", help="Comma-separated tags") +p_rl.add_argument("--project", help="Associated project name") +p_rl.add_argument("--render", action="store_true", help="Render reading-log.md for one or all projects") +``` + +- [ ] **Step 2: Update write_reading_note() in events.py** + +Add optional parameters: `context`, `project`, `tags`. Update payload dict to include them. + +In `paperforge/memory/events.py`, modify `write_reading_note()`: + +```python +def write_reading_note(vault: Path, paper_id: str, section: str, + excerpt: str, usage: str = "", note: str = "", + context: str = "", project: str = "", + tags: list[str] | None = None) -> bool: + payload = { + "section": section, + "excerpt": excerpt, + "context": context, + "usage": usage, + "note": note, + "project": project, + "tags": tags or [], + } + # ... rest unchanged +``` + +- [ ] **Step 3: Add JSONL write + render logic to reading_log.py** + +In `commands/reading_log.py`, update the `run()` function: + +```python +# Add import at top: +from paperforge.memory.permanent import ( + append_reading_note, + get_reading_notes_for_paper, + read_all_reading_notes, +) + +# In run(), update the write section (around line 268): +if args.paper_id and args.excerpt: + # 1. Write to permanent JSONL (source of truth) + import json as _json + tags = [] + if getattr(args, "tags", ""): + tags = [t.strip() for t in args.tags.split(",") if t.strip()] + + result = append_reading_note( + vault, + args.paper_id, + getattr(args, "section", "") or "", + args.excerpt, + getattr(args, "usage", "") or "", + getattr(args, "context", "") or "", + getattr(args, "note", "") or "", + getattr(args, "project", "") or "", + tags, + ) + +# 3. Also write to paper_events for FTS in DB + db_ok = write_reading_note( + vault, args.paper_id, + getattr(args, "section", "") or "", + args.excerpt, + getattr(args, "usage", "") or "", + getattr(args, "note", "") or "", + getattr(args, "context", "") or "", + getattr(args, "project", "") or "", + tags, + ) + + # 4. Auto-render if project specified + if getattr(args, "project", ""): + _render_reading_log_md(vault, args.project) + + ok = result.get("ok", False) + result_obj = PFResult( + ok=ok, + command="reading-log", + version=PF_VERSION, + data={"written": ok, "id": result.get("id")}, + ) + # ... output logic +``` + +- [ ] **Step 4: Add render function + correction write path** + +Add to `reading_log.py`: + +```python +# Add import at top of file: +from paperforge.memory.permanent import ( + append_reading_note, + get_reading_notes_for_paper, + read_all_reading_notes, +) + +def _render_reading_log_md(vault: Path, project: str = "") -> None: + """Render reading-log.md for a specific project from JSONL.""" + all_notes = read_all_reading_notes(vault) + + # Filter by project if specified + if project: + notes = [n for n in all_notes if n.get("project") == project] + else: + notes = all_notes + + if not notes: + return + + # Build markdown + lines = ["# Reading Log", ""] + if project: + lines[0] = f"# Reading Log — {project}" + lines.append("> Auto-generated from reading-log.jsonl. Do not edit manually.") + lines.append("") + + # Group by paper_id + from collections import defaultdict + by_paper = defaultdict(list) + for n in notes: + by_paper[n["paper_id"]].append(n) + + for paper_id, paper_notes in sorted(by_paper.items()): + first = paper_notes[0] + lines.append(f"## {paper_id}") + if first.get("project"): + lines.append(f"**Project:** {first['project']}") + lines.append("") + + for entry in sorted(paper_notes, key=lambda x: x.get("created_at", "")): + lines.append(f"### {entry.get('section', '(no section)')}") + lines.append(f"**Info:** \"{entry['excerpt']}\"") + if entry.get("context"): + lines.append("") + lines.append(f"> {entry['context']}") + lines.append(f"**Use:** {entry.get('usage', '')}") + if entry.get("note"): + lines.append(f"**Note:** {entry['note']}") + if entry.get("tags"): + lines.append(f"**Tags:** {', '.join(entry['tags'])}") + if entry.get("verified"): + lines.append("**Verified:** yes") + lines.append("") + lines.append("---") + lines.append("") + + # Write to project directory or general location + from paperforge.config import paperforge_paths + paths = paperforge_paths(vault) + + if project: + resource_dir = paths.get("resources") + if resource_dir: + output_dir = resource_dir / "Projects" / project + else: + output_dir = vault / "Projects" / project + else: + output_dir = paths.get("paperforge", vault / "System" / "PaperForge") / "logs" / "rendered" + output_dir.mkdir(parents=True, exist_ok=True) + + output_path = output_dir / "reading-log.md" + output_path.write_text("\n".join(lines), encoding="utf-8") +``` + +- [ ] **Step 6: Add correction_note write support** + +In `reading_log.py`, add a `--correct` path for writing correction notes. These reference a prior reading_note by ID and record the correction in `paper_events`. + +```python +# In run(), before the existing export logic, add: +if getattr(args, "correct_id", None): + correction = getattr(args, "correction", "") or "" + reason = getattr(args, "reason", "") or "" + if not correction: + result = PFResult(ok=False, command="reading-log", version=PF_VERSION, + error=PFError(code=ErrorCode.VALIDATION_ERROR, message="--correction is required for --correct")) + ... + return 1 + + # Write correction_note to paper_events + payload = { + "ref_id": args.correct_id, + "correction": correction, + "reason": reason, + } + conn = get_connection(get_memory_db_path(vault), read_only=False) + try: + conn.execute( + "INSERT INTO paper_events (paper_id, event_type, payload_json) VALUES (?, 'correction_note', ?)", + (args.correct_id.split("_")[0], json.dumps(payload, ensure_ascii=False)), + ) + conn.commit() + finally: + conn.close() + ... +``` + +Also add to CLI subparser: +```python +p_rl.add_argument("--correct", dest="correct_id", help="ID of prior reading note to correct") +p_rl.add_argument("--correction", help="Correction text") +p_rl.add_argument("--reason", help="Reason for correction (e.g. 'Rechecked figure legend')") +``` + +- [ ] **Step 7: Commit** + +```python +if getattr(args, "render", False): + project = getattr(args, "project", "") or "" + _render_reading_log_md(vault, project) + result = PFResult( + ok=True, command="reading-log", version=PF_VERSION, + data={"rendered": True, "project": project}, + ) + if args.json: + print(result.to_json()) + else: + print(f"Rendered reading-log.md for {'all projects' if not project else project}") + return 0 +``` + +- [ ] **Step 6: Commit** + +```bash +git add paperforge/commands/reading_log.py paperforge/cli.py paperforge/memory/events.py +git commit -m "feat: upgrade reading-log to JSONL with context/tags/project fields and auto-render" +``` + +--- + +## Task 4: Paper-Context CLI Command + +**Files:** +- Create: `paperforge/commands/paper_context.py` +- Modify: `paperforge/cli.py` (add subparser + dispatch) + +Build the `paperforge paper-context --json` command. + +- [ ] **Step 1: Add CLI subparser** + +In `cli.py`, after the `paper-status` subparser (line 286): + +```python +p_pc = sub.add_parser("paper-context", help="Get full context for a paper (metadata + reading notes + corrections)") +p_pc.add_argument("key", help="Zotero key") +p_pc.add_argument("--json", action="store_true", help="Output as JSON") +``` + +- [ ] **Step 2: Add dispatch in main()** + +In `cli.py` main(), add after paper-status dispatch: + +```python +if args.command == "paper-context": + from paperforge.commands import paper_context + return paper_context.run(args) +``` + +- [ ] **Step 3: Write paper_context.py** + +```python +from __future__ import annotations + +import argparse +import sys + +from paperforge import __version__ as PF_VERSION +from paperforge.core.errors import ErrorCode +from paperforge.core.result import PFError, PFResult +from paperforge.memory.db import get_connection, get_memory_db_path +from paperforge.memory.permanent import get_reading_notes_for_paper + + +def _build_paper_context(vault, key: str) -> dict | None: + """Build full context for a paper: metadata + reading notes + corrections.""" + + # Get paper from DB + db_path = get_memory_db_path(vault) + if not db_path.exists(): + return None + + conn = get_connection(db_path, read_only=True) + try: + row = conn.execute( + """SELECT zotero_key, citation_key, title, year, doi, journal, + first_author, domain, collection_path, has_pdf, + ocr_status, analyze, deep_reading_status, lifecycle, + next_step, pdf_path, note_path, fulltext_path, paper_root + FROM papers WHERE zotero_key = ?""", + (key,), + ).fetchone() + + if not row: + return None + + paper = dict(row) + + # Get reading notes from permanent JSONL + prior_notes = get_reading_notes_for_paper(vault, key) + + # Get corrections from paper_events + corrections = [] + corr_rows = conn.execute( + """SELECT created_at, payload_json + FROM paper_events + WHERE paper_id = ? AND event_type = 'correction_note' + ORDER BY created_at DESC""", + (key,), + ).fetchall() + for cr in corr_rows: + import json + payload = json.loads(cr["payload_json"]) + corrections.append({ + "created_at": cr["created_at"], + "previous_note_id": payload.get("ref_id", ""), + "correction": payload.get("correction", ""), + "reason": payload.get("reason", ""), + }) + + # Build recheck targets (unverified notes) + recheck_targets = [] + for n in prior_notes: + if not n.get("verified", False): + recheck_targets.append( + f"{n.get('section', 'unknown')}: {n.get('excerpt', '')[:80]}..." + ) + + return { + "warning": "Prior reading notes are not verified facts. Re-check source before reuse.", + "paper": paper, + "prior_notes": prior_notes, + "corrections": corrections, + "recheck_targets": recheck_targets, + } + finally: + conn.close() + + +def run(args: argparse.Namespace) -> int: + vault = args.vault_path + key = args.key + + context = _build_paper_context(vault, key) + + if context is None: + result = PFResult( + ok=False, + command="paper-context", + version=PF_VERSION, + error=PFError( + code=ErrorCode.PATH_NOT_FOUND, + message=f"No paper found for key: {key}", + ), + ) + else: + result = PFResult( + ok=True, + command="paper-context", + version=PF_VERSION, + data=context, + ) + + if args.json: + print(result.to_json()) + else: + if result.ok: + p = result.data["paper"] + print(f"Paper: {p.get('title', key)}") + print(f" Key: {p.get('zotero_key', '')}") + print(f" OCR: {p.get('ocr_status', 'unknown')}") + print(f" Lifecycle: {p.get('lifecycle', '')}") + notes = result.data.get("prior_notes", []) + print(f" Reading notes: {len(notes)}") + print(f" Corrections: {len(result.data.get('corrections', []))}") + if result.data.get("recheck_targets"): + print(f" Recheck targets: {len(result.data['recheck_targets'])}") + else: + print(f"Error: {result.error.message}", file=sys.stderr) + + return 0 if result.ok else 1 +``` + +- [ ] **Step 4: Commit** + +```bash +git add paperforge/commands/paper_context.py paperforge/cli.py +git commit -m "feat: add paper-context CLI command for reading-log safety loop" +``` + +--- + +## Task 5: Project-Log CLI Command + +**Files:** +- Create: `paperforge/commands/project_log.py` +- Modify: `paperforge/cli.py` (add subparser + dispatch) + +- [ ] **Step 1: Add CLI subparser** + +In `cli.py`, after the reading-log subparser block: + +```python +p_pl = sub.add_parser("project-log", help="Record or render project work logs") +p_pl.add_argument("--write", action="store_true", help="Write a new project log entry") +p_pl.add_argument("--payload", help="JSON payload for the entry") +p_pl.add_argument("--project", help="Project name (required for write/list/render)") +p_pl.add_argument("--list", action="store_true", help="List all entries for a project") +p_pl.add_argument("--render", action="store_true", help="Render project-log.md") +p_pl.add_argument("--limit", type=int, default=50, help="Max entries to list") +p_pl.add_argument("--json", action="store_true", help="Output as PFResult JSON") +``` + +- [ ] **Step 2: Add dispatch** + +In `cli.py` main(): + +```python +if args.command == "project-log": + from paperforge.commands import project_log + return project_log.run(args) +``` + +- [ ] **Step 3: Write project_log.py** + +```python +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + +from paperforge import __version__ as PF_VERSION +from paperforge.core.errors import ErrorCode +from paperforge.core.result import PFError, PFResult +from paperforge.config import paperforge_paths +from paperforge.memory.permanent import ( + append_project_entry, + get_project_entries, + read_all_project_entries, +) + + +def _render_project_log_md(vault: Path, project: str) -> None: + """Render project-log.md from JSONL.""" + entries = get_project_entries(vault, project) + if not entries: + return + + lines = [f"# Project Log — {project}", ""] + lines.append("> Auto-generated from project-log.jsonl. Do not edit manually.") + lines.append("") + + for entry in sorted(entries, key=lambda x: x.get("created_at", ""), reverse=True): + lines.append(f"## {entry.get('date', '')} — {entry.get('title', '(untitled)')}") + lines.append(f"**Type:** {entry.get('type', '')}") + lines.append("") + + if entry.get("decisions"): + lines.append("### 核心决策") + for d in entry["decisions"]: + lines.append(f"- {d}") + lines.append("") + + if entry.get("detours"): + lines.append("### 弯路与修正") + for dt in entry["detours"]: + if isinstance(dt, dict): + lines.append(f"- **错误:** {dt.get('wrong', '')}") + lines.append(f" **纠正:** {dt.get('correction', '')}") + lines.append(f" **解决:** {dt.get('resolution', '')}") + else: + lines.append(f"- {dt}") + lines.append("") + + if entry.get("reusable"): + lines.append("### 可复用方法论") + for r in entry["reusable"]: + lines.append(f"- {r}") + lines.append("") + + if entry.get("todos"): + lines.append("### 待办") + for t in entry["todos"]: + done = "x" if t.get("done", False) else " " + lines.append(f"- [{done}] {t.get('content', '')}") + lines.append("") + + if entry.get("tags"): + lines.append(f"**Tags:** {', '.join(entry['tags'])}") + + lines.append("---") + lines.append("") + + from paperforge.config import paperforge_paths + paths = paperforge_paths(vault) + resource_dir = paths.get("resources") + if resource_dir: + output_dir = resource_dir / "Projects" / project + else: + output_dir = vault / "Projects" / project + output_dir.mkdir(parents=True, exist_ok=True) + output_path = output_dir / "project-log.md" + output_path.write_text("\n".join(lines), encoding="utf-8") + + +def run(args: argparse.Namespace) -> int: + vault = args.vault_path + + if getattr(args, "write", False): + project = getattr(args, "project", "") + payload_str = getattr(args, "payload", "") + + if not project: + result = PFResult(ok=False, command="project-log", version=PF_VERSION, + error=PFError(code=ErrorCode.VALIDATION_ERROR, message="--project is required for --write")) + print(result.to_json() if getattr(args, "json", False) else result.error.message, + file=sys.stderr if not getattr(args, "json", False) else sys.stdout) + return 1 + + if not payload_str: + result = PFResult(ok=False, command="project-log", version=PF_VERSION, + error=PFError(code=ErrorCode.VALIDATION_ERROR, message="--payload is required for --write")) + print(result.to_json() if getattr(args, "json", False) else result.error.message, + file=sys.stderr if not getattr(args, "json", False) else sys.stdout) + return 1 + + try: + entry = json.loads(payload_str) + entry["project"] = project + result_data = append_project_entry(vault, entry) + + # Auto-render + _render_project_log_md(vault, project) + + result = PFResult(ok=True, command="project-log", version=PF_VERSION, data=result_data) + except json.JSONDecodeError as e: + result = PFResult(ok=False, command="project-log", version=PF_VERSION, + error=PFError(code=ErrorCode.VALIDATION_ERROR, message=f"Invalid JSON: {e}")) + + if getattr(args, "json", False): + print(result.to_json()) + else: + print("Written." if result.ok else f"Error: {result.error.message}") + return 0 if result.ok else 1 + + if getattr(args, "list", False): + project = getattr(args, "project", "") + if not project: + result = PFResult(ok=False, command="project-log", version=PF_VERSION, + error=PFError(code=ErrorCode.VALIDATION_ERROR, message="--project is required for --list")) + print(result.to_json() if getattr(args, "json", False) else result.error.message, + file=sys.stderr if not getattr(args, "json", False) else sys.stdout) + return 1 + + entries = get_project_entries(vault, project) + data = {"project": project, "entries": entries[:getattr(args, "limit", 50)], "count": len(entries)} + result = PFResult(ok=True, command="project-log", version=PF_VERSION, data=data) + + if getattr(args, "json", False): + print(result.to_json()) + else: + print(f"{len(entries)} entries for project '{project}'") + for e in entries[:5]: + print(f" [{e['date']}] {e['type']}: {e['title']}") + return 0 + + if getattr(args, "render", False): + project = getattr(args, "project", "") + if not project: + result = PFResult(ok=False, command="project-log", version=PF_VERSION, + error=PFError(code=ErrorCode.VALIDATION_ERROR, message="--project is required for --render")) + print(result.to_json() if getattr(args, "json", False) else result.error.message, + file=sys.stderr if not getattr(args, "json", False) else sys.stdout) + return 1 + + _render_project_log_md(vault, project) + result = PFResult(ok=True, command="project-log", version=PF_VERSION, + data={"rendered": True, "project": project}) + if getattr(args, "json", False): + print(result.to_json()) + else: + print(f"Rendered project-log.md for '{project}'") + return 0 + + # Default: show all projects with entry counts + all_entries = read_all_project_entries(vault) + from collections import Counter + project_counts = Counter(e["project"] for e in all_entries if e.get("project")) + + result = PFResult(ok=True, command="project-log", version=PF_VERSION, + data={"projects": dict(project_counts)}) + if getattr(args, "json", False): + print(result.to_json()) + else: + if project_counts: + print("Projects with log entries:") + for proj, cnt in project_counts.most_common(): + print(f" {proj}: {cnt} entries") + else: + print("No project log entries found.") + return 0 +``` + +- [ ] **Step 4: Commit** + +```bash +git add paperforge/commands/project_log.py paperforge/cli.py +git commit -m "feat: add project-log CLI command with JSONL write + auto-render" +``` + +--- + +## Task 6: DB Builder — Import JSONL on rebuild + +**Files:** +- Modify: `paperforge/memory/builder.py` + +When `paperforge memory build` runs, it should import reading-log.jsonl → reading_log table and project-log.jsonl → project_log table. + +- [ ] **Step 1: Add import functions** + +In `builder.py`, add: + +```python +import json as _json +from paperforge.memory.permanent import read_all_reading_notes, read_all_project_entries + + +def _import_reading_log(conn, vault: Path) -> int: + """Import reading-log.jsonl into reading_log table. Returns count.""" + notes = read_all_reading_notes(vault) + conn.execute("DELETE FROM reading_log") + count = 0 + for note in notes: + conn.execute( + """INSERT INTO reading_log (id, paper_id, project, section, excerpt, context, usage, note, tags_json, created_at, agent, verified) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""", + ( + note["id"], note["paper_id"], + note.get("project", ""), + note["section"], note["excerpt"], + note.get("context", ""), note["usage"], + note.get("note", ""), + _json.dumps(note.get("tags", []), ensure_ascii=False), + note["created_at"], + note.get("agent", ""), + 1 if note.get("verified") else 0, + ), + ) + count += 1 + return count + + +def _import_project_log(conn, vault: Path) -> int: + """Import project-log.jsonl into project_log table. Returns count.""" + entries = read_all_project_entries(vault) + conn.execute("DELETE FROM project_log") + count = 0 + for entry in entries: + conn.execute( + """INSERT INTO project_log (id, project, date, type, title, decisions_json, detours_json, reusable_json, todos_json, related_papers_json, tags_json, created_at, agent) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""", + ( + entry["id"], entry["project"], + entry.get("date", ""), entry["type"], entry["title"], + _json.dumps(entry.get("decisions", []), ensure_ascii=False), + _json.dumps(entry.get("detours", []), ensure_ascii=False), + _json.dumps(entry.get("reusable", []), ensure_ascii=False), + _json.dumps(entry.get("todos", []), ensure_ascii=False), + _json.dumps(entry.get("related_papers", []), ensure_ascii=False), + _json.dumps(entry.get("tags", []), ensure_ascii=False), + entry.get("created_at", ""), + entry.get("agent", ""), + ), + ) + count += 1 + return count +``` + +- [ ] **Step 2: Call import functions in build flow** + +Find the build function in `builder.py` and add after papers import: + +```python +reading_count = _import_reading_log(conn, vault) +logger.info(f"Imported {reading_count} reading notes from JSONL") + +project_count = _import_project_log(conn, vault) +logger.info(f"Imported {project_count} project log entries from JSONL") + +# Clear paper_events to prevent duplicate accumulation from previous rebuilds +# Note: correction_note events also live here; consider dual-writing corrections to +# reading-log.jsonl in future to preserve them across rebuilds. +conn.execute("DELETE FROM paper_events WHERE event_type != 'correction_note';") +``` + +- [ ] **Step 3: Commit** + +```bash +git add paperforge/memory/builder.py +git commit -m "feat: import JSONL into DB on memory build" +``` + +--- + +## Task 7: METHODOLOGY_COMPACT.md Creation + +**Files:** +- Create: `fixtures/methodology/METHODOLOGY_COMPACT.md` +- Modify: `paperforge/setup_wizard.py` (optional, deploy template) + +Create the default methodology compact file. This goes in: + +- **Vault location**: `/PaperForge/methodology/METHODOLOGY_COMPACT.md` (NOT in `archive/`) +- **Package fixture**: `fixtures/methodology/METHODOLOGY_COMPACT.md` (copied during setup) +- **Note**: `pf_bootstrap.py` scans `archive/` for method cards; `METHODOLOGY_COMPACT.md` is deliberately outside `archive/` since it's a system file, not a searchable card. + +- [ ] **Step 1: Create the file** + +```markdown +# PaperForge Methodology Compact + +## General +- Separate source fact, interpretation, and intended use. +- Prior reading-log is not verified fact; re-check source before reuse. +- When user corrects a judgment, record the correction if relevant. + +## Literature work +- Do not collapse heterogeneous studies without comparing model, parameter, endpoint, and measurement layer. +- Distinguish device-level settings from local biological exposure. +- Confirm within-study internal chain (material→output→effect) before making cross-study claims. + +## Clinical research +- Separate candidate variables, selected variables, final model variables, and sensitivity variables. +- Do not infer causality from predictive variables. + +## Writing +- Do not write unsupported claims. Every factual claim must have a source reference. +- Prefer bounded conclusions over broad overclaims. +- Distinguish "the paper says X" from "I infer Y from X". +``` + +- [ ] **Step 2: Ensure setup wizard creates methodology directory** + +In `setup_wizard.py` or `setup/` modules, ensure `System/PaperForge/methodology/archive/` is created and METHODOLOGY_COMPACT.md is copied. + +- [ ] **Step 3: Verify methodology index scanning in pf_bootstrap.py** + +The `_scan_methodology_archive()` function already reads from `System/PaperForge/methodology/archive/`. Verify it works with the actual file structure. + +- [ ] **Step 4: Commit** + +```bash +git add fixtures/methodology/METHODOLOGY_COMPACT.md +git commit -m "feat: add METHODOLOGY_COMPACT.md for agent guidance" +``` + +--- + +## Task 8: Integration — Wire Everything Together + +**Files:** +- Modify: `paperforge/skills/paperforge/SKILL.md` (verify CLI references are correct) + +- [ ] **Step 1: Verify skill SKILL.md references correct commands** + +Check that SKILL.md and all workflow files reference the correct CLI commands (`paper-context`, `reading-log --write`, etc.) with correct parameter names. + +- [ ] **Step 2: Smoke test the full flow** + +```bash +# Test paper-context +$PYTHON -m paperforge paper-context ABC12345 --json --vault + +# Test reading-log write +$PYTHON -m paperforge reading-log --write ABC12345 \ + --section "Results Fig.3" --excerpt "test" --usage "test" \ + --context "test context" --project "test-project" \ + --vault + +# Test reading-log render +$PYTHON -m paperforge reading-log --render --project "test-project" --vault + +# Test project-log write +$PYTHON -m paperforge project-log --write --project "test-project" \ + --payload '{"date":"2026-05-14","type":"note","title":"test"}' \ + --vault + +# Test project-log render +$PYTHON -m paperforge project-log --render --project "test-project" --vault +``` + +- [ ] **Step 3: Commit** + +```bash +git add paperforge/skills/paperforge/SKILL.md +git commit -m "chore: verify skill CLI references match new commands" +``` + +--- + +## Summary + +| Task | Files | Description | +|------|-------|-------------| +| 1 | `memory/schema.py` | Add reading_log + project_log DB tables | +| 2 | `memory/permanent.py` (new) | JSONL append/read for both log types | +| 3 | `commands/reading_log.py`, `cli.py`, `memory/events.py` | Upgrade reading-log CLI with JSONL + new fields + render | +| 4 | `commands/paper_context.py` (new), `cli.py` | New paper-context command | +| 5 | `commands/project_log.py` (new), `cli.py` | New project-log CLI | +| 6 | `memory/builder.py` | Import JSONL → DB on rebuild | +| 7 | `fixtures/methodology/METHODOLOGY_COMPACT.md` (new) | Default methodology compact | +| 8 | Skill files | Verify references, smoke test | + +### Build order dependencies +Task 2 → Task 3 (JSONL storage needed for CLI) +Task 2 → Task 4 (paper-context reads reading notes from JSONL) +Task 2 → Task 5 (project-log writes to JSONL) +Task 2 → Task 6 (DB import reads from JSONL) +Task 7 is independent diff --git a/docs/superpowers/plans/2026-05-14-user-side-stabilization.md b/docs/superpowers/plans/2026-05-14-user-side-stabilization.md new file mode 100644 index 00000000..908c6eb9 --- /dev/null +++ b/docs/superpowers/plans/2026-05-14-user-side-stabilization.md @@ -0,0 +1,364 @@ +# User-Side Skill Stabilization Plan + +> **For agentic workers:** Use subagent-driven-development to implement. +> Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make paperforge skill + memory layer reliable for local self-use — no data loss, no split truth sources, no agent misrouting. + +**Architecture:** JSONL as single source of truth, SQLite as derived index, CLI as sole write interface, bootstrap as sole path resolver. + +**Tech Stack:** Python, JSONL, SQLite, paperforge CLI + +--- + +## Task 1: Unify reading-log data flow (JSONL as single source) + +**Files:** +- Modify: `paperforge/commands/reading_log.py` (lookup + export data sources, remove write_reading_note call) +- Modify: `paperforge/memory/events.py` (mark write_reading_note deprecated) + +**Goal:** `--lookup` and `export` read from JSONL, not paper_events. `--write` only calls `append_reading_note`. + +- [ ] **Step 1: Change lookup_paper_events to read from JSONL** + +In `reading_log.py`, replace `lookup_paper_events()` body to use `get_reading_notes_for_paper()` from JSONL instead of querying paper_events. Keep the function return shape the same. Join with papers table for title/year/citation_key. + +```python +def lookup_paper_events(vault: Path, key: str) -> dict: + notes = get_reading_notes_for_paper(vault, key) + if not notes: + return {"ok": True, "zotero_key": key, "title": "", "entries": [], "count": 0} + + # Try to get paper metadata from DB for richer display + db_path = get_memory_db_path(vault) + title = "" + if db_path.exists(): + conn = get_connection(db_path, read_only=True) + try: + row = conn.execute( + "SELECT title FROM papers WHERE zotero_key = ?", (key,) + ).fetchone() + if row: + title = row["title"] or "" + finally: + conn.close() + + entries = [] + for n in notes: + entries.append({ + "created_at": n.get("created_at", ""), + "section": n.get("section", ""), + "excerpt": n.get("excerpt", ""), + "usage": n.get("usage", ""), + "note": n.get("note", ""), + }) + return {"ok": True, "zotero_key": key, "title": title, "entries": entries, "count": len(entries)} +``` + +- [ ] **Step 2: Change export_reading_log to read from JSONL** + +In `events.py`, add a new `export_reading_log_from_jsonl()` that reads from `read_all_reading_notes()`, joins with DB for metadata. Or in `reading_log.py`, replace the export path (around line 286) to use JSONL directly with a local join helper. + +- [ ] **Step 3: Remove write_reading_note from write path** + +In `reading_log.py` `run()`, remove the `write_reading_note()` call (the second write). Only `append_reading_note()` remains. + +- [ ] **Step 4: Mark write_reading_note deprecated** + +Add deprecation docstring to `write_reading_note` in `events.py`. Do NOT remove the function (backward compat for import_reading_log which may still use it). + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/commands/reading_log.py paperforge/memory/events.py +git commit -m "refactor: unify reading-log reads to JSONL, deprecate paper_events writes" +``` + +--- + +## Task 2: Permanent correction-log.jsonl + +**Files:** +- Modify: `paperforge/memory/permanent.py` (add correction append/read functions) +- Modify: `paperforge/commands/paper_context.py` (read corrections from JSONL, fix ref_id → original_id) +- Modify: `paperforge/commands/reading_log.py` (write correction to JSONL, not just paper_events) +- Modify: `paperforge/memory/builder.py` (import corrections on rebuild) + +**Goal:** Correction notes live in `correction-log.jsonl`, survive DB rebuilds. + +- [ ] **Step 1: Add correction helpers to permanent.py** + +```python +# ── Correction Log ───────────────────────────────────────────────────────── + +def get_correction_log_path(vault: Path) -> Path: + return _logs_dir(vault) / "correction-log.jsonl" + + +def append_correction( + vault: Path, + paper_id: str, + original_id: str, + correction: str, + reason: str = "", + agent: str = "", +) -> dict: + if not paper_id: + return {"ok": False, "error": "paper_id is required"} + if not original_id: + return {"ok": False, "error": "original_id is required"} + if not correction: + return {"ok": False, "error": "correction is required"} + + date_str = datetime.date.today().strftime("%Y%m%d") + entry_id = f"corr_{date_str}_{secrets.token_hex(4)}" + now = datetime.datetime.now(datetime.timezone.utc).isoformat() + + entry: dict[str, object] = { + "id": entry_id, + "event_type": "correction", + "created_at": now, + "paper_id": paper_id, + "original_id": original_id, + "correction": correction, + "reason": reason, + "agent": agent, + } + + log_dir = _ensure_logs_dir(vault) + filepath = log_dir / "correction-log.jsonl" + + try: + with filepath.open("a", encoding="utf-8") as f: + f.write(json.dumps(entry, ensure_ascii=False) + "\n") + except OSError as e: + return {"ok": False, "error": str(e)} + + return {"ok": True, "id": entry_id, "path": str(filepath)} + + +def read_all_corrections(vault: Path) -> list[dict]: + filepath = get_correction_log_path(vault) + return _read_jsonl(filepath) + + +def get_corrections_for_paper(vault: Path, paper_id: str) -> list[dict]: + all_corrections = read_all_corrections(vault) + return [c for c in all_corrections if c.get("paper_id") == paper_id] +``` + +- [ ] **Step 2: Fix paper_context.py to read from JSONL** + +Replace the `paper_events` query for corrections with `get_corrections_for_paper()`. Fix `ref_id` → `original_id`. + +- [ ] **Step 3: Dual-write correction in reading_log.py** + +In `reading_log.py` `run()`, when writing a correction, write to BOTH: +1. `append_correction()` → `correction-log.jsonl` (permanent) +2. `write_correction_note()` → `paper_events` (FTS search, optional — keep for now) + +- [ ] **Step 4: Add correction import to builder.py** + +Add `_import_correction_log()` that reads `correction-log.jsonl` and inserts into paper_events (so corrections are searchable after rebuild). + +- [ ] **Step 5: Commit** + +```bash +git add paperforge/memory/permanent.py paperforge/commands/paper_context.py \ + paperforge/commands/reading_log.py paperforge/memory/builder.py +git commit -m "feat: permanent correction-log.jsonl, fix original_id field alignment" +``` + +--- + +## Task 3: Fix pf_bootstrap.py path resolution + Python fallback + +**Files:** +- Modify: `paperforge/skills/paperforge/scripts/pf_bootstrap.py` + +**Goal:** Support `vault_config` nested config, fallback Python to `python`/`python3`/`sys.executable`. + +- [ ] **Step 1: Add resolve_cfg()** + +```python +DEFAULTS = { + "system_dir": "System", + "resources_dir": "Resources", + "literature_dir": "Literature", + "control_dir": "LiteratureControl", + "base_dir": "Bases", +} + +def resolve_cfg(raw: dict) -> dict: + cfg = DEFAULTS.copy() + nested = raw.get("vault_config", {}) + if isinstance(nested, dict): + cfg.update({k: v for k, v in nested.items() if v}) + cfg.update({k: raw[k] for k in DEFAULTS if raw.get(k)}) + return cfg +``` + +Replace direct `cfg.get(...)` calls with `cfg = resolve_cfg(...)`. + +- [ ] **Step 2: Fix Python fallback** + +In `_find_python_with_paperforge()`, if no paperforge-capable Python is found: +- Try `python`, `python3` +- Fall back to `sys.executable` +- Return `"python"` (not None) with `python_verified: false` + +In `main()`, after python_candidate: +```python +if result.get("python_candidate"): + result["python_verified"] = True +else: + result["python_candidate"] = "python" + result["python_verified"] = False +``` + +- [ ] **Step 3: Commit** + +```bash +git add paperforge/skills/paperforge/scripts/pf_bootstrap.py +git commit -m "fix: bootstrap vault_config nest, python fallback with verified flag" +``` + +--- + +## Task 4: FTS safe query with fallback + +**Files:** +- Modify: `paperforge/memory/fts.py` + +**Goal:** Three-level fallback for FTS queries. + +- [ ] **Step 1: Add tokenizer and fallback logic** + +```python +import re + +def tokenize_for_fts(q: str) -> str: + """Extract alphanumeric + CJK tokens and quote them for safe FTS.""" + tokens = re.findall(r"[\w\u4e00-\u9fff]+", q) + if not tokens: + return q + return " OR ".join(f'"{t}"' for t in tokens) + + +def _fts_search(conn, query, params, limit): + """Try raw FTS, fall back to quoted tokens, then LIKE.""" + from contextlib import closing + + conditions = ["paper_fts MATCH ?"] + all_params = [query] + params + + try: + where = " AND ".join(conditions) + sql = f""" + SELECT ... FROM paper_fts f JOIN papers p ON p.rowid = f.rowid + WHERE {where} ORDER BY rank LIMIT ? + """ + all_params.append(limit) + conn.row_factory = sqlite3.Row + return conn.execute(sql, all_params).fetchall() + except sqlite3.OperationalError: + pass + + # Level 2: quoted tokens + try: + token_query = tokenize_for_fts(query) + all_params = [token_query] + params + [limit] + sql = f""" + SELECT ... FROM paper_fts f JOIN papers p ON p.rowid = f.rowid + WHERE paper_fts MATCH ? {'AND ...' * len(params)} ORDER BY rank LIMIT ? + """ + return conn.execute(sql, all_params).fetchall() + except sqlite3.OperationalError: + pass + + # Level 3: LIKE fallback + like_param = f"%{query}%" + conditions = [ + "(p.title LIKE ? OR p.abstract LIKE ? OR p.doi LIKE ? OR p.citation_key LIKE ?)" + ] + all_params = [like_param, like_param, like_param, like_param] + params + [limit] + # ... construct and execute LIKE query + return rows +``` + +- [ ] **Step 2: Update search_papers() to use _fts_search** + +Replace the direct FTS query with the safe fallback. + +- [ ] **Step 3: Commit** + +```bash +git add paperforge/memory/fts.py +git commit -m "feat: safe FTS search with token-quote and LIKE fallback" +``` + +--- + +## Task 5: SKILL.md — router + PYTHON fallback + workflow rules + +**Files:** +- Create: `paperforge/skills/paperforge/workflows/project-engineering.md` +- Modify: `paperforge/skills/paperforge/SKILL.md` +- Modify: `paperforge/skills/paperforge/workflows/deep-reading.md` +- Modify: `paperforge/skills/paperforge/workflows/paper-qa.md` +- Modify: `paperforge/skills/paperforge/workflows/project-log.md` +- Modify: `paperforge/skills/paperforge/workflows/methodology.md` + +**Goal:** Engineering router, PYTHON fallback, workflow safety reinforcement. + +- [ ] **Step 1: Add PYTHON fallback to SKILL.md** + +After bootstrap section, add: +```markdown +If `python_verified` is `false` or `python_candidate` is `null`: +Try `python` → `python3` → `sys.executable` in order. +If all fail, stop and tell user to set `python_path` in `paperforge.json`. +``` + +- [ ] **Step 2: Add project-engineering to router table** + +Add to routing table: +```markdown +| "branch" "代码审查" "feature" "dashboard" "memory layer" "用户反馈" "报错" "安装失败" "Git" "Zotero" "BetterBibTeX" "OCR" "插件" | `workflows/project-engineering.md` | +``` + +- [ ] **Step 3: Create project-engineering workflow stub** + +Minimal workflow for engineering tasks — routes to appropriate context retrieval. + +- [ ] **Step 4: Reinforce safety rule in deep-reading.md and paper-qa.md** + +Add at top of each: +```markdown +> Prior reading-log entries are **recheck targets only**, never factual answers. +> Always verify against original source before using any reading-log content. +``` + +- [ ] **Step 5: Clarify project-log vs methodology boundary** + +In project-log.md: "Record what happened this session." +In methodology.md: "Only archive methods reusable across multiple projects/tasks." + +- [ ] **Step 6: Commit** + +```bash +git add paperforge/skills/paperforge/ +git commit -m "chore: add PYTHON fallback, engineering router, workflow safety rules" +``` + +--- + +## Summary + +| Task | Priority | Files | Risk | +|------|----------|-------|------| +| 1 — reading-log unify | P0 | reading_log.py, events.py | Low — just changes data source | +| 2 — correction permanent | P0 | permanent.py, paper_context.py, reading_log.py, builder.py | Low — new file, additive | +| 3 — bootstrap fix | P0 | pf_bootstrap.py | Low — no changes to CLI | +| 4 — FTS safe query | P0 | fts.py | Low — additive fallback | +| 5 — SKILL/router/rules | P1 | 6 files under skills/ | Trivial — docs only | diff --git a/docs/superpowers/plans/2026-05-14-vector-stabilization.md b/docs/superpowers/plans/2026-05-14-vector-stabilization.md new file mode 100644 index 00000000..83aaa9bd --- /dev/null +++ b/docs/superpowers/plans/2026-05-14-vector-stabilization.md @@ -0,0 +1,398 @@ +# Vector DB Flow Stabilization Plan + +> **For agentic workers:** Use subagent-driven-development to implement. +> Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the user flow "install → enable vector DB → embed build → retrieve" stable — no missing deps, no silent failures, no misleading UI. + +**Architecture:** Plugin settings drive the feature toggle; CLI preflight checks all deps before build; retrieve refuses to query empty index. + +**Tech Stack:** esbuild (plugin), chromadb, sentence-transformers, openai, Python CLI + +--- + +## Task 1: Remove ghost Setting code from main.js + +**Files:** +- Modify: `paperforge/plugin/main.js:200-215` + +**Goal:** Delete the misplaced `new Setting(containerEl)` block that sits inside `runSubprocess`. All vector settings already exist correctly in `_renderFeaturesTab`. + +- [ ] **Step 1: Delete lines 200-215** + +Remove the extra `});` and the two `new Setting(containerEl).setName('API Model')` blocks that are at lines 200-215. + +The code to remove: +```javascript + }); + new Setting(containerEl) + .setName('API Model') + .setDesc('Which OpenAI-compatible embedding model to use.') + new Setting(containerEl) + .setName('API Model') + .setDesc('Embedding model name (e.g., text-embedding-3-small, qwen-3-embedding)') + .addText(text => { + text.setPlaceholder('text-embedding-3-small') + .setValue(this.plugin.settings.vector_db_api_model || 'text-embedding-3-small') + .onChange(value => { + this.plugin.settings.vector_db_api_model = value; + this.plugin.saveSettings(); + }); + }); + } +``` + +- [ ] **Step 2: Verify syntax** + +Run: `node --check paperforge/plugin/main.js` +Expected: PASS (exit code 0) + +- [ ] **Step 3: Commit** + +```bash +git add paperforge/plugin/main.js +git commit -m "fix: remove ghost vector API Model setting from runSubprocess scope" +``` + +--- + +## Task 2: Add vector optional dependencies to pyproject.toml + +**Files:** +- Modify: `pyproject.toml` + +**Goal:** Users can `pip install paperforge[vector]` to get chromadb + embeddings deps. + +- [ ] **Step 1: Add vector extra** + +In `pyproject.toml`, under `[project]` (or at end of file), add: + +```toml +[project.optional-dependencies] +vector = [ + "chromadb>=0.5.0", + "sentence-transformers>=3.0.0", + "openai>=1.0.0", +] +``` + +- [ ] **Step 2: Add install hint to plugin setup page** + +In `main.js` `_renderSetupTab()`, after dependency check output, add a notice: + +```javascript +// In the deps status display section, add: +const vectorHint = containerEl.createEl('p', { + cls: 'paperforge-settings-desc', + text: 'Vector Database requires additional dependencies. ' + + 'Run: pip install "paperforge[vector]"' +}); +``` + +- [ ] **Step 3: Verify** + +Run: `python -c "import chromadb; import sentence_transformers; import openai; print('vector deps OK')"` +(Only needed if deps are already installed — otherwise verify pyproject.toml syntax) + +- [ ] **Step 4: Commit** + +```bash +git add pyproject.toml paperforge/plugin/main.js +git commit -m "feat: add vector optional deps and install hint in plugin UI" +``` + +--- + +## Task 3: Add preflight to embed build + +**Files:** +- Modify: `paperforge/worker/vector_db.py` (add `_preflight_check()`) +- Modify: `paperforge/commands/embed.py` (call preflight before build) + +**Goal:** Before building embeddings, check all prerequisites and give clear error messages. + +- [ ] **Step 1: Add _preflight_check() to vector_db.py** + +```python +def _preflight_check(vault: Path, settings: dict) -> dict: + """Check all prerequisites for embed build. + + Returns {"ok": True} or {"ok": False, "error": "...", "fix": "..."} + """ + # 1. Check vector_db feature toggle + if not settings.get("features", {}).get("vector_db", False): + return { + "ok": False, + "error": "Vector DB is not enabled", + "fix": "Enable 'Vector Database' in PaperForge plugin settings (Features tab).", + } + + # 2. Check chromadb import + try: + import chromadb + except ImportError: + return { + "ok": False, + "error": "chromadb is not installed", + "fix": 'Run: pip install "paperforge[vector]"', + } + + # 3. Check mode-specific deps + mode = settings.get("vector_db_mode", "local") + if mode == "local": + try: + import sentence_transformers + except ImportError: + return { + "ok": False, + "error": "sentence-transformers is not installed (required for local mode)", + "fix": 'Run: pip install "paperforge[vector]" or switch to API mode in plugin settings.', + } + elif mode == "api": + try: + import openai + except ImportError: + return { + "ok": False, + "error": "openai is not installed (required for API mode)", + "fix": 'Run: pip install "paperforge[vector]" or switch to local mode in plugin settings.', + } + api_key = ( + settings.get("vector_db_api_key") + or os.environ.get("OPENAI_API_KEY") + or os.environ.get("VECTOR_DB_API_KEY") + ) + if not api_key: + return { + "ok": False, + "error": "API key not configured for API mode", + "fix": "Set API Key in PaperForge plugin settings (Features tab) or OPENAI_API_KEY in .env.", + } + + # 4. Check OCR done papers + from paperforge.worker._utils import pipeline_paths, read_json + paths = pipeline_paths(vault) + index_path = paths.get("indexes", Path()) / "formal-library.json" + if not index_path.exists(): + return { + "ok": False, + "error": "Index not found", + "fix": "Run paperforge memory build first.", + } + + index_data = read_json(index_path) + items = index_data.get("items", []) if isinstance(index_data, dict) else index_data + papers_with_ocr = [item for item in (items or []) if item.get("ocr_status") == "done"] + + if not papers_with_ocr: + return { + "ok": False, + "error": "No papers with OCR completed", + "fix": "Run paperforge ocr first, or set do_ocr: true on papers with PDFs.", + } + + # 5. Check fulltext files exist + unreadable = 0 + for item in papers_with_ocr[:5]: # sample check + fulltext = item.get("fulltext_path", "") + if fulltext and not Path(fulltext).exists(): + unreadable += 1 + + return { + "ok": True, + "ocr_done_count": len(papers_with_ocr), + "fulltext_unreachable_sample": unreadable, + } +``` + +- [ ] **Step 2: Call preflight in embed build** + +In `embed.py` (or wherever `embed_build` is called), before the main loop: + +```python +settings = read_plugin_settings(vault) # read from data.json +preflight = _preflight_check(vault, settings) +if not preflight["ok"]: + return PFResult( + ok=False, + command="embed-build", + version=PF_VERSION, + error=PFError(code=ErrorCode.VALIDATION_ERROR, message=preflight["error"]), + data={"fix": preflight.get("fix", "")}, + ) +``` + +- [ ] **Step 3: Verify** + +Test manually: `python -m paperforge embed build` without vector deps → should get clear error. + +- [ ] **Step 4: Commit** + +```bash +git add paperforge/worker/vector_db.py paperforge/commands/embed.py +git commit -m "feat: add preflight checks before embed build" +``` + +--- + +## Task 4: Gating — respect feature toggle in CLI + +**Files:** +- Modify: `paperforge/worker/vector_db.py` (preflight already checks this) +- Modify: `paperforge/skills/paperforge/scripts/pf_bootstrap.py` (fix vector_search reading) + +**Goal:** If user hasn't enabled vector_db in settings, CLI should refuse to build, and bootstrap should report accurately. + +- [ ] **Step 1: Verify preflight checks vector_db toggle** + +(Already done in Task 3 step 1 — the `_preflight_check` checks `features.vector_db`) + +- [ ] **Step 2: Fix bootstrap vector_search field** + +In `pf_bootstrap.py`, change the memory_layer check to: + +```python +# Read plugin data.json for real settings +dc_json = vault / ".obsidian" / "plugins" / "paperforge" / "data.json" +vector_enabled = False +if dc_json.exists(): + try: + with open(dc_json, encoding="utf-8") as f: + plugin_data = json.load(f) + vector_enabled = plugin_data.get("features", {}).get("vector_db", False) + except: + pass +memory_layer["vector_search"] = vector_enabled +``` + +(Current code already does this — verify it reads from the right path.) + +- [ ] **Step 3: Commit** + +```bash +git add paperforge/skills/paperforge/scripts/pf_bootstrap.py +git commit -m "fix: ensure bootstrap reads vector_db toggle from plugin settings" +``` + +--- + +## Task 5: Retrieve guard — refuse empty index + +**Files:** +- Modify: `paperforge/commands/retrieve.py` +- Modify: `paperforge/worker/vector_db.py` (add get_embed_status) + +**Goal:** `paperforge retrieve` checks if index has chunks before loading models. + +- [ ] **Step 1: Add get_embed_status() to vector_db.py** + +```python +def get_embed_status(vault: Path) -> dict: + """Check if vector index exists and has content.""" + from paperforge.config import paperforge_paths + paths = paperforge_paths(vault) + vectors_dir = paths.get("vectors", paths["paperforge"] / "vectors") + + status = { + "exists": False, + "chunk_count": 0, + "collection_name": "", + "embedding_model": "", + } + + if not vectors_dir.exists(): + return status + + # Try to read Chroma collection metadata + try: + import chromadb + client = chromadb.PersistentClient(path=str(vectors_dir)) + collections = client.list_collections() + if collections: + col = collections[0] + status["exists"] = True + status["collection_name"] = col.name + status["chunk_count"] = col.count() + status["embedding_model"] = col.metadata.get("embedding_model", "") if col.metadata else "" + except Exception: + pass + + return status +``` + +- [ ] **Step 2: Guard retrieve command** + +In `retrieve.py`, before calling `retrieve_chunks()`: + +```python +status = get_embed_status(vault) +if status["chunk_count"] == 0: + result = PFResult( + ok=False, + command="retrieve", + version=PF_VERSION, + error=PFError( + code=ErrorCode.PATH_NOT_FOUND, + message="Vector index is empty. Run paperforge embed build first.", + ), + data={"next_action": "paperforge embed build"}, + ) + if args.json: + print(result.to_json()) + else: + print(f"Error: {result.error.message}", file=sys.stderr) + return 1 +``` + +- [ ] **Step 3: Commit** + +```bash +git add paperforge/commands/retrieve.py paperforge/worker/vector_db.py +git commit -m "feat: guard retrieve against empty index, add embed status check" +``` + +--- + +## Task 6: Local mode HF download hint + +**Files:** +- Modify: `paperforge/plugin/main.js` (_renderFeaturesTab) + +**Goal:** Users on local mode know they need HF access and can set a mirror. + +- [ ] **Step 1: Add HF download hint** + +In `_renderFeaturesTab`, after the local model selector, add: + +```javascript +new Setting(containerEl) + .setName('HF Download Notice') + .setDesc( + 'Local mode downloads models from Hugging Face on first use. ' + + 'If inaccessible, set HF Endpoint below (e.g. https://hf-mirror.com) ' + + 'or switch to API mode.' + ) + .setDisabled(true); +``` + +(Use Obsidian Setting's `setDisabled` to make it a read-only notice, or just use an `info` div.) + +- [ ] **Step 2: Commit** + +```bash +git add paperforge/plugin/main.js +git commit -m "feat: add HF download notice for local mode in plugin settings" +``` + +--- + +## Summary + +| Task | Priority | Files | Risk | +|------|----------|-------|------| +| 1 — Remove ghost code | P0 | main.js | None — deletes orphan code | +| 2 — Vector deps | P0 | pyproject.toml, main.js | Low — additive | +| 3 — Preflight | P0 | vector_db.py, embed.py | Low — additive guard | +| 4 — Feature gating | P1 | vector_db.py, pf_bootstrap.py | Low — already mostly done | +| 5 — Retrieve guard | P1 | retrieve.py, vector_db.py | Low — additive guard | +| 6 — HF hint | P2 | main.js | None — UI text only | diff --git a/paperforge/cli.py b/paperforge/cli.py index e289d445..c7c85a40 100644 --- a/paperforge/cli.py +++ b/paperforge/cli.py @@ -264,6 +264,7 @@ def build_parser() -> argparse.ArgumentParser: p_embed_build = p_embed_sp.add_parser("build", help="Build vector index from OCR fulltext") p_embed_build.add_argument("--json", action="store_true") p_embed_build.add_argument("--force", action="store_true") + p_embed_build.add_argument("--resume", action="store_true", help="Skip papers already in vector index") p_embed_status = p_embed_sp.add_parser("status", help="Check vector DB status") p_embed_status.add_argument("--json", action="store_true") diff --git a/paperforge/commands/embed.py b/paperforge/commands/embed.py index f59fd02c..f0b17f91 100644 --- a/paperforge/commands/embed.py +++ b/paperforge/commands/embed.py @@ -10,6 +10,7 @@ from paperforge.memory.chunker import chunk_fulltext from paperforge.memory.vector_db import ( delete_paper_vectors, embed_paper, + get_collection, get_embed_status, get_vector_db_path, )