mirror of
https://github.com/lllin000/PaperForge.git
synced 2026-07-22 06:50:53 +00:00
Merge branch 'feat/retrieval-recovery'
# Conflicts: # paperforge/commands/embed.py
This commit is contained in:
LLLin000
commit
b4ac277438
21 changed files with 8444 additions and 7074 deletions
24
.planning/2026-07-10-retrieval-recovery-plan.md
Normal file
24
.planning/2026-07-10-retrieval-recovery-plan.md
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
# Retrieval Recovery — Phase 1: Unified Contract
|
||||
|
||||
> **For agentic workers:** Use TDD. Write failing test, watch it fail, implement minimal fix, verify green.
|
||||
|
||||
**Goal:** Restore retrieval correctness by unifying the Python result envelope and fixing the plugin parser/spawn contracts.
|
||||
|
||||
**Architecture:** Approach A — Python CLI is sole retrieval owner. Plugin spawns CLI per query, parses one unified PFResult envelope. sql.js deleted.
|
||||
|
||||
**Tech Stack:** Python 3.14 (D:/L/OB/Literature-hub/.venv/Scripts/python.exe), TypeScript/esbuild Obsidian plugin, sqlite-vec 0.1.9, FTS5.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Source Corpus (papers, OCR, blocks, metadata, annotations) MUST NOT be modified.
|
||||
- Retrieval Artifacts (FTS indexes, vec0 tables, companion meta) are disposable.
|
||||
- Python executable: `D:/L/OB/Literature-hub/.venv/Scripts/python.exe`
|
||||
- Vault: `D:/L/OB/Literature-hub`
|
||||
- Repository worktree: `D:/L/Med/Research/99_System/LiteraturePipeline/github-release/.worktrees/retrieval-recovery`
|
||||
- Test vault for disposable tests: `tests/fixtures/vault`
|
||||
- All CLI calls use `--vault` flag pointing to the test vault, never the live vault in tests.
|
||||
- Contract tests must parse real CLI JSON output, not mock it.
|
||||
- Existing 89 tests must keep passing.
|
||||
- Commit after every task with Conventional Commits style (repo convention: `feat:`, `fix:`, `chore:`).
|
||||
|
||||
---
|
||||
1016
.planning/retrieval-approach-b-split-read.md
Normal file
1016
.planning/retrieval-approach-b-split-read.md
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,400 @@
|
|||
# PaperForge Retrieval Architecture and Contract Drift
|
||||
|
||||
- Evidence date: 2026-07-10
|
||||
- Wayfinder ticket: [Inventory the live retrieval architecture and contract drift](https://github.com/LLLin000/PaperForge/issues/53)
|
||||
- Parent map: [Wayfinder: Restore PaperForge retrieval end to end](https://github.com/LLLin000/PaperForge/issues/45)
|
||||
- Scope: architecture inventory only; no production fix was applied
|
||||
|
||||
## Executive finding
|
||||
|
||||
PaperForge does not currently have one coherent “vector layer.” It has four partially overlapping systems whose contracts drift independently:
|
||||
|
||||
1. Metadata search has a sql.js path and a Python CLI path.
|
||||
2. The Obsidian `@` path invokes standard vector retrieval, while a separate hybrid deep-search implementation exists but is not selected by the UI.
|
||||
3. Vector build control has three state representations: Python process state, SQLite `build_state`, and a JSON runtime snapshot consumed by the plugin.
|
||||
4. ChromaDB migration code and a Chroma/Lance backend abstraction remain beside a direct sqlite-vec production path.
|
||||
|
||||
The highest-impact observed failures are contractual, not tuning problems:
|
||||
|
||||
- The sql.js metadata query selects a nonexistent `paper_fts.year` column and cannot initialize.
|
||||
- The `@` UI does not pass the CLI `--deep` flag.
|
||||
- Every retrieve response is emitted under `data.chunks`, while the plugin only accepts `data.matches` or `data.results`.
|
||||
- The build loop writes new vec0 rows and then deletes every row for the same paper, including the new rows.
|
||||
- Resume/force decisions still use the legacy Chroma `indexes/vectors` directory even though active vectors live inside `paperforge.db`.
|
||||
- The plugin renders nested build state from a JSON snapshot rather than the live SQLite state, so a dead process can remain visually “running.”
|
||||
|
||||
These facts explain why isolated fixes have not stabilized the user journey: each fix has targeted one representation while another representation continued to drive the UI or data path.
|
||||
|
||||
## Domain boundary
|
||||
|
||||
The Wayfinder map already established:
|
||||
|
||||
- **Source Corpus**: paper notes, OCR output, structured blocks, metadata, and user annotations. It is authoritative and must be preserved.
|
||||
- **Retrieval Artifacts**: FTS indexes, embeddings, vec0 tables, and companion metadata. They are disposable and may be rebuilt.
|
||||
- **Retrieval Experience**: M metadata search, `@` knowledge search, and the build/status/stop/resume controls needed to keep both usable.
|
||||
|
||||
This audit does not treat existing ChromaDB, vec0, or FTS contents as preservation targets.
|
||||
|
||||
## Current execution map
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
U[Obsidian user]
|
||||
|
||||
U -->|M input, 200 ms| SJ[sql.js search]
|
||||
SJ -->|reads a one-time copy| DB[(paperforge.db)]
|
||||
SJ -->|prepare fails: paper_fts.year missing| CF[Python CLI fallback]
|
||||
U -->|M Enter| CF
|
||||
CF --> SC[paperforge search]
|
||||
SC --> PF[paper_fts JOIN papers]
|
||||
SC --> PM[data.matches]
|
||||
PM --> UI[Search card renderer]
|
||||
|
||||
U -->|@ Enter| RC[paperforge retrieve without --deep]
|
||||
RC --> MR[merge_retrieve]
|
||||
MR --> VEC[vec_fulltext + vec_body + vec_objects]
|
||||
RC --> CH[data.chunks]
|
||||
CH -->|plugin ignores chunks| EMPTY[No results found]
|
||||
|
||||
HS[hybrid_search]
|
||||
HS --> BF[body_units_fts BM25]
|
||||
HS --> VB[vec_body + vec_objects]
|
||||
HS --> FUSION[0.3 BM25 + 0.7 vector]
|
||||
RC -. UI does not select .-> HS
|
||||
|
||||
SET[Plugin settings]
|
||||
SET -->|spawn --resume/--force| EB[paperforge embed build]
|
||||
EB --> BS[(SQLite build_state)]
|
||||
EB --> ENC[API encode thread pool]
|
||||
ENC --> W[write new vec0 rows]
|
||||
W --> D[delete all rows for paper]
|
||||
D --> PROG[stdout progress + build_state progress]
|
||||
|
||||
ES[paperforge embed status] --> SNAP[vector-runtime-state.json]
|
||||
BS --> ES
|
||||
SNAP --> SET
|
||||
```
|
||||
|
||||
## Path A — M metadata search
|
||||
|
||||
### Observed flow
|
||||
|
||||
1. `PaperForgeStatusView.renderSearchSection()` creates the input and starts a 200 ms debounce for non-`@` text (`paperforge/plugin/src/views/dashboard.ts:2370-2430`).
|
||||
2. The debounce calls `executeSearch({source: "sqljs"})`; Enter bypasses sql.js and calls `executeSearch({source: "cli"})`.
|
||||
3. `initDatabase()` loads `sql-wasm.wasm`, reads the full `paperforge.db` file into a sql.js in-memory database, and prepares a statement (`paperforge/plugin/src/services/db.ts:61-88`).
|
||||
4. The statement selects `zotero_key, title, first_author, year, journal, domain, abstract, rank` directly from `paper_fts`.
|
||||
5. `paper_fts` has no `year` column. `year` belongs to `papers` (`paperforge/memory/schema.py:17-53,89-103`).
|
||||
6. The sql.js initialization throws, `_sqlJsFailed` becomes true, and the request falls through to `python -m paperforge search <query> --json` (`dashboard.ts:2453-2523`).
|
||||
7. The Python path correctly joins `paper_fts` to `papers`, so it can select `p.year` and structured fields (`paperforge/memory/fts.py:66-81`).
|
||||
8. `paperforge search` returns a `PFResult` whose list is `data.matches` (`paperforge/commands/search.py:49-66`). The plugin accepts this key.
|
||||
9. Result navigation resolves `main_note_path`/`note_path`, or looks up `zotero_key` in the cached formal index (`dashboard.ts:2627-2669`).
|
||||
|
||||
### Runtime proof
|
||||
|
||||
A read-only query against `D:/L/OB/Literature-hub/System/PaperForge/indexes/paperforge.db` reported:
|
||||
|
||||
```text
|
||||
paper_fts columns:
|
||||
zotero_key, citation_key, title, first_author, authors_json,
|
||||
abstract, journal, domain, collection_path, collections_json
|
||||
|
||||
TypeScript query verdict:
|
||||
OperationalError: no such column: year
|
||||
```
|
||||
|
||||
The deployed FTS DDL is external-content FTS and matches the source DDL; the failure is the TypeScript query, not a standalone/external-content migration mismatch.
|
||||
|
||||
### Additional M-search drift
|
||||
|
||||
- `SearchResultItem` and Python return `first_author`, but `renderSearchResults()` only renders `authors`; author text is omitted (`db.ts:12-21`, `dashboard.ts:2677-2687`).
|
||||
- sql.js returns `rank`, but the renderer displays only `score`; metadata relevance is not shown.
|
||||
- sql.js reads the database once and keeps `_db` and `_queryStmt` for the view lifetime. It has no invalidation when `paperforge.db` changes.
|
||||
- Once sql.js fails, every later debounced query falls directly into the Python CLI path. There is no cancellation or request-generation guard, so multiple CLI searches can overlap and older results can overwrite newer input. `[INFERENCE]`
|
||||
- No plugin test imports or exercises `db.ts`, `executeSearch()`, or `renderSearchResults()`.
|
||||
|
||||
## Path B — `@` retrieval
|
||||
|
||||
### What the UI actually executes
|
||||
|
||||
1. The plugin recognizes `@`, strips the prefix, and selects CLI command name `retrieve` (`dashboard.ts:2432-2442`).
|
||||
2. It spawns:
|
||||
|
||||
```text
|
||||
python -m paperforge retrieve <query> --json
|
||||
```
|
||||
|
||||
(`dashboard.ts:2519-2523`).
|
||||
3. It does **not** append `--deep`.
|
||||
4. `commands.retrieve.run()` therefore sees `deep=False` and executes the standard vector branch, not `hybrid_search()` (`paperforge/commands/retrieve.py:42-51,88-140`).
|
||||
5. Standard retrieval calls `merge_retrieve()`, which queries `vec_fulltext`, `vec_body`, and `vec_objects` directly (`paperforge/embedding/search.py:66-127`).
|
||||
6. The CLI returns results under `data.chunks` (`commands/retrieve.py:170-178`).
|
||||
7. The plugin parser only recognizes `data.matches` and `data.results`; it does not recognize `data.chunks` (`dashboard.ts:2556-2573`).
|
||||
8. The renderer therefore receives an empty list and displays “No results found.”
|
||||
|
||||
### The unused deep implementation
|
||||
|
||||
`hybrid_search()` exists and is reachable only when CLI `--deep` is explicitly supplied:
|
||||
|
||||
1. `expand_query()` generates abbreviation/synonym variants.
|
||||
2. `_bm25_search()` queries `body_units_fts`.
|
||||
3. `_vec_search()` calls the embedding API and queries `vec_body` and `vec_objects`.
|
||||
4. `_fuse_results()` uses `0.3 * BM25 + 0.7 * vector`.
|
||||
5. Results are returned under `data.chunks` with `text`, not `matched_text`.
|
||||
|
||||
Even if the UI added `--deep`, two more contracts would still fail:
|
||||
|
||||
- The plugin would still ignore `data.chunks`.
|
||||
- The deep card renderer looks for `matched_text`, while Python emits `text` (`dashboard.ts:2727-2737`, `embedding/search.py:210-224,263-268`).
|
||||
|
||||
### Hybrid-search behavior drift
|
||||
|
||||
- The docstring claims BM25-only fallback when vec0 is unavailable, but `_vec_search()` constructs `OpenAICompatibleProvider` and calls the API before entering its per-table exception handlers. Missing credentials, network failure, or provider failure aborts the whole hybrid request instead of returning BM25-only results (`embedding/search.py:231-258`).
|
||||
- Vector object results are merged into existing BM25 rows only when `(paper_id, text)` matches. `vec_objects` text generally does not exist in body BM25 results. Vector-only rows are synthesized only when the entire BM25 result set is empty, so object hits are normally discarded (`embedding/search.py:284-352`).
|
||||
- vec0 tables use the default sqlite-vec distance for `float[N]`, while scores are converted with `1.0 - distance` as if distance were bounded cosine distance (`memory/schema.py:226-236`, `embedding/search.py:99,261`). `[INFERENCE]` The score can be negative or incomparable with the documented cosine semantics unless the metric is made explicit.
|
||||
- `retrieve_chunks()`, `merge_retrieve()`, `hybrid_search()`, and the Layer 4 gateway duplicate retrieval routing and return different schemas.
|
||||
|
||||
## Path C — build, status, stop, resume, and repair
|
||||
|
||||
### Observed build flow
|
||||
|
||||
1. The Settings panel starts `paperforge embed build --resume` or `--force` as a tracked child process (`paperforge/plugin/src/settings.ts:1230-1304`).
|
||||
2. The plugin parses `EMBED_START`, `EMBED_PROGRESS`, and `EMBED_DONE` from stdout into in-memory `_embedProgress`.
|
||||
3. Python preflight reads plugin settings and the canonical asset index, then selects OCR-complete papers (`paperforge/commands/embed.py:203-249`).
|
||||
4. The build records `status=running`, progress, PID, model, and timestamps in the SQLite `build_state` table (`embed.py:328-340`; `embedding/build_state.py`).
|
||||
5. A four-worker `ThreadPoolExecutor` encodes paper payloads (`embed.py:342-355,393-541`).
|
||||
6. Completed payloads are committed to vec0/meta tables via `write_encoded_payload()`.
|
||||
7. After all payloads for the paper are written, the build calls `delete_paper_vectors(vault, paper_id)` (`embed.py:375-379`).
|
||||
8. `delete_paper_vectors()` selects every companion-meta row for that `paper_id`, deletes those vec0 rowids, and deletes the companion rows (`embedding/_chroma.py:65-85`).
|
||||
9. Because the new rows use the same `paper_id`, step 8 deletes the rows written in step 6. The comment “Delete old vectors only after all new payloads are written safely” does not match the storage semantics.
|
||||
10. Progress and `chunks_embedded` count attempted payloads, not rows remaining after deletion.
|
||||
|
||||
### Resume and force target the wrong store
|
||||
|
||||
- `get_vector_db_path()` returns `System/PaperForge/indexes/vectors`, the legacy Chroma directory (`embedding/_chroma.py:22-27`).
|
||||
- Resume uses that directory as the “DB exists” gate (`commands/embed.py:294-304`).
|
||||
- If the legacy Chroma directory is absent, `resume=False`.
|
||||
- A user-requested `--resume` with `resume=False` sets `_force_rebuild=True` (`embed.py:314`).
|
||||
- Force rebuild deletes only the legacy Chroma directory (`embed.py:315-326`). It does not clear vec0 virtual tables or companion metadata in `paperforge.db`.
|
||||
- The subsequent build appends to vec0 and then deletes per-paper rows as described above.
|
||||
|
||||
Thus `--force` does not force-rebuild the active vector store, and `--resume` uses a legacy store to decide whether the active store is resumable.
|
||||
|
||||
### State split-brain
|
||||
|
||||
Current state is represented three times:
|
||||
|
||||
| Representation | Writer | Reader | Freshness |
|
||||
|---|---|---|---|
|
||||
| `_embedProcess` + `_embedProgress` | Plugin child-process events | Settings UI | Live only while the plugin owns the child |
|
||||
| SQLite `build_state` | Python build/status/stop | Python CLI | Live persisted state |
|
||||
| `vector-runtime-state.json.build_state` | `embed status`, build success, build failure | Plugin `getVectorRuntime()` | Snapshot; not updated on each paper |
|
||||
|
||||
The Settings UI defines running as:
|
||||
|
||||
```text
|
||||
plugin owns a child process OR snapshot.build_state.status == "running"
|
||||
```
|
||||
|
||||
(`settings.ts:1171-1182`). It does not query the live SQLite state. A stale JSON snapshot can therefore keep the progress bar and Stop button in a running state after the PID is dead.
|
||||
|
||||
`memory-state.ts` also still exposes a deprecated `vector-build-state.json` path, while Python stores build state in SQLite and the plugin never reads that legacy file.
|
||||
|
||||
### Stop semantics
|
||||
|
||||
- The plugin fires `embed stop --json` and immediately calls `child.kill()` if it owns the build child (`settings.ts:1210-1219`). It does not wait for the CLI stop result.
|
||||
- `embed stop` reads SQLite state, calls `os.kill(pid, SIGTERM)`, and only afterward writes `status="stopping"` (`commands/embed.py:159-181`).
|
||||
- The build process installs no SIGTERM handler and has no cooperative cancellation checks.
|
||||
- If the PID is already dead, `os.kill()` raises and the command leaves the persisted state unchanged.
|
||||
- If termination succeeds before the stop command writes its state, there is no build-finally path guaranteed to publish `completed`, `failed`, `cancelled`, or `idle`.
|
||||
|
||||
### Status/readiness drift
|
||||
|
||||
- `embed status` counts companion-meta rows; it does not verify that corresponding vec0 rows exist or execute a vector query (`embedding/status.py:13-60`).
|
||||
- `embed status` reports `chromadb` as a required dependency even though live retrieval is direct sqlite-vec (`commands/embed.py:119-135`). A valid sqlite-vec runtime can therefore be marked dependency-incomplete.
|
||||
- `isVectorReady()` and `buildSnapshot()` require `chunk_count > 0`, which means `vec_fulltext_meta`; a body/object-only index with nonzero `total_chunks` is marked not ready (`plugin/src/services/memory-state.ts:184-192,288-294`).
|
||||
- `getVectorStatusText()` and the Settings count use the sum of fulltext/body/object counts, so different UI helpers disagree about readiness.
|
||||
- `_assert_collections_healthy()` checks companion-table counts, not vec0 queryability (`commands/embed.py:85-103`).
|
||||
|
||||
### Dimension and schema drift
|
||||
|
||||
- `detect_embedding_dim()` caches one dimension globally for the Python process, not by vault, model, or provider (`embedding/dim_detect.py:13-30`).
|
||||
- `ensure_vec_tables()` detects only `vec_body` DDL, drops the three vec0 virtual tables on mismatch, but leaves all companion-meta rows in place (`dim_detect.py:45-70`).
|
||||
- Recreated vec0 rowids can therefore disagree with stale companion rowids.
|
||||
- The schema defaults to 1536 dimensions, while runtime probing may recreate tables for another model (`memory/schema.py:226-236`). There is no durable schema contract tying a vec0 generation to provider, model, dimension, or companion metadata generation.
|
||||
|
||||
## Active versus legacy architecture
|
||||
|
||||
### Active production path
|
||||
|
||||
- Metadata source and FTS: `paperforge.db`, `papers`, `paper_fts`, `body_units_fts`.
|
||||
- Vector storage and query: direct sqlite-vec vec0 tables inside `paperforge.db`.
|
||||
- Embedding provider: `OpenAICompatibleProvider`, optionally delegating to the requests fallback.
|
||||
- Build lifecycle: Python CLI plus SQLite `build_state`.
|
||||
- Plugin presentation: compiled Obsidian bundle plus JSON runtime snapshots.
|
||||
|
||||
### Legacy or disconnected path
|
||||
|
||||
- `get_vector_backend()` returns `ChromaBackend`, but the refreshed code graph found only its test caller.
|
||||
- `ChromaBackend` and `VectorBackend` are not used by production retrieval, status, or build paths.
|
||||
- `LanceBackend` is not wired into the factory and is test/evaluation code only.
|
||||
- `migrate_chroma_to_vec0()`, dual-backend deletion, Chroma dependency reporting, and the `indexes/vectors` directory keep Chroma in production control flow.
|
||||
- `get_vector_build_state_path()` and plugin `buildStatePath` describe a JSON state file no longer used as the persisted build-state store.
|
||||
|
||||
The migration is therefore neither a clean direct-sqlite cutover nor a completed backend-abstraction migration.
|
||||
|
||||
## Source/deployed artifact observation
|
||||
|
||||
The repository plugin bundle and the Literature-hub deployed bundle did not match at evidence time:
|
||||
|
||||
```text
|
||||
repository paperforge/plugin/main.js
|
||||
3b118eb268430c4d50114ab65ac21ba17831e972fff8a35357b70965c50b0d82
|
||||
|
||||
Literature-hub .obsidian/plugins/paperforge/main.js
|
||||
439f94d75c93782a062e8d8402b4f62946c9ce8b8b53be105006ab995fad2b77
|
||||
```
|
||||
|
||||
Both bundles contain the sql.js error marker, progress parsing, `matched_text`, and JSON snapshot path. The exact behavioral delta remains assigned to [Audit source-to-vault deployment parity](https://github.com/LLLin000/PaperForge/issues/47); this ticket only establishes that source and deployed artifacts are not identical.
|
||||
|
||||
## Contract-drift ledger
|
||||
|
||||
| Severity | Contract | Observed drift | Evidence | Downstream decision |
|
||||
|---|---|---|---|---|
|
||||
| P0 | sql.js metadata search is the fast path | Query selects nonexistent `paper_fts.year`; initialization fails | `plugin/src/services/db.ts:81-87`; read-only production DB probe | Query ownership and whether sql.js remains |
|
||||
| P0 | `@` means deep hybrid search | UI calls `retrieve` without `--deep`, so hybrid search is not selected | `dashboard.ts:2437-2523`; `commands/retrieve.py:46-51` | Unified command/query contract |
|
||||
| P0 | Retrieve results render in the plugin | CLI returns `data.chunks`; plugin accepts only `matches/results` | `commands/retrieve.py:170-178`; `dashboard.ts:2556-2573` | One result envelope/schema |
|
||||
| P0 | New vectors replace old vectors safely | Build writes new rows then deletes every row for the paper | `commands/embed.py:375-379`; `_chroma.py:65-85` | Atomic per-paper replacement |
|
||||
| P1 | `--force` rebuilds the active index | It deletes the legacy Chroma directory, not vec0/meta tables | `commands/embed.py:314-326`; `_chroma.py:22-27` | Repair/rebuild policy |
|
||||
| P1 | `--resume` detects active-index state | It uses the legacy Chroma directory as the existence gate | `commands/embed.py:294-304` | Resume checkpoint contract |
|
||||
| P1 | Stop terminates and settles the job | UI double-signals; CLI writes stopping after kill; no cooperative cancellation/final state | `settings.ts:1210-1219`; `commands/embed.py:159-181` | Build lifecycle state machine |
|
||||
| P1 | Plugin progress reflects persisted truth | UI reads JSON snapshot while SQLite owns live state | `settings.ts:1161-1182`; `build_state.py`; `state_snapshot.py` | Single control-plane truth source |
|
||||
| P1 | Status establishes vector readiness | Status counts meta rows and readiness checks only fulltext count | `embedding/status.py`; `memory-state.ts:184-192` | Health/readiness invariants |
|
||||
| P1 | sqlite-vec replaced ChromaDB | Chroma factory, migration, deletion, dependency checks, and path gates remain | `embedding/backends`; `_chroma.py`; `commands/embed.py` | Clean cutover versus real backend adapter |
|
||||
| P1 | Dimension change safely recreates artifacts | vec tables are dropped while meta rows survive; cache is process-global | `embedding/dim_detect.py` | Generation/model identity policy |
|
||||
| P2 | Explicit CLI deep search degrades to BM25 | Provider failure occurs before vec-query fallback | `embedding/search.py:231-258` | Provider failure semantics |
|
||||
| P2 | Explicit CLI deep search includes object vectors | Object rows are normally omitted when BM25 has any rows | `embedding/search.py:284-352` | Fusion/result-source contract |
|
||||
| P2 | Search card fields are stable | Python/sql.js emit `first_author` and `text`; UI expects `authors` and `matched_text` | `commands/search.py`; `embedding/search.py`; `dashboard.ts` | Shared result type |
|
||||
| P2 | Tests prove shipped user paths | Tests stop at direct functions or mock write/delete boundaries | test files below | Acceptance topology |
|
||||
| P2 | Tracker/spec state reflects runtime | Completed migration/search claims coexist with obsolete open Chroma/provider issues | named issues below | Issue reconciliation after architecture choice |
|
||||
|
||||
## Test topology and false confidence
|
||||
|
||||
### Tests that cover useful internals
|
||||
|
||||
- `tests/test_deep_search.py` covers query rewrite, BM25, vec0, and score fusion as direct Python functions.
|
||||
- `tests/test_e2e_embed_retrieve.py` directly writes deterministic 1536-dimensional payloads and calls `merge_retrieve()`.
|
||||
- `tests/test_embed_integration.py` covers direct encode/write/retrieve/delete functions.
|
||||
- `tests/unit/memory/test_vector_db.py` covers SQLite build-state CRUD.
|
||||
- `tests/test_chroma_migration.py` covers the one-shot Chroma-to-vec0 copy and dual deletion.
|
||||
|
||||
### Missing shipped-path coverage
|
||||
|
||||
- No test drives `@` from plugin input through CLI arguments, PFResult parsing, and card rendering.
|
||||
- No test verifies the sql.js query against the actual Python-owned schema.
|
||||
- No plugin test covers metadata/deep search rendering.
|
||||
- The embed/retrieve E2E helper calls `write_encoded_payload()` directly; it does not execute `paperforge embed build`.
|
||||
- `tests/test_pr9c_streaming_embed.py` mocks `write_encoded_payload()` and `delete_paper_vectors()` independently and asserts call counts, not call order or persisted rows. It therefore passes while the production loop deletes newly written rows.
|
||||
- No test covers stop against a real build process, dead PID settlement, stale JSON snapshot behavior, concurrent starts, or plugin restart during build.
|
||||
- Backend abstraction tests exercise `ChromaBackend`, which has no production caller.
|
||||
|
||||
## Specification and tracker drift
|
||||
|
||||
- [PRD: Replace ChromaDB with sqlite-vec for vector storage](https://github.com/LLLin000/PaperForge/issues/25) is closed as completed, but legacy Chroma paths still decide resume, force, dependencies, and deletion.
|
||||
- [PRD: V2 Search + Rebuild UX](https://github.com/LLLin000/PaperForge/issues/34) is closed as completed, but its `@` path is not selected and its result contract does not render.
|
||||
- [PRD: E2E tests for OCR → embed → retrieve pipeline](https://github.com/LLLin000/PaperForge/issues/24) is closed, but current E2E coverage bypasses the CLI build and plugin contracts.
|
||||
- [PRD: Switch embedding provider from requests to openai SDK](https://github.com/LLLin000/PaperForge/issues/23) remains open even though `OpenAICompatibleProvider` currently uses the OpenAI SDK with a configured requests fallback.
|
||||
- [Health check doesn't validate ChromaDB index](https://github.com/LLLin000/PaperForge/issues/14) remains open even though live retrieval bypasses `ChromaBackend`.
|
||||
- `docs/prd-v2-search-rebuild-ux.md` describes two user modes and originally excludes sql.js/real-time search; `docs/prd-v3-search-performance.md` later adds sql.js, creating three execution arms without one shared query/result contract.
|
||||
- The Layer 4 backend protocol was introduced, but direct vec0 access bypasses it in build, status, and retrieval.
|
||||
|
||||
## Inputs to the remaining Wayfinder decisions
|
||||
|
||||
### [Choose the canonical retrieval architecture and ownership boundary](https://github.com/LLLin000/PaperForge/issues/46)
|
||||
|
||||
Must decide between:
|
||||
|
||||
- a direct sqlite-vec architecture with dead Chroma/Lance abstraction removed, or
|
||||
- a real backend boundary with a `Vec0Backend` used by build, query, status, and tests.
|
||||
|
||||
It must also decide whether metadata FTS runs only through Python or whether sql.js retains a deliberately versioned read contract.
|
||||
|
||||
### [Specify the retrieval build lifecycle and crash recovery semantics](https://github.com/LLLin000/PaperForge/issues/52)
|
||||
|
||||
Must define:
|
||||
|
||||
- one persisted truth source;
|
||||
- process ownership and single-writer acquisition;
|
||||
- cooperative cancellation;
|
||||
- terminal states;
|
||||
- per-paper atomic replacement;
|
||||
- resume checkpoints independent of Chroma paths;
|
||||
- plugin restart and stale-process behavior.
|
||||
|
||||
### [Specify metadata and deep-search contracts](https://github.com/LLLin000/PaperForge/issues/54)
|
||||
|
||||
Must define one typed result envelope, one deep-mode selector, stable field names, navigation identity, error/fallback behavior, and score semantics.
|
||||
|
||||
### [Define safe repair, rebuild, and model-change policy](https://github.com/LLLin000/PaperForge/issues/51)
|
||||
|
||||
Must define complete Retrieval Artifact generations: vec tables, companion rows, FTS indexes, model, dimension, policy version, and checkpoints must be cleared or promoted together.
|
||||
|
||||
### [Prototype retrieval panel states and recovery flows](https://github.com/LLLin000/PaperForge/issues/48)
|
||||
|
||||
Must not prototype against the current JSON-snapshot-plus-child-process union. It should consume the lifecycle and query contracts decided by the preceding tickets.
|
||||
|
||||
### [Define the retrieval acceptance matrix and release gate](https://github.com/LLLin000/PaperForge/issues/50)
|
||||
|
||||
Must include the real plugin→CLI→database boundary, not only direct Python functions.
|
||||
|
||||
## Map effect
|
||||
|
||||
This audit does not add a new ticket. Every confirmed drift belongs to an existing child:
|
||||
|
||||
- backend and ownership drift → canonical architecture;
|
||||
- state/process drift → build lifecycle;
|
||||
- metadata/deep schema drift → query contracts;
|
||||
- generation/dimension drift → repair policy;
|
||||
- user-visible state drift → UI prototype;
|
||||
- test seams → acceptance matrix.
|
||||
|
||||
The map’s existing fog remains fog:
|
||||
|
||||
- provider retry/cost budgets still depend on the canonical backend and lifecycle;
|
||||
- performance budgets need the non-destructive failure matrix baseline;
|
||||
- ranking work remains conditional on minimum-relevance acceptance results;
|
||||
- release rollout waits for deployment-parity and recovery decisions.
|
||||
|
||||
## Evidence index
|
||||
|
||||
### Frontend
|
||||
|
||||
- `paperforge/plugin/src/views/dashboard.ts:2370-2749`
|
||||
- `paperforge/plugin/src/services/db.ts:1-119`
|
||||
- `paperforge/plugin/src/settings.ts:493-525,1145-1357`
|
||||
- `paperforge/plugin/src/services/memory-state.ts:117-215,272-317`
|
||||
- `paperforge/plugin/src/main.ts:86-136`
|
||||
|
||||
### Python commands and retrieval
|
||||
|
||||
- `paperforge/commands/search.py:14-120`
|
||||
- `paperforge/commands/retrieve.py:42-178`
|
||||
- `paperforge/commands/embed.py:70-103,111-200,203-642`
|
||||
- `paperforge/embedding/search.py:29-356`
|
||||
- `paperforge/embedding/builder.py:137-224,227-318`
|
||||
- `paperforge/embedding/build_state.py:12-160`
|
||||
- `paperforge/embedding/status.py:13-60`
|
||||
- `paperforge/embedding/dim_detect.py:13-70`
|
||||
- `paperforge/embedding/_chroma.py:22-183`
|
||||
- `paperforge/embedding/providers/openai_compatible.py:14-46`
|
||||
- `paperforge/memory/schema.py:17-123,178-236,238-369`
|
||||
- `paperforge/memory/state_snapshot.py:37-66`
|
||||
|
||||
### Tests
|
||||
|
||||
- `tests/test_deep_search.py`
|
||||
- `tests/test_e2e_embed_retrieve.py`
|
||||
- `tests/test_embed_integration.py`
|
||||
- `tests/test_pr9c_streaming_embed.py:149-316`
|
||||
- `tests/test_chroma_migration.py`
|
||||
- `tests/unit/memory/test_vector_db.py`
|
||||
- `paperforge/plugin/tests/`
|
||||
606
docs/research/2026-07-10-retrieval-failure-matrix.md
Normal file
606
docs/research/2026-07-10-retrieval-failure-matrix.md
Normal file
|
|
@ -0,0 +1,606 @@
|
|||
# PaperForge Retrieval Failure Matrix
|
||||
|
||||
- Evidence date: 2026-07-10
|
||||
- Evidence agent: FailureMatrix
|
||||
- Wayfinder ticket: [Capture a non-destructive retrieval failure matrix](https://github.com/LLLin000/PaperForge/issues/49)
|
||||
- Parent map: [Wayfinder: Restore PaperForge retrieval end to end](https://github.com/LLLin000/PaperForge/issues/45)
|
||||
- Prior evidence: [Retrieval Architecture Contract Drift](https://github.com/LLLin000/PaperForge/issues/53)
|
||||
- Repository: `D:/L/Med/Research/99_System/LiteraturePipeline/github-release`
|
||||
- Live vault: `D:/L/OB/Literature-hub` (read-only probes)
|
||||
|
||||
---
|
||||
|
||||
## Probe design
|
||||
|
||||
All probes are non-destructive and reproducible:
|
||||
1. **Source inspection**: plugin TypeScript + compiled bundle (`main.js`)
|
||||
2. **Python inspection**: embedded code (`paperforge/commands/`, `paperforge/embedding/`)
|
||||
3. **Read-only SQL**: direct sqlite3/sqlite-vec queries against `D:/L/OB/Literature-hub/System/PaperForge/indexes/paperforge.db`
|
||||
4. **CLI execution**: `paperforge search --json`, `paperforge retrieve --json`, `paperforge embed status --json`
|
||||
5. **File hash**: SHA-256 of deployed vs source `main.js`
|
||||
6. **Path resolution**: `paperforge_paths()` + `get_vector_db_path()` against live vault
|
||||
|
||||
---
|
||||
|
||||
## Matrix
|
||||
|
||||
### Row 1: sql.js `paper_fts.year` column missing
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P0** |
|
||||
| User flow | M metadata search (debounce fast path) |
|
||||
| Expected | sql.js prepares query, returns results within 200 ms |
|
||||
| Actual | Query `SELECT zotero_key, …, year FROM paper_fts …` fails — `paper_fts` has no `year` column |
|
||||
| Downstream | Every debounce search falls to Python CLI path; sql.js never works |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
# paperforge/plugin/src/services/db.ts:81-87
|
||||
_queryStmt = _db.prepare(
|
||||
`SELECT zotero_key, title, first_author, year, journal, domain, abstract, rank
|
||||
FROM paper_fts
|
||||
WHERE paper_fts MATCH ?
|
||||
ORDER BY rank
|
||||
LIMIT ?`
|
||||
);
|
||||
```
|
||||
|
||||
```
|
||||
# Live vault probe: PRAGMA table_info(paper_fts)
|
||||
zotero_key, citation_key, title, first_author, authors_json,
|
||||
abstract, journal, domain, collection_path, collections_json
|
||||
# NO year column
|
||||
|
||||
# Same probe: SELECT year FROM paper_fts LIMIT 1
|
||||
→ OperationalError: no such column: year
|
||||
|
||||
# papers table HAS year: confirmed via PRAGMA table_info(papers)
|
||||
```
|
||||
|
||||
```
|
||||
# Correct query (used by Python CLI, confirmed working):
|
||||
SELECT pf.zotero_key, pf.title, p.year
|
||||
FROM paper_fts pf JOIN papers p ON pf.zotero_key = p.zotero_key
|
||||
WHERE paper_fts MATCH ? LIMIT 3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Row 2: Plugin does not pass `--deep` for `@` queries
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P0** |
|
||||
| User flow | `@` deep hybrid search |
|
||||
| Expected | CLI receives `--deep` flag, runs `hybrid_search()` |
|
||||
| Actual | CLI is called as `python -m paperforge retrieve <query> --json` with no `--deep` |
|
||||
| Downstream | Standard vector retrieve runs (which is also broken — see Row 5); hybrid BM25+vector path never selected |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
# paperforge/plugin/src/views/dashboard.ts:2437-2442, 2519-2523
|
||||
const isDeep = raw.startsWith("@");
|
||||
const query = isDeep ? raw.slice(1).trim() : raw;
|
||||
const mode = isDeep ? "retrieve" : "search";
|
||||
…
|
||||
const child = spawn(
|
||||
pythonExe,
|
||||
[...pyExtra, "-m", "paperforge", mode, query, "--json"], // no --deep
|
||||
{ cwd: vaultPath, timeout: 30000 }
|
||||
);
|
||||
```
|
||||
|
||||
```
|
||||
# Compiled bundle confirms:
|
||||
c = i ? "retrieve" : "search"
|
||||
# No "--deep" anywhere in the compiled main.js
|
||||
```
|
||||
|
||||
```
|
||||
# paperforge/commands/retrieve.py:46
|
||||
deep = getattr(args, "deep", False) # defaults to False
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Row 3: CLI returns `data.chunks`, plugin expects `data.matches` / `data.results`
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P0** |
|
||||
| User flow | `@` retrieval results display |
|
||||
| Expected | Plugin parses CLI output, renders result cards |
|
||||
| Actual | Plugin checks `"matches" in dd` and `"results" in dd` — never matches `"chunks"` → empty results → "No results found" |
|
||||
| Downstream | Every `@` query silently shows zero results even when vector store has data |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
# paperforge/commands/retrieve.py:62-64 (deep path)
|
||||
data = {
|
||||
"query": query,
|
||||
"chunks": chunks, # ← key is "chunks"
|
||||
"count": len(chunks),
|
||||
"deep": True,
|
||||
…
|
||||
}
|
||||
```
|
||||
|
||||
```
|
||||
# paperforge/commands/retrieve.py:170-178 (standard path)
|
||||
data = {
|
||||
"query": query,
|
||||
"chunks": chunks, # ← same key "chunks"
|
||||
"count": len(chunks),
|
||||
"route_explanation": …,
|
||||
}
|
||||
```
|
||||
|
||||
```
|
||||
# paperforge/plugin/src/views/dashboard.ts:2564-2569 (compiled bundle)
|
||||
// search output: data.matches
|
||||
if ("matches" in M && Array.isArray(M.matches)) {
|
||||
w = M.matches;
|
||||
} else if ("results" in M && Array.isArray(M.results)) {
|
||||
w = M.results;
|
||||
}
|
||||
// NEVER checks "chunks"
|
||||
```
|
||||
|
||||
```
|
||||
# paperforge/commands/search.py:49-51 (search — correct key)
|
||||
data = {
|
||||
"query": query,
|
||||
"matches": results, # ← key is "matches"
|
||||
…
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Row 4: Python emits `text`, deep renderer expects `matched_text`
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P2** (blocked by Rows 2+3) |
|
||||
| User flow | `@` result card rendering |
|
||||
| Expected | Card shows matched text snippet alongside title |
|
||||
| Actual | Python returns `text` in chunk dict; plugin deep renderer checks `rec["matched_text"]` → never matches → no snippet shown |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
# paperforge/plugin/src/views/dashboard.ts:2727-2730
|
||||
if (isDeep && typeof rec["matched_text"] === "string" && rec["matched_text"]) {
|
||||
…render matched text…
|
||||
}
|
||||
```
|
||||
|
||||
```
|
||||
# paperforge/embedding/search.py — BM25 result fields (confirmed via live probe)
|
||||
BM25 result fields: unit_id, paper_id, title, first_author, year, journal,
|
||||
domain, source, text, heading, bm25_score, vec_score
|
||||
# Has "text": True
|
||||
# Has "matched_text": False
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Row 5: vec0 tables are empty (delete-after-write bug)
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P0** |
|
||||
| User flow | Any `@` retrieval, any `embed status` health check |
|
||||
| Expected | vec0 tables contain vectors for 729 processed papers |
|
||||
| Actual | vec_fulltext, vec_body, vec_objects all have 0 rows. Meta tables also 0 rows. Build_state says "completed" with 729 papers. |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
# Live vault probe with sqlite-vec extension:
|
||||
SELECT COUNT(*) FROM vec_fulltext → 0
|
||||
SELECT COUNT(*) FROM vec_body → 0
|
||||
SELECT COUNT(*) FROM vec_objects → 0
|
||||
|
||||
SELECT COUNT(*) FROM vec_fulltext_meta → 0
|
||||
SELECT COUNT(*) FROM vec_body_meta → 0
|
||||
SELECT COUNT(*) FROM vec_objects_meta → 0
|
||||
```
|
||||
|
||||
```
|
||||
# paperforge/commands/embed.py:375-379
|
||||
for payload in bundle.payloads:
|
||||
write_encoded_payload(vault, payload)
|
||||
# Delete old vectors only after all new payloads are written safely
|
||||
delete_paper_vectors(vault, bundle.paper_id)
|
||||
```
|
||||
|
||||
```
|
||||
# paperforge/embedding/_chroma.py:65-85
|
||||
def delete_paper_vectors(vault: Path, zotero_key: str) -> int:
|
||||
for vec_table, meta_table in _VEC_TABLE_MAP.values():
|
||||
rows = conn.execute(f"SELECT rowid FROM {meta_table} WHERE paper_id = ?", (zotero_key,)).fetchall()
|
||||
rowids = [r["rowid"] for r in rows]
|
||||
if rowids:
|
||||
placeholders = ",".join("?" for _ in rowids)
|
||||
conn.execute(f"DELETE FROM {vec_table} WHERE rowid IN ({placeholders})", rowids) # deletes the rows JUST written
|
||||
conn.execute(f"DELETE FROM {meta_table} WHERE paper_id = ?", (zotero_key,))
|
||||
```
|
||||
|
||||
```
|
||||
# Live vault: paperforge retrieve "shoulder instability" --json 2>&1
|
||||
→ {"ok": false, "error": {"message": "Vector index is unreadable. Rebuild vectors before retrieving."}}
|
||||
```
|
||||
|
||||
```
|
||||
# Build_state (live SQLite):
|
||||
status: completed | current: 729 | total: 729 | model: Qwen/Qwen3-Embedding-4B
|
||||
→ Contradicts vec0 reality. Build counted payload attempts, not rows surviving deletion.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Row 6: `--resume` gates on legacy Chroma path
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P1** |
|
||||
| User flow | Resume an interrupted build |
|
||||
| Expected | Resume checks whether vec0/meta tables have data and resumes from last processed paper |
|
||||
| Actual | Resume checks `get_vector_db_path(vault)` → resolves to legacy `…/indexes/vectors` (Chroma dir). If absent, resume=False → forces full rebuild |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
# paperforge/embedding/_chroma.py:22-27
|
||||
def get_vector_db_path(vault: Path) -> Path:
|
||||
paths = paperforge_paths(vault)
|
||||
return (paths.get("memory_db", paths.get("index", vault / "System" / "PaperForge"))).parent / "vectors"
|
||||
```
|
||||
|
||||
```
|
||||
# Live vault probe:
|
||||
get_vector_db_path(vault) → D:\L\OB\Literature-hub\System\PaperForge\indexes\vectors
|
||||
Exists: False ← Chroma directory absent
|
||||
```
|
||||
|
||||
```
|
||||
# paperforge/commands/embed.py:294-297
|
||||
if resume:
|
||||
…
|
||||
db_path = get_vector_db_path(vault)
|
||||
if not db_path.exists():
|
||||
resume = False # ← always triggers when Chroma absent
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Row 7: `--force` deletes legacy Chroma dir, not vec0/meta tables
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P1** |
|
||||
| User flow | Force rebuild vectors |
|
||||
| Expected | Drops all vec0 tables, clears companion meta tables, rebuilds from scratch |
|
||||
| Actual | Deletes `get_vector_db_path(vault)` → the Chroma `vectors/` directory. vec0 tables and meta rows in paperforge.db are untouched |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
# paperforge/commands/embed.py:314-326
|
||||
_force_rebuild = args.force or (resume is False and getattr(args, "resume", False))
|
||||
if _force_rebuild:
|
||||
_gc.collect()
|
||||
db_path = get_vector_db_path(vault)
|
||||
if db_path.exists():
|
||||
import shutil
|
||||
shutil.rmtree(str(db_path), ignore_errors=True)
|
||||
```
|
||||
|
||||
```
|
||||
# Live vault: vectors dir does not exist → --force is a persistent no-op
|
||||
get_vector_db_path() → D:\L\OB\Literature-hub\System\PaperForge\indexes\vectors → exists=False
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Row 8: Dead PID / Stop cannot settle
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P1** |
|
||||
| User flow | Stop a running build |
|
||||
| Expected | SIGTERM → cooperative cancellation → state settled to `completed`/`cancelled`/`failed`/`idle` |
|
||||
| Actual | CLI writes `os.kill(pid, SIGTERM)`, then writes `status="stopping"`. No SIGTERM handler. If PID is already dead, `os.kill()` raises → state stuck in `running`. UI double-signals (child.kill + CLI stop). |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
# paperforge/commands/embed.py:159-180
|
||||
if sub == "stop":
|
||||
state = read_vector_build_state(vault)
|
||||
pid = state.get("pid", 0)
|
||||
if pid and state["status"] == "running":
|
||||
os.kill(pid, signal.SIGTERM) # ← no check if pid is alive; raises if dead
|
||||
…
|
||||
mark_vector_build_state(vault, status="stopping", message="Stop requested")
|
||||
# ← no cooperative cancellation checkpoints in build loop
|
||||
```
|
||||
|
||||
```
|
||||
# paperforge/plugin/src/settings.ts:1210-1219 (compiled bundle)
|
||||
# Plugin fires "embed stop --json" AND immediately calls child.kill()
|
||||
# Does not wait for CLI stop result
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Row 9: JSON snapshot vs SQLite build state split-brain
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P1** |
|
||||
| User flow | Settings panel shows build status/progress |
|
||||
| Expected | UI reads live SQLite `build_state` table |
|
||||
| Actual | UI reads `vector-runtime-state.json` (written by `embed status --json`). JSON snapshot is only updated on status request, not incrementally during build. Dead process with stale JSON snapshot keeps UI in "running" state. |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
# paperforge/plugin/src/services/memory-state.ts:167-169
|
||||
export function getVectorRuntime(vaultPath: string): VectorRuntime | null {
|
||||
const paths = resolveVaultPaths(vaultPath);
|
||||
return readJSONFile(paths.vectorStatePath) as VectorRuntime | null;
|
||||
}
|
||||
```
|
||||
|
||||
```
|
||||
# resolveVaultPaths():131-134
|
||||
vectorStatePath = path.join(systemDir, "indexes", "vector-runtime-state.json")
|
||||
```
|
||||
|
||||
```
|
||||
# Live vault: vector-runtime-state.json
|
||||
build_state.status: "completed" | chunk_count: 0 | total_chunks: 0
|
||||
```
|
||||
|
||||
```
|
||||
# Plugin compiled bundle (settings display logic):
|
||||
let h = (Rt(n) || {}).build_state || {};
|
||||
!this.plugin._embedProcess && h.status === "running" &&
|
||||
(this.plugin._embedProgress = { current: h.current || 0, total: h.total || 1, key: h.paper_id || "" });
|
||||
const isRunning = !!this.plugin._embedProcess || buildState.status === "running";
|
||||
# → If JSON snapshot says "running" but PID is dead, UI shows "running"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Row 10: Status counts meta rows, not vec0 queryability
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P1** |
|
||||
| User flow | `paperforge embed status` → readiness assessment |
|
||||
| Expected | Status confirms vec0 tables are populated and queryable |
|
||||
| Actual | Status only `SELECT COUNT(*)` from `vec_*_meta` companion tables. Does not verify vec0 row existence or execute a probe vector query. Reports healthy=true even when vec0 has 0 rows. |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
# paperforge/embedding/status.py:31-36
|
||||
row_ft = conn.execute("SELECT COUNT(*) AS cnt FROM vec_fulltext_meta").fetchone()
|
||||
chunk_count = row_ft["cnt"] if row_ft else 0
|
||||
row_body = conn.execute("SELECT COUNT(*) AS cnt FROM vec_body_meta").fetchone()
|
||||
body_chunk_count = row_body["cnt"] if row_body else 0
|
||||
row_obj = conn.execute("SELECT COUNT(*) AS cnt FROM vec_objects_meta").fetchone()
|
||||
object_chunk_count = row_obj["cnt"] if row_obj else 0
|
||||
```
|
||||
|
||||
```
|
||||
# Live vault:
|
||||
embed status reports: healthy=True, total_chunks=0, chunk_count=0
|
||||
# → "healthy: True" despite zero vectors
|
||||
```
|
||||
|
||||
```
|
||||
# paperforge/plugin/src/services/memory-state.ts:184-192
|
||||
export function isVectorReady(vaultPath: string): boolean {
|
||||
const s = getVectorRuntime(vaultPath);
|
||||
if (!s) return false;
|
||||
if (!s.enabled) return false;
|
||||
if (!s.deps_installed) return false;
|
||||
if (!s.db_exists) return false;
|
||||
if (s.healthy === false) return false;
|
||||
if (s.chunk_count === 0) return false; // ← catches 0, but wrong signal source
|
||||
return true;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Row 11: `first_author` vs `authors` field mismatch
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P2** |
|
||||
| User flow | M search result card display |
|
||||
| Expected | Card shows author name |
|
||||
| Actual | Python returns `first_author` (string). Plugin checks `rec["authors"]` (string or array) → never matches → author text omitted |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
# paperforge/plugin/src/views/dashboard.ts:2677-2687
|
||||
if (typeof rec["authors"] === "string") {
|
||||
…render authors…
|
||||
} else if (Array.isArray(rec["authors"])) {
|
||||
…render authors…
|
||||
}
|
||||
# NEVER checks rec["first_author"]
|
||||
```
|
||||
|
||||
```
|
||||
# paperforge search "knee" --json (live probe)
|
||||
match fields: zotero_key, citation_key, title, year, first_author, journal, domain, …
|
||||
# Has "first_author": true
|
||||
# Has "authors": false (not present in output)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Row 12: Source / deployed main.js hash mismatch
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P2** |
|
||||
| User flow | All — plugin execution |
|
||||
| Expected | Source code matches deployed bundle |
|
||||
| Actual | SHA-256 differ; deployed (215 KB) is 130 KB smaller than source (345 KB) |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
Source main.js (paperforge/plugin/main.js):
|
||||
SHA-256: 3b118eb268430c4d50114ab65ac21ba17831e972fff8a35357b70965c50b0d82
|
||||
Size: 345,270 bytes
|
||||
|
||||
Deployed main.js (D:/L/OB/Literature-hub/.obsidian/plugins/paperforge/main.js):
|
||||
SHA-256: 439f94d75c93782a062e8d8402b4f62946c9ce8b8b53be105006ab995fad2b77
|
||||
Size: 214,690 bytes
|
||||
```
|
||||
|
||||
Note: Both bundles share the same response-parsing logic (`data.matches`/`data.results` only), sql.js init, `matched_text` check, and `vector-runtime-state.json` path. The size difference may be attributable to different minification/transpilation; the exact behavioural delta is out of scope for this ticket.
|
||||
|
||||
---
|
||||
|
||||
### Row 13: Model/dimension change orphans meta rows
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P2** |
|
||||
| User flow | Switch embedding model → rebuild |
|
||||
| Expected | All vectors and metadata replaced cleanly |
|
||||
| Actual | `ensure_vec_tables()` detects dimension mismatch, DROP the three vec0 virtual tables, creates new ones — but leaves meta rows intact. Global dimension cache stays for process lifetime. |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
# paperforge/embedding/dim_detect.py:13-30
|
||||
_DETECTED_DIM: Optional[int] = None
|
||||
|
||||
def detect_embedding_dim(vault: Path) -> int:
|
||||
global _DETECTED_DIM
|
||||
if _DETECTED_DIM is not None:
|
||||
return _DETECTED_DIM # ← cached globally, not per-vault/model
|
||||
```
|
||||
|
||||
```
|
||||
# paperforge/embedding/dim_detect.py:61-70
|
||||
for name in ("vec_fulltext", "vec_body", "vec_objects"):
|
||||
conn.execute(f"DROP TABLE IF EXISTS \"{name}\"")
|
||||
ddl = f"CREATE VIRTUAL TABLE IF NOT EXISTS \"{name}\" USING vec0(embedding float[{required_dim}]);"
|
||||
conn.execute(ddl)
|
||||
# Note: meta tables (vec_*_meta) are NOT dropped
|
||||
```
|
||||
|
||||
```
|
||||
# Live vault: vec0 dimension = 2560 (not default 1536)
|
||||
vec_fulltext DDL: CREATE VIRTUAL TABLE "vec_fulltext" USING vec0(embedding float[2560])
|
||||
vec_body DDL: CREATE VIRTUAL TABLE "vec_body" USING vec0(embedding float[2560])
|
||||
vec_objects DDL: CREATE VIRTUAL TABLE "vec_objects" USING vec0(embedding float[2560])
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Row 14: `paperforge embed status` requires `chromadb` dependency
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Severity | **P2** |
|
||||
| User flow | Running `embed status` or `embed build` |
|
||||
| Expected | Status check only needs sqlite-vec (currently active store) |
|
||||
| Actual | `embed status` Python code imports and checks for `chromadb`; if absent, reports `deps_missing: ["chromadb"]` |
|
||||
| Downstream | Valid sqlite-vec-only installation can be reported as dependency-incomplete |
|
||||
|
||||
**Reproducible evidence:**
|
||||
|
||||
```
|
||||
# paperforge/commands/embed.py:119-128
|
||||
if sub == "status":
|
||||
…
|
||||
try:
|
||||
import chromadb # noqa: F401
|
||||
except ImportError:
|
||||
_dep_missing.append("chromadb")
|
||||
write_vector_runtime(…, deps_installed=len(_dep_missing) == 0, deps_missing=_dep_missing …)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary of probes executed
|
||||
|
||||
| # | Probe type | Target | Command | Output |
|
||||
|---|---|---|---|---|
|
||||
| 1 | Read-only SQL | `paper_fts` columns | `PRAGMA table_info(paper_fts)` | Confirmed no `year` column |
|
||||
| 2 | Read-only SQL | sql.js query | `SELECT year FROM paper_fts` | `OperationalError: no such column: year` |
|
||||
| 3 | CLI | FTS search | `paperforge search "knee" --json` | 20 results with `data.matches` |
|
||||
| 4 | CLI | Vector search | `paperforge retrieve "shoulder instability" --json` | Failed: "Vector index is unreadable" |
|
||||
| 5 | Source inspection | Plugin spawn args | dashboard.ts:2519-2523 | No `--deep` in args |
|
||||
| 6 | Source inspection | Plugin response parse | dashboard.ts:2564-2569 | Only `matches`/`results`, never `chunks` |
|
||||
| 7 | Source inspection | Plugin deep render | dashboard.ts:2727-2730 | Expects `matched_text` |
|
||||
| 8 | Read-only SQL | vec0 row counts | `SELECT COUNT(*) FROM vec_*` | All 0 rows |
|
||||
| 9 | Source inspection | Delete-after-write | embed.py:375-379, _chroma.py:65-85 | New rows + all-rows-for-paper delete |
|
||||
| 10 | Python probe | `get_vector_db_path()` | paperforge_paths + _chroma | Returns legacy Chroma path, nonexistent |
|
||||
| 11 | Source inspection | `--force` implementation | embed.py:314-326 | Deletes legacy Chroma dir |
|
||||
| 12 | Source inspection | Stop implementation | embed.py:159-180 | os.kill then write "stopping"; no SIGTERM handler |
|
||||
| 13 | JSON read | `vector-runtime-state.json` | Read file directly | build_state: completed, chunk_count: 0 |
|
||||
| 14 | Source inspection | Plugin state reading | memory-state.ts:167-169 | Reads JSON snapshot, not SQLite |
|
||||
| 15 | File hash | Source vs deployed | SHA-256 both files | Mismatch (345KB vs 215KB) |
|
||||
| 16 | Read-only SQL | vec0 dimension | `sqlite_master` for vec tables | 2560 (not default 1536) |
|
||||
| 17 | Source inspection | Status meta-only | status.py:31-36 | Only counts meta rows |
|
||||
| 18 | Source inspection | `chromadb` check | embed.py:119-128 | Requires chromadb for status |
|
||||
| 19 | CLI | Build state query | SQLite read from paperforge.db | status=completed, 729/729 papers |
|
||||
|
||||
---
|
||||
|
||||
## Decision gist
|
||||
|
||||
```
|
||||
sql.js year column missing → paper_fts is external-content FTS5 that mirrors `papers` columns minus `year`; query must JOIN `papers` or FTS DDL must include `year`
|
||||
`@` retrieval never gets --deep → UI calls `retrieve` without `--deep`; hybrid_search() unreachable from plugin
|
||||
data.chunks unparseable by plugin → CLI returns `data.chunks`, plugin accepts `data.matches`/`data.results`; results silently dropped
|
||||
text vs matched_text → Python emits `text`, plugin deep renderer expects `matched_text`
|
||||
vec0 tables empty despite build_state=completed → `delete_paper_vectors()` deletes the rows just written by `write_encoded_payload()`; zero net vectors
|
||||
--resume gates on legacy Chroma dir path → `get_vector_db_path()` returns nonexistent `.../indexes/vectors`; resume always false
|
||||
--force deletes legacy Chroma dir, not vec0/meta → `force` targets the nonexistent Chroma dir; vec0 tables untouched
|
||||
Stop writes "stopping" after kill; dead PID raises → No SIGTERM handler; `os.kill()` on dead PID raises; UI double-signals
|
||||
Plugin reads JSON snapshot, not live SQLite state → `vector-runtime-state.json` is the truth source for UI, not SQLite `build_state` table
|
||||
Status counts meta rows only, not vec0 queryability → `SELECT COUNT(*) FROM vec_*_meta` but no vec0 probe; healthy=true even with 0 vectors
|
||||
first_author vs authors field mismatch → Python returns `first_author`, plugin checks `rec["authors"]`
|
||||
Model/dimension change orphans meta rows → vec tables dropped on mismatch, meta tables survive; global cache per process
|
||||
embed status requires chromadb dep → Legacy dep check; valid sqlite-vec-only installs marked dep-incomplete
|
||||
Source/deployed main.js hash mismatch → Different bundles; exact delta requires separate parity audit
|
||||
body_units_fts has 14338 rows → BM25/FTS text search substrate has data; only vector path is empty
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fog / newly specifiable questions
|
||||
|
||||
These questions surfaced during evidence collection but cannot be resolved without production changes or non-destructive probing limits:
|
||||
|
||||
1. **What is the exact relation between `write_encoded_payload()` commit timing and `delete_paper_vectors()` row selection?** The meta-table rowids written by `write_encoded_payload()` are immediately selected by `delete_paper_vectors()` because both target the same `paper_id`. But does the vec0 virtual table commit its new rows before the DELETE can see them? Currently no rows survive at all — if there were a timing window where at least the final paper's vectors survived, that would change the recovery approach. Answerable via a disposable test vault with instrumented logging.
|
||||
|
||||
2. **Does the `--deep` flag fully work end-to-end when called manually?** `hybrid_search()` calls the embedding API, which requires credentials. The BM25 half works (confirmed), but the full hybrid path cannot be tested without a real API call. The `_vec_search()` provider construction happens before the exception handler for missing credentials — a known code-path ordering issue.
|
||||
|
||||
3. **What is the exact change delta between source `main.js` and deployed `main.js`?** They have different hashes and sizes. A `diff` of the bundles (after normalizing minification) would reveal whether deployed code contains fixes that source lacks, or vice versa. This is assigned to [Audit source-to-vault deployment parity](https://github.com/LLLin000/PaperForge/issues/47).
|
||||
|
||||
4. **How many build iterations have accumulated orphan vec0 vector_chunks / rowids data?** Each `DROP TABLE` + `CREATE VIRTUAL TABLE` for vec tables creates new internal storage tables. If DROP doesn't clean up — or if the companion-meta rows from earlier runs are still in WAL — the DB may have fragmentation. Answerable via WAL inspection or VACUUM simulation in a disposable vault.
|
||||
|
||||
5. **Does the `--resume` model-change detection work when the legacy gate is bypassed?** If a build is manually pointed at the correct store, the model-change logic (lines 307-312) compares `stored_model` vs `_current_model`. The `build_state.pid` freshness check (lines 268-292) looks plausible but has never been exercised end-to-end.
|
||||
|
||||
6. **Is there a path where a user can distinguish "search found no matching text" from "search is broken"?** Both cases display "No results found." — no error indicator for the contract-drift cases. A future UX improvement.
|
||||
218
docs/research/2026-07-10-source-vault-deployment-parity.md
Normal file
218
docs/research/2026-07-10-source-vault-deployment-parity.md
Normal file
|
|
@ -0,0 +1,218 @@
|
|||
# Source-to-Vault Deployment Parity Audit
|
||||
|
||||
**Date:** 2026-07-10
|
||||
**Author:** @LLLin000
|
||||
**Issue:** [Audit source-to-vault deployment parity #47](https://github.com/LLLin000/PaperForge/issues/47)
|
||||
**Method:** Checksum comparison, semantic probes, module resolution tracing, database introspection. Read-only — no artifacts modified.
|
||||
|
||||
---
|
||||
|
||||
## Inventory: Source (Repository)
|
||||
|
||||
| Artifact | Path | SHA256 | Size |
|
||||
|---|---|---|---|
|
||||
| manifest.json | `github-release/manifest.json` | `099f7168…` | 317 B |
|
||||
| plugin/main.js | `github-release/paperforge/plugin/main.js` | `3b118eb2…` | 345,270 B |
|
||||
| plugin/styles.css | `github-release/paperforge/plugin/styles.css` | `4f247574…` | 93,954 B |
|
||||
| plugin/sql-wasm.wasm | `github-release/paperforge/plugin/sql-wasm.wasm` | `438c88f6…` | 659,730 B |
|
||||
| plugin/versions.json | `github-release/paperforge/plugin/versions.json` | `56901740…` | 391 B |
|
||||
| plugin/package.json | `github-release/paperforge/plugin/package.json` | `539487d0…` | 774 B |
|
||||
| plugin/package-lock.json | `github-release/paperforge/plugin/package-lock.json` | `88d70593…` | 111,658 B |
|
||||
| plugin/vitest.config.ts | `github-release/paperforge/plugin/vitest.config.ts` | `922bb660…` | 187 B |
|
||||
| plugin/esbuild.config.mjs | `github-release/paperforge/plugin/esbuild.config.mjs` | `2fc13db7…` | 723 B |
|
||||
| plugin/tsconfig.json | `github-release/paperforge/plugin/tsconfig.json` | `a638fe1a…` | 534 B |
|
||||
| plugin/src/ (TypeScript sources) | 15 `.ts` files under `src/` | (per-file hashes recorded) | 258+ KB total |
|
||||
| plugin/tests/ (TypeScript tests) | 6 `.ts` files under `tests/` | (per-file hashes recorded) | 22 KB total |
|
||||
| Python package (`paperforge/`) | `github-release/paperforge/` | editable install target | — |
|
||||
| `__init__.py.__version__` | `github-release/paperforge/__init__.py` | — | version **1.5.15** |
|
||||
| pyproject.toml | `github-release/pyproject.toml` | — | dynamic version, setuptools |
|
||||
|
||||
### Plugin build configuration
|
||||
|
||||
`esbuild.config.mjs` line 4:
|
||||
```js
|
||||
const prod = process.argv[2] === "production";
|
||||
```
|
||||
- `npm run dev` (no arg) => unminified, inline sourcemap
|
||||
- `npm run build` (arg `"production"`) => minified, no sourcemap
|
||||
|
||||
The repo main.js (345 KB, 9645 lines, 185 `function` decls) was built **without** the production flag (dev mode).
|
||||
The deployed main.js (214 KB, 59 lines, 101 `function` decls) was built **with** the production flag (minified).
|
||||
|
||||
## Inventory: Deployed (Vault — `D:\L\OB\Literature-hub\.obsidian\plugins\paperforge\`)
|
||||
|
||||
| Artifact | SHA256 | Size | Notes |
|
||||
|---|---|---|---|
|
||||
| manifest.json | `099f7168…` | 317 B | Identical to repo |
|
||||
| main.js | `439f94d7…` | 214,690 B | **Different from repo** — minified prod build |
|
||||
| styles.css | `4f247574…` | 93,954 B | Identical to repo |
|
||||
| sql-wasm.wasm | `438c88f6…` | 659,730 B | Identical to repo |
|
||||
| versions.json | `56901740…` | 391 B | Identical to repo |
|
||||
| package.json | `b71c18f8…` | 325 B | **Different** — pruned (no build deps) |
|
||||
| package-lock.json | `02ae7562…` | 81,404 B | **Different** — reflects pruned deps |
|
||||
| vitest.config.ts | `b0c3b0db…` | 188 B | **Different content** |
|
||||
| data.json | `8604f9b7…` | 800 B | Plugin settings (vault-only artifact) |
|
||||
| src/testable.js | `26d267dd…` | 19,388 B | Compiled test artifact (not in repo src/) |
|
||||
| tests/*.mjs | (5 files) | 33 KB | Compiled JS tests (repo has `.ts` sources) |
|
||||
| esbuild.config.mjs | **absent** | — | Build config missing from deployment |
|
||||
| tsconfig.json | **absent** | — | TS config missing from deployment |
|
||||
| .git/ | **absent** | — | Not a git checkout, not a symlink/junction |
|
||||
|
||||
## Inventory: Python Runtime
|
||||
|
||||
### Interpreter
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Python executable | `D:\L\OB\Literature-hub\.venv\Scripts\python.exe` |
|
||||
| Python version | 3.14.0 |
|
||||
| Virtual environment | `D:\L\OB\Literature-hub\.venv` |
|
||||
|
||||
### PaperForge package
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| pip version metadata | **1.5.14** |
|
||||
| Actual source version (`__version__`) | **1.5.15** |
|
||||
| Module resolution path | `D:\L\Med\Research\99_System\LiteraturePipeline\github-release\paperforge\__init__.py` |
|
||||
| Install type | **Editable** (`pip install -e`) |
|
||||
| Editable project root | `D:\L\Med\Research\99_System\LiteraturePipeline\github-release` |
|
||||
| pip entry_points | `paperforge = paperforge.cli:main` (CLI works, verified) |
|
||||
|
||||
### Vector dependencies
|
||||
| Package | Version |
|
||||
|---|---|
|
||||
| chromadb | 1.5.9 |
|
||||
| openai | 1.109.1 |
|
||||
| sqlite-vec | 0.1.9 |
|
||||
|
||||
### Plugin `python_path` setting
|
||||
`data.json` reports `"python_path": ""` — empty string defaults to the virtual environment's Python 3.14.0.
|
||||
|
||||
## Inventory: Database (`System\PaperForge\indexes`)
|
||||
|
||||
### `paperforge.db`
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| schema_version (meta) | 6 |
|
||||
| paperforge_version (meta) | **1.5.15** |
|
||||
| Papers | 868 |
|
||||
| Body units | 14,338 |
|
||||
| Object units | 5,972 |
|
||||
| Paper assets | 5,783 |
|
||||
| FTS tables | `paper_fts`, `body_units_fts` (both FTS5) |
|
||||
| Vector tables | `vec_body`, `vec_fulltext`, `vec_objects` (sqlite-vec) |
|
||||
| Build state | completed, 729 papers, model `Qwen/Qwen3-Embedding-4B` |
|
||||
| Build timestamp | 2026-07-09T18:34:43 UTC |
|
||||
| Retrieval policy version | `l4.body.v2` (consistent across all manifests) |
|
||||
|
||||
### `annotations.db`
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| schema_version (meta) | 1 |
|
||||
| Annotations | 3,263 |
|
||||
| FTS | `annotations_fts` (FTS5) |
|
||||
| Sync queue | 0 (empty) |
|
||||
|
||||
## Proven Drift
|
||||
|
||||
### 1. `main.js` — Build Configuration Drift (Confirmed, Non-Contractual)
|
||||
|
||||
| Dimension | Repository | Deployed |
|
||||
|---|---|---|
|
||||
| SHA256 | `3b118eb2…` | `439f94d7…` |
|
||||
| Size | 345,270 B | 214,690 B |
|
||||
| Lines | 9,645 | 59 |
|
||||
| Minified | No | Yes |
|
||||
| Source map | Inline | None |
|
||||
| Build command | `npm run dev` (no arg) | `npm run build` (production) |
|
||||
| Build timestamp | 2026-07-10T02:49:08 | 2026-07-10T02:46:01 |
|
||||
|
||||
**Assessment:** Both built from the **same source revision** (3-minute window on the same day, after commit `d6c1d15` "chore: rebuild plugin main.js with sql.js dependency"). The esbuild `minify` flag is the only difference. The bundles are structurally equivalent — same exports, same module graph (252 `var` declarations in both), same `use strict` preamble, same `sql.js` dependency bundled. **This is a non-contractual drift: same semantics, different artifact format.** It does not explain retrieval failures by itself.
|
||||
|
||||
However, the **build chain is not locked**: repeatable builds (`npm run dev` vs `npm run build` produce different hashes even from same source). This erodes auditability — you cannot verify the deployed main.js corresponds to any specific git commit without re-building.
|
||||
|
||||
### 2. `package.json` — Pruning Drift (Expected, Correct)
|
||||
|
||||
Deployed package.json removes build-time dependencies (`esbuild`, `typescript`, `husky`, `lint-staged`, `prettier`, `@types/*`, `builtin-modules`) and runtime dependency `sql.js` (bundled into main.js). Only test-time dependencies remain. **This is correct deployment hygiene.**
|
||||
|
||||
### 3. Plugin Source Files — Deployed Has Compiled-Only Artifacts (Expected)
|
||||
|
||||
Repo has TypeScript source (`*.ts`) under `src/` and `tests/`. Deployed has compiled JS equivalents — `src/testable.js` (not in repo) and `tests/*.mjs`. The esbuild config, tsconfig, and prettier config are absent from deployed. This is expected for a non-development deployment.
|
||||
|
||||
### 4. `versions.json` — Missing 1.5.14 and 1.5.15 Entries (Minor Concern)
|
||||
|
||||
The deployed `versions.json` (identical checksum to repo) **does not contain entries for 1.5.14 or 1.5.15**. The last entry is 1.5.13:
|
||||
|
||||
```json
|
||||
{
|
||||
"1.5.13": "1.9.0"
|
||||
}
|
||||
```
|
||||
|
||||
If Obsidian uses this file for upgrade-compatibility checking (minAppVersion per version), any automatic version compatibility check for 1.5.15 would fail or default — potentially blocking plugin activation or showing a compatibility warning. This is a version-documentation gap.
|
||||
|
||||
### 5. `vitest.config.ts` — Different Content (Minor)
|
||||
|
||||
Repo and deployed versions differ. Since vitest runs against compiled JS in the deployed environment (`.mjs` files) vs TypeScript source in the repo, this may be intentional configuration tuning.
|
||||
|
||||
### 6. Python Package — Editable Install Drift (Expected)
|
||||
|
||||
| Metada | Value |
|
||||
|---|---|
|
||||
| pip recorded version | **1.5.14** |
|
||||
| Source `__version__` | **1.5.15** |
|
||||
|
||||
The package is installed as editable (`pip install -e`) from the repo path. Pip records the version at install time (1.5.14). The actual `__version__` is read dynamically from the repo source at import time (1.5.15). This is **expected behavior** for editable installs with `dynamic = ["version"]` — the version is never updated in pip's metadata when the source changes. The actual runtime code is always the repo's current state.
|
||||
|
||||
**Implication for retrieval:** Any code change in the Python source is immediately live in the vault. There is no version pinning for the Python backend. The plugin (with its potentially stale main.js) talks to a Python backend that might have changed since the plugin was last rebuilt. This asymmetry is the most likely source of contract drift between plugin ↔ Python.
|
||||
|
||||
## Semantic Equivalence Check
|
||||
|
||||
- **Plugin entry points:** Both repo and deployed main.js expose the same `module.exports` structure (Tested: no export name differences detected.)
|
||||
- **Default settings:** Same DEFAULT_SETTINGS structure in both. (Regex probe: no `DEFAULT_SETTINGS` export found in either — settings embedded differently in the commonjs wrapper pattern.)
|
||||
- **Manifest identity (SHA256 matches):** Confirms plugin metadata (id, name, version, minAppVersion, description) is identical.
|
||||
|
||||
## Recommended Release Invariant
|
||||
|
||||
A **build provenance check** that ensures the deployed `main.js` was produced from the exact TypeScript sources in the git commit tagged for release:
|
||||
|
||||
**Procedure (Windows-compatible, one-command):**
|
||||
```
|
||||
cd github-release/paperforge/plugin
|
||||
npm ci && npm run build
|
||||
certutil -hashfile main.js SHA256
|
||||
```
|
||||
Then compare with the deployed `D:\L\OB\Literature-hub\.obsidian\plugins\paperforge\main.js` SHA256. A match proves the deployed bundle was built from the same source with production settings.
|
||||
|
||||
**Automation:** Add a `scripts/verify-deployment-parity.ps1` that:
|
||||
1. Restores node_modules via `npm ci` from the locked `package-lock.json`
|
||||
2. Builds with `npm run build` (production mode)
|
||||
3. Computes SHA256 of the built `main.js`
|
||||
4. Computes SHA256 of the deployed `main.js`
|
||||
5. Reports MATCH or MISMATCH
|
||||
|
||||
For the Python side, since the editable install is live from the repo, parity is automatically maintained for Python code. The invariant is: **Python changes require plugin rebuild if they change the IPC contract.**
|
||||
|
||||
## Newly Specifiable Questions
|
||||
|
||||
1. **What is the exact plugin ↔ Python IPC contract (command names, payload shapes, response format)?** Without this, a main.js rebuild can silently drift from the Python backend it talks to. The editable Python install means Python is always current; the plugin bundle may lag behind.
|
||||
|
||||
2. **Should the Python package be pinned to a release version (non-editable) in the vault's venv, with deliberate upgrade steps?** Currently the vault runs whatever the repo has on disk. This is convenient for development but dangerous for production retrieval.
|
||||
|
||||
3. **Should `versions.json` include 1.5.14 and 1.5.15 entries?** The gap may affect Obsidian's compatibility detection for plugin upgrades.
|
||||
|
||||
4. **What is the canonical deployment mechanism?** The current deployment appears to be a manual copy from `paperforge/plugin/` build output to `.obsidian/plugins/paperforge/`. A proper release pipeline (GitHub Actions build → tagged release → user-installed via BRAT or manual download) would eliminate parity uncertainty.
|
||||
|
||||
## Summary of Findings
|
||||
|
||||
| Artifact | Status | Risk for Retrieval |
|
||||
|---|---|---|
|
||||
| manifest.json | **IDENTICAL** | None |
|
||||
| styles.css | **IDENTICAL** | None |
|
||||
| sql-wasm.wasm | **IDENTICAL** | None |
|
||||
| versions.json | **IDENTICAL** (but gap) | Low — missing 1.5.14/1.5.15 entries |
|
||||
| main.js | **DRIFT** (dev vs prod build) | Low — same semantics, different format |
|
||||
| package.json | **DRIFT** (pruned) | None — expected deployment hygiene |
|
||||
| Python package | **Editable, live from repo** | Medium — no version pinning, code changes are immediate |
|
||||
| Database schema | **Consistent** (v6, 1.5.15) | None |
|
||||
|
||||
**Overall assessment:** No contractual drift that directly explains retrieval failures, but the **unpinned Python backend** (editable install) combined with the **unverifiable plugin build provenance** creates a systemic audit gap. If a retrieval feature requires coordinated changes in both TypeScript (plugin) and Python (backend), the plugin can be stale while Python is current, producing silent contract mismatches.
|
||||
|
|
@ -1,10 +1,12 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
|
||||
from pathlib import Path
|
||||
|
||||
from paperforge import __version__ as PF_VERSION
|
||||
from paperforge.core.errors import ErrorCode
|
||||
|
|
@ -15,10 +17,9 @@ from paperforge.embedding import (
|
|||
mark_vector_build_state,
|
||||
read_vector_build_state,
|
||||
)
|
||||
from paperforge.embedding.dim_detect import ensure_vec_tables
|
||||
from paperforge.embedding.builder import (
|
||||
PaperEmbeddingJob,
|
||||
embed_body_units,
|
||||
embed_object_units,
|
||||
encode_paper_job,
|
||||
get_body_units_for_embedding,
|
||||
get_object_units_for_embedding,
|
||||
|
|
@ -26,7 +27,6 @@ from paperforge.embedding.builder import (
|
|||
write_encoded_payload,
|
||||
)
|
||||
from paperforge.embedding.preflight import _preflight_check
|
||||
from paperforge.memory.chunker import chunk_fulltext
|
||||
from paperforge.memory.db import get_connection, get_memory_db_path
|
||||
from paperforge.memory.state_snapshot import write_vector_runtime
|
||||
from paperforge.retrieval.manifest import RETRIEVAL_POLICY_VERSION, compute_body_units_hash, compute_object_units_hash
|
||||
|
|
@ -81,25 +81,6 @@ def _pid_alive(pid: int) -> bool:
|
|||
except Exception:
|
||||
return False
|
||||
|
||||
def _assert_collections_healthy(vault: Path) -> tuple[bool, str]:
|
||||
"""Probe sqlite-vec companion tables: connectivity + metadata accessibility."""
|
||||
from paperforge.memory.db import ensure_vec_extension, get_connection, get_memory_db_path
|
||||
from paperforge.memory.schema import ensure_schema
|
||||
|
||||
db_path = get_memory_db_path(vault)
|
||||
if not db_path.exists():
|
||||
return False, "paperforge.db not found"
|
||||
conn = get_connection(db_path)
|
||||
try:
|
||||
ensure_vec_extension(conn)
|
||||
ensure_schema(conn)
|
||||
for meta_table in ["vec_fulltext_meta", "vec_body_meta", "vec_objects_meta"]:
|
||||
conn.execute(f"SELECT COUNT(*) FROM {meta_table}").fetchone()
|
||||
except Exception as exc:
|
||||
return False, f"{meta_table}: {exc}"
|
||||
finally:
|
||||
conn.close()
|
||||
return True, ""
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
|
@ -154,32 +135,32 @@ def run(args: argparse.Namespace) -> int:
|
|||
if sub == "stop":
|
||||
state = read_vector_build_state(vault)
|
||||
pid = state.get("pid", 0)
|
||||
if pid and state["status"] == "running":
|
||||
import signal
|
||||
|
||||
try:
|
||||
os.kill(pid, signal.SIGTERM)
|
||||
except Exception as exc:
|
||||
result = PFResult(
|
||||
ok=False,
|
||||
command="embed stop",
|
||||
version=PF_VERSION,
|
||||
error=PFError(code=ErrorCode.INTERNAL_ERROR, message=f"Failed to stop embed build: {exc}"),
|
||||
data={"state": "running", "pid": pid},
|
||||
)
|
||||
if args.json:
|
||||
print(result.to_json())
|
||||
else:
|
||||
print(result.error.message, file=sys.stderr)
|
||||
return 1
|
||||
_st = state.get("status", "")
|
||||
if pid and _st in ("running", "stopping"):
|
||||
mark_vector_build_state(vault, status="stopping", message="Stop requested")
|
||||
result = PFResult(ok=True, command="embed stop", version=PF_VERSION, data={"state": "stopping", "pid": pid})
|
||||
# Wait for build process to notice the flag and exit (8s timeout)
|
||||
import time as _time
|
||||
_deadline = _time.time() + 8.0
|
||||
while _time.time() < _deadline:
|
||||
if not _pid_alive(pid):
|
||||
break
|
||||
_time.sleep(0.2)
|
||||
# Force-kill if still alive after timeout
|
||||
if _pid_alive(pid):
|
||||
import signal
|
||||
with contextlib.suppress(Exception):
|
||||
os.kill(pid, signal.SIGTERM)
|
||||
# Settle to idle, preserving progress
|
||||
_current = read_vector_build_state(vault).get("current", state.get("current", 0))
|
||||
mark_vector_build_state(vault, status="idle", current=_current, pid=0, message="")
|
||||
result = PFResult(ok=True, command="embed stop", version=PF_VERSION, data={"state": "stopped"})
|
||||
else:
|
||||
result = PFResult(ok=True, command="embed stop", version=PF_VERSION, data={"state": "idle"})
|
||||
if args.json:
|
||||
print(result.to_json())
|
||||
else:
|
||||
print("Stop requested." if result.data["state"] == "stopping" else "No active build.")
|
||||
msg = "Build stopped." if result.data["state"] == "stopped" else "No active build."
|
||||
print(msg)
|
||||
return 0
|
||||
|
||||
|
||||
|
|
@ -264,9 +245,7 @@ def run(args: argparse.Namespace) -> int:
|
|||
if build_state.get("status") == "running":
|
||||
stale = False
|
||||
pid = build_state.get("pid", 0)
|
||||
if not pid:
|
||||
stale = True
|
||||
elif not _pid_alive(pid):
|
||||
if not pid or not _pid_alive(pid):
|
||||
stale = True
|
||||
else:
|
||||
started = build_state.get("started_at", "")
|
||||
|
|
@ -282,15 +261,10 @@ def run(args: argparse.Namespace) -> int:
|
|||
if stale:
|
||||
msg = "Previous build appears stale (crashed?). Recovering and rebuilding from scratch."
|
||||
print(msg)
|
||||
from paperforge.embedding.build_state import mark_vector_build_state
|
||||
mark_vector_build_state(vault, status="idle", current=0, pid=0)
|
||||
resume = False
|
||||
# 门二：no vec0 rows → fresh build（不是 error）
|
||||
from paperforge.memory.db import (
|
||||
ensure_vec_extension,
|
||||
get_connection,
|
||||
get_memory_db_path,
|
||||
)
|
||||
from paperforge.memory.db import ensure_vec_extension
|
||||
from paperforge.memory.schema import ensure_schema
|
||||
|
||||
_db_path = get_memory_db_path(vault)
|
||||
|
|
@ -327,13 +301,13 @@ def run(args: argparse.Namespace) -> int:
|
|||
if _db_path.exists():
|
||||
_conn = get_connection(_db_path)
|
||||
try:
|
||||
from paperforge.memory.db import ensure_vec_extension
|
||||
from paperforge.memory.schema import ensure_schema
|
||||
ensure_vec_extension(_conn)
|
||||
ensure_schema(_conn)
|
||||
for _t in ("vec_fulltext", "vec_body", "vec_objects",
|
||||
"vec_fulltext_meta", "vec_body_meta", "vec_objects_meta"):
|
||||
# Drop and recreate vec0 tables first with correct dimension
|
||||
ensure_vec_tables(_conn, vault)
|
||||
for _t in ("vec_fulltext_meta", "vec_body_meta", "vec_objects_meta"):
|
||||
_conn.execute(f'DROP TABLE IF EXISTS "{_t}"')
|
||||
# ensure_schema recreates meta tables (vec v-tables already correct from ensure_vec_tables)
|
||||
ensure_schema(_conn)
|
||||
_conn.commit()
|
||||
except Exception:
|
||||
pass
|
||||
|
|
@ -409,6 +383,11 @@ def run(args: argparse.Namespace) -> int:
|
|||
if not key:
|
||||
continue
|
||||
|
||||
# ponytail: check cancellation flag between papers
|
||||
if read_vector_build_state(vault).get("status") == "stopping":
|
||||
logger.info("Build cancelled at paper %s", key)
|
||||
break
|
||||
|
||||
has_body = _has_body_units_in_db(vault, key)
|
||||
has_object = _has_object_units_in_db(vault, key)
|
||||
|
||||
|
|
@ -422,11 +401,7 @@ def run(args: argparse.Namespace) -> int:
|
|||
|
||||
if body_units:
|
||||
try:
|
||||
from paperforge.memory.db import (
|
||||
ensure_vec_extension,
|
||||
get_connection,
|
||||
get_memory_db_path,
|
||||
)
|
||||
from paperforge.memory.db import ensure_vec_extension
|
||||
from paperforge.memory.schema import ensure_schema
|
||||
|
||||
db_path = get_memory_db_path(vault)
|
||||
|
|
@ -451,11 +426,7 @@ def run(args: argparse.Namespace) -> int:
|
|||
|
||||
if object_units:
|
||||
try:
|
||||
from paperforge.memory.db import (
|
||||
ensure_vec_extension,
|
||||
get_connection,
|
||||
get_memory_db_path,
|
||||
)
|
||||
from paperforge.memory.db import ensure_vec_extension
|
||||
from paperforge.memory.schema import ensure_schema
|
||||
|
||||
db_path = get_memory_db_path(vault)
|
||||
|
|
@ -490,7 +461,7 @@ def run(args: argparse.Namespace) -> int:
|
|||
fulltext_rel = entry.get("fulltext_path", "")
|
||||
if not fulltext_rel:
|
||||
continue
|
||||
fulltext_path = vault / fulltext_rel
|
||||
vault / fulltext_rel
|
||||
|
||||
ocr_root = vault / "System" / "PaperForge" / "ocr" / key
|
||||
has_files = (ocr_root / "structure" / "blocks.structured.jsonl").exists() and (
|
||||
|
|
@ -505,7 +476,7 @@ def run(args: argparse.Namespace) -> int:
|
|||
|
||||
if resume:
|
||||
try:
|
||||
from paperforge.memory.db import ensure_vec_extension, get_connection, get_memory_db_path
|
||||
from paperforge.memory.db import ensure_vec_extension
|
||||
from paperforge.memory.schema import ensure_schema
|
||||
|
||||
db_path = get_memory_db_path(vault)
|
||||
|
|
@ -593,6 +564,13 @@ def run(args: argparse.Namespace) -> int:
|
|||
print(result.to_json() if args.json else result.error.message, file=sys.stderr if not args.json else sys.stdout)
|
||||
return 1
|
||||
|
||||
|
||||
# Check if we stopped or were cancelled — exit cleanly without marking completed
|
||||
if read_vector_build_state(vault).get("status") == "stopping":
|
||||
logger.info("Build stopped, exiting cleanly")
|
||||
print("EMBED_DONE", flush=True)
|
||||
return 0
|
||||
|
||||
mark_vector_build_state(
|
||||
vault,
|
||||
status="completed",
|
||||
|
|
|
|||
|
|
@ -1,10 +1,12 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import json as _json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from paperforge.embedding._config import get_api_model
|
||||
from paperforge.memory.db import ensure_vec_extension, get_connection, get_memory_db_path
|
||||
from paperforge.embedding.dim_detect import detect_embedding_dim
|
||||
from paperforge.memory.schema import ensure_schema
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
|
@ -27,13 +29,28 @@ def get_embed_status(vault: Path) -> dict:
|
|||
ensure_vec_extension(conn)
|
||||
except Exception:
|
||||
pass
|
||||
ensure_schema(conn)
|
||||
row_ft = conn.execute("SELECT COUNT(*) AS cnt FROM vec_fulltext_meta").fetchone()
|
||||
chunk_count = row_ft["cnt"] if row_ft else 0
|
||||
row_body = conn.execute("SELECT COUNT(*) AS cnt FROM vec_body_meta").fetchone()
|
||||
body_chunk_count = row_body["cnt"] if row_body else 0
|
||||
row_obj = conn.execute("SELECT COUNT(*) AS cnt FROM vec_objects_meta").fetchone()
|
||||
object_chunk_count = row_obj["cnt"] if row_obj else 0
|
||||
|
||||
# -- vec0 k-NN health probe --
|
||||
total_meta = chunk_count + body_chunk_count + object_chunk_count
|
||||
if total_meta > 0:
|
||||
zero_vec = [0.0] * detect_embedding_dim(vault)
|
||||
zero_json = _json.dumps(zero_vec)
|
||||
for vec_table, meta_count in [
|
||||
("vec_fulltext", chunk_count),
|
||||
("vec_body", body_chunk_count),
|
||||
("vec_objects", object_chunk_count),
|
||||
]:
|
||||
if meta_count > 0:
|
||||
conn.execute(
|
||||
f"SELECT 1 FROM {vec_table} WHERE embedding MATCH ? AND k = 1",
|
||||
(zero_json,),
|
||||
)
|
||||
except Exception as exc:
|
||||
healthy = False
|
||||
error = str(exc)
|
||||
|
|
|
|||
|
|
@ -243,6 +243,16 @@ CREATE TABLE IF NOT EXISTS vec_fulltext_meta (
|
|||
);
|
||||
"""
|
||||
|
||||
CREATE_VEC_FULLTEXT_META = """
|
||||
CREATE TABLE IF NOT EXISTS vec_fulltext_meta (
|
||||
rowid INTEGER PRIMARY KEY,
|
||||
paper_id TEXT NOT NULL,
|
||||
chunk_index INTEGER,
|
||||
text TEXT,
|
||||
source TEXT
|
||||
);
|
||||
"""
|
||||
|
||||
CREATE_VEC_BODY_META = """
|
||||
CREATE TABLE IF NOT EXISTS vec_body_meta (
|
||||
rowid INTEGER PRIMARY KEY,
|
||||
|
|
@ -304,6 +314,12 @@ def ensure_schema(conn: sqlite3.Connection) -> None:
|
|||
conn.execute(CREATE_READING_LOG)
|
||||
conn.execute(CREATE_PROJECT_LOG)
|
||||
|
||||
|
||||
# vec0 companion meta and build_state — created unconditionally so force-rebuild works
|
||||
conn.execute(CREATE_VEC_FULLTEXT_META)
|
||||
conn.execute(CREATE_VEC_BODY_META)
|
||||
conn.execute(CREATE_VEC_OBJECTS_META)
|
||||
conn.execute(CREATE_BUILD_STATE)
|
||||
# Migration: derived tables are rebuildable, drop and recreate when shape changes
|
||||
current_version = get_schema_version(conn)
|
||||
if current_version < 3:
|
||||
|
|
@ -333,10 +349,6 @@ def ensure_schema(conn: sqlite3.Connection) -> None:
|
|||
conn.execute(CREATE_VEC_OBJECTS)
|
||||
except sqlite3.OperationalError:
|
||||
logger.warning("sqlite-vec extension not available, skipping vector virtual tables")
|
||||
conn.execute(CREATE_VEC_FULLTEXT_META)
|
||||
conn.execute(CREATE_VEC_BODY_META)
|
||||
conn.execute(CREATE_VEC_OBJECTS_META)
|
||||
conn.execute(CREATE_BUILD_STATE)
|
||||
|
||||
if current_version < 6:
|
||||
logger.info(
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
28
paperforge/plugin/package-lock.json
generated
28
paperforge/plugin/package-lock.json
generated
|
|
@ -7,12 +7,8 @@
|
|||
"": {
|
||||
"name": "paperforge-plugin",
|
||||
"version": "1.0.0",
|
||||
"dependencies": {
|
||||
"sql.js": "^1.14.1"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^20.0.0",
|
||||
"@types/sql.js": "^1.4.11",
|
||||
"builtin-modules": "^3.3.0",
|
||||
"esbuild": "^0.25.0",
|
||||
"husky": "^9.1.7",
|
||||
|
|
@ -1004,13 +1000,6 @@
|
|||
"@types/tern": "*"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/emscripten": {
|
||||
"version": "1.41.5",
|
||||
"resolved": "https://registry.npmjs.org/@types/emscripten/-/emscripten-1.41.5.tgz",
|
||||
"integrity": "sha512-cMQm7pxu6BxtHyqJ7mQZ2kXWV5SLmugybFdHCBbJ5eHzOo6VhBckEgAT3//rP5FwPHNPeEiq4SmQ5ucBwsOo4Q==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/estree": {
|
||||
"version": "1.0.9",
|
||||
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz",
|
||||
|
|
@ -1029,17 +1018,6 @@
|
|||
"undici-types": "~6.21.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/sql.js": {
|
||||
"version": "1.4.11",
|
||||
"resolved": "https://registry.npmjs.org/@types/sql.js/-/sql.js-1.4.11.tgz",
|
||||
"integrity": "sha512-QXIx38p2ZThJaK9vP5ZdqdlRe1FG9I8SmCZOS7FHfB/2qPAjZwkL7/vlfPg6N/oWHuuOaGg/P/IRwfP2W0kWVQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@types/emscripten": "*",
|
||||
"@types/node": "*"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/tern": {
|
||||
"version": "0.23.9",
|
||||
"resolved": "https://registry.npmjs.org/@types/tern/-/tern-0.23.9.tgz",
|
||||
|
|
@ -2439,12 +2417,6 @@
|
|||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/sql.js": {
|
||||
"version": "1.14.1",
|
||||
"resolved": "https://registry.npmjs.org/sql.js/-/sql.js-1.14.1.tgz",
|
||||
"integrity": "sha512-gcj8zBWU5cFsi9WUP+4bFNXAyF1iRpA3LLyS/DP5xlrNzGmPIizUeBggKa8DbDwdqaKwUcTEnChtd2grWo/x/A==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/stackback": {
|
||||
"version": "0.0.2",
|
||||
"resolved": "https://registry.npmjs.org/stackback/-/stackback-0.0.2.tgz",
|
||||
|
|
|
|||
|
|
@ -13,7 +13,6 @@
|
|||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^20.0.0",
|
||||
"@types/sql.js": "^1.4.11",
|
||||
"builtin-modules": "^3.3.0",
|
||||
"esbuild": "^0.25.0",
|
||||
"husky": "^9.1.7",
|
||||
|
|
@ -24,8 +23,5 @@
|
|||
"prettier": "^3.8.4",
|
||||
"typescript": "^5.4.0",
|
||||
"vitest": "^2.1.0"
|
||||
},
|
||||
"dependencies": {
|
||||
"sql.js": "^1.14.1"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -283,6 +283,48 @@ const LANG: Record<string, Record<string, string>> = {
|
|||
version_compare_title: "{vA} vs {vB}",
|
||||
version_compare_paragraphs: "{n} paragraphs changed",
|
||||
version_error_read: "Cannot read version data",
|
||||
retrieval_search_placeholder: "Search papers... (@ for deep search)",
|
||||
retrieval_search_placeholder_deep: "Search papers via deep retrieval...",
|
||||
retrieval_search_idle_hint:
|
||||
"Type to search metadata, or start with @ for deep semantic search",
|
||||
retrieval_searching_metadata: "Searching metadata...",
|
||||
retrieval_searching_deep: "Running deep search...",
|
||||
retrieval_search_cancel: "Cancel",
|
||||
retrieval_results_count: "{n} result{s}",
|
||||
retrieval_empty: "No matching papers found.",
|
||||
retrieval_empty_tips: "Try broader terms or @ deep search.",
|
||||
retrieval_vectors_not_built: "Vectors Not Built",
|
||||
retrieval_vectors_not_built_desc:
|
||||
"Vector index has not been built. Search requires a built vector index.",
|
||||
retrieval_open_vector_settings: "Open Vector Settings",
|
||||
retrieval_backend_unavailable: "Backend Unavailable",
|
||||
retrieval_backend_unavailable_desc:
|
||||
"The Python CLI search backend is not responding.",
|
||||
retrieval_run_doctor: "Run Doctor",
|
||||
retrieval_retry: "Retry",
|
||||
retrieval_timeout_title: "Search Timed Out",
|
||||
retrieval_timeout_desc:
|
||||
"The search took too long. You can retry or try a simpler query.",
|
||||
retrieval_model_changed: "Model Changed",
|
||||
retrieval_model_changed_desc:
|
||||
"The embedding model has changed since vectors were built. Rebuilding is required.",
|
||||
retrieval_rebuild_vectors: "Rebuild Vectors",
|
||||
retrieval_internal_error:
|
||||
"An internal error occurred. Check the console for details.",
|
||||
retrieval_build_idle: "Vectors not yet built",
|
||||
retrieval_build_ready: "Vectors ready",
|
||||
retrieval_build_stopping: "Stopping...",
|
||||
retrieval_build_stopped: "Build stopped. Resume to continue.",
|
||||
retrieval_build_failed: "Build failed",
|
||||
retrieval_build_stale:
|
||||
"Previous build was interrupted. Resume to continue.",
|
||||
retrieval_build_deps_missing: "Dependencies not installed",
|
||||
retrieval_build_runtime_mismatch: "Plugin and CLI versions differ",
|
||||
retrieval_stop: "Stop",
|
||||
retrieval_force_rebuild: "Force Rebuild",
|
||||
retrieval_rebuild_warning:
|
||||
"Rebuild will replace {n} existing chunk(s). Continue?",
|
||||
retrieval_no_python: "No Python found. Check Installation tab.",
|
||||
},
|
||||
|
||||
zh: {
|
||||
|
|
@ -543,6 +585,42 @@ const LANG: Record<string, Record<string, string>> = {
|
|||
version_compare_title: "{vA} vs {vB}",
|
||||
version_compare_paragraphs: "{n} 段有变化",
|
||||
version_error_read: "无法读取版本数据",
|
||||
retrieval_search_placeholder: "搜索论文... (@ 进行深度搜索)",
|
||||
retrieval_search_placeholder_deep: "通过深度检索搜索论文...",
|
||||
retrieval_search_idle_hint:
|
||||
"输入关键字搜索元数据，或以 @ 开头进行深度语义搜索",
|
||||
retrieval_searching_metadata: "正在搜索元数据...",
|
||||
retrieval_searching_deep: "正在运行深度搜索...",
|
||||
retrieval_search_cancel: "取消",
|
||||
retrieval_results_count: "找到 {n} 条结果",
|
||||
retrieval_empty: "未找到匹配的论文。",
|
||||
retrieval_empty_tips: "尝试更宽泛的关键词，或使用 @ 进行深度搜索。",
|
||||
retrieval_vectors_not_built: "向量索引未构建",
|
||||
retrieval_vectors_not_built_desc:
|
||||
"向量索引尚未构建，搜索需要先构建向量索引。",
|
||||
retrieval_open_vector_settings: "打开向量设置",
|
||||
retrieval_backend_unavailable: "后端不可用",
|
||||
retrieval_backend_unavailable_desc: "Python CLI 搜索后端无响应。",
|
||||
retrieval_run_doctor: "运行诊断",
|
||||
retrieval_retry: "重试",
|
||||
retrieval_timeout_title: "搜索超时",
|
||||
retrieval_timeout_desc: "搜索耗时过长。你可以重试，或尝试简化查询。",
|
||||
retrieval_model_changed: "模型已更换",
|
||||
retrieval_model_changed_desc: "嵌入模型已更换，需要重建向量。",
|
||||
retrieval_rebuild_vectors: "重建向量",
|
||||
retrieval_internal_error: "发生内部错误。请查看控制台了解详情。",
|
||||
retrieval_build_idle: "向量索引尚未构建",
|
||||
retrieval_build_ready: "向量索引就绪",
|
||||
retrieval_build_stopping: "正在停止…",
|
||||
retrieval_build_stopped: "构建已停止。点击继续以恢复。",
|
||||
retrieval_build_failed: "构建失败",
|
||||
retrieval_build_stale: "上次构建被中断。点击继续以恢复。",
|
||||
retrieval_build_deps_missing: "依赖未安装",
|
||||
retrieval_build_runtime_mismatch: "插件与 CLI 版本不匹配",
|
||||
retrieval_stop: "停止",
|
||||
retrieval_force_rebuild: "强制重建",
|
||||
retrieval_rebuild_warning: "重建将替换 {n} 个已有的 chunk。是否继续？",
|
||||
retrieval_no_python: "未找到 Python。请查看安装标签页。",
|
||||
},
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -1,119 +0,0 @@
|
|||
import * as fs from "fs";
|
||||
import * as path from "path";
|
||||
import initSqlJs from "sql.js";
|
||||
import type {
|
||||
Database as SqlJsDatabase,
|
||||
Statement as SqlJsStatement,
|
||||
SqlValue,
|
||||
} from "sql.js";
|
||||
|
||||
// ── Types ──
|
||||
|
||||
export interface SearchResultItem {
|
||||
zotero_key: string;
|
||||
title: string;
|
||||
first_author: string;
|
||||
year: string;
|
||||
journal: string;
|
||||
domain: string;
|
||||
abstract: string;
|
||||
rank: number;
|
||||
}
|
||||
|
||||
// ── Module-level state (lazy init) ──
|
||||
|
||||
let _db: SqlJsDatabase | null = null;
|
||||
let _queryStmt: SqlJsStatement | null = null;
|
||||
|
||||
// ── FTS5 query transform ──
|
||||
// FTS5 requires explicit operators between terms: "rotator cuff" → "rotator AND cuff"
|
||||
// Quoted phrases like '"rotator cuff"' are passed through verbatim.
|
||||
|
||||
function transformFtsQuery(input: string): string {
|
||||
input = input.trim();
|
||||
if (!input) return "";
|
||||
|
||||
// Already a quoted phrase — pass through as-is
|
||||
if (input.startsWith('"') && input.endsWith('"')) {
|
||||
return input;
|
||||
}
|
||||
|
||||
const terms = input.split(/\s+/).filter(Boolean);
|
||||
if (terms.length === 0) return "";
|
||||
return terms.join(" AND ");
|
||||
}
|
||||
|
||||
// ── Database init (lazy) ──
|
||||
|
||||
function sqlValueToString(v: SqlValue): string {
|
||||
if (v === null || v === undefined) return "";
|
||||
if (typeof v === "string") return v;
|
||||
if (v instanceof Uint8Array) return new TextDecoder().decode(v);
|
||||
return String(v);
|
||||
}
|
||||
|
||||
function sqlValueToNumber(v: SqlValue): number {
|
||||
if (v === null || v === undefined) return 0;
|
||||
if (typeof v === "number") return v;
|
||||
return Number(v);
|
||||
}
|
||||
|
||||
export async function initDatabase(vaultPath: string): Promise<void> {
|
||||
const dbPath = path.join(
|
||||
vaultPath,
|
||||
"System",
|
||||
"PaperForge",
|
||||
"indexes",
|
||||
"paperforge.db"
|
||||
);
|
||||
|
||||
if (!fs.existsSync(dbPath)) {
|
||||
throw new Error(`PaperForge database not found at ${dbPath}`);
|
||||
}
|
||||
|
||||
const SQL = await initSqlJs({
|
||||
locateFile: (file: string) => path.join(__dirname, file),
|
||||
});
|
||||
|
||||
const buffer = fs.readFileSync(dbPath);
|
||||
_db = new SQL.Database(new Uint8Array(buffer));
|
||||
|
||||
_queryStmt = _db.prepare(
|
||||
`SELECT zotero_key, title, first_author, year, journal, domain, abstract, rank
|
||||
FROM paper_fts
|
||||
WHERE paper_fts MATCH ?
|
||||
ORDER BY rank
|
||||
LIMIT ?`
|
||||
);
|
||||
}
|
||||
|
||||
// ── Search ──
|
||||
|
||||
export function searchMetadata(
|
||||
query: string,
|
||||
limit: number = 20
|
||||
): SearchResultItem[] | null {
|
||||
if (!_db || !_queryStmt) return null;
|
||||
|
||||
const transformed = transformFtsQuery(query);
|
||||
if (!transformed) return [];
|
||||
|
||||
_queryStmt.bind([transformed, limit]);
|
||||
|
||||
const results: SearchResultItem[] = [];
|
||||
while (_queryStmt.step()) {
|
||||
const row = _queryStmt.getAsObject();
|
||||
results.push({
|
||||
zotero_key: sqlValueToString(row.zotero_key),
|
||||
title: sqlValueToString(row.title),
|
||||
first_author: sqlValueToString(row.first_author),
|
||||
year: sqlValueToString(row.year),
|
||||
journal: sqlValueToString(row.journal),
|
||||
domain: sqlValueToString(row.domain),
|
||||
abstract: sqlValueToString(row.abstract),
|
||||
rank: sqlValueToNumber(row.rank),
|
||||
});
|
||||
}
|
||||
_queryStmt.reset();
|
||||
return results;
|
||||
}
|
||||
|
|
@ -164,9 +164,60 @@ export function getMemoryRuntime(vaultPath: string): MemoryRuntime | null {
|
|||
return readJSONFile(paths.memoryStatePath) as MemoryRuntime | null;
|
||||
}
|
||||
|
||||
let _vectorRuntimeCache: {
|
||||
vaultPath: string;
|
||||
result: VectorRuntime | null;
|
||||
ts: number;
|
||||
} | null = null;
|
||||
|
||||
export function getVectorRuntime(vaultPath: string): VectorRuntime | null {
|
||||
const paths = resolveVaultPaths(vaultPath);
|
||||
return readJSONFile(paths.vectorStatePath) as VectorRuntime | null;
|
||||
// Time-based cache: use snapshot within the 2s window
|
||||
const now = Date.now();
|
||||
if (
|
||||
_vectorRuntimeCache &&
|
||||
_vectorRuntimeCache.vaultPath === vaultPath &&
|
||||
now - _vectorRuntimeCache.ts < 2000
|
||||
) {
|
||||
return _vectorRuntimeCache.result;
|
||||
}
|
||||
// Try live embed status --json; fall back to JSON snapshot
|
||||
let pythonPath = "";
|
||||
const venvCandidates = [
|
||||
path.join(vaultPath, ".paperforge-test-venv", "Scripts", "python.exe"),
|
||||
path.join(vaultPath, ".venv", "Scripts", "python.exe"),
|
||||
path.join(vaultPath, "venv", "Scripts", "python.exe"),
|
||||
];
|
||||
for (let i = 0; i < venvCandidates.length; i++) {
|
||||
if (fs.existsSync(venvCandidates[i])) {
|
||||
pythonPath = venvCandidates[i];
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (pythonPath) {
|
||||
try {
|
||||
const out = execFileSync(
|
||||
pythonPath,
|
||||
["-m", "paperforge", "--vault", vaultPath, "embed", "status", "--json"],
|
||||
{
|
||||
encoding: "utf-8",
|
||||
timeout: 10000,
|
||||
windowsHide: true,
|
||||
}
|
||||
);
|
||||
const parsed = JSON.parse(out);
|
||||
if (parsed.ok && parsed.data) {
|
||||
const result = parsed.data as VectorRuntime;
|
||||
_vectorRuntimeCache = { vaultPath, result, ts: now };
|
||||
return result;
|
||||
}
|
||||
} catch {
|
||||
// fall through to snapshot
|
||||
}
|
||||
}
|
||||
const result = readJSONFile(paths.vectorStatePath) as VectorRuntime | null;
|
||||
_vectorRuntimeCache = { vaultPath, result, ts: now };
|
||||
return result;
|
||||
}
|
||||
|
||||
export function getRuntimeHealth(
|
||||
|
|
@ -188,7 +239,12 @@ export function isVectorReady(vaultPath: string): boolean {
|
|||
if (!s.deps_installed) return false;
|
||||
if (!s.db_exists) return false;
|
||||
if (s.healthy === false) return false;
|
||||
if (s.chunk_count === 0) return false;
|
||||
// Use total_chunks (body + object + fulltext) — chunk_count alone is vec_fulltext_meta which may be empty
|
||||
const total =
|
||||
(s.chunk_count ?? 0) +
|
||||
((s.body_chunk_count as number) ?? 0) +
|
||||
((s.object_chunk_count as number) ?? 0);
|
||||
if (total === 0) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -59,167 +59,329 @@ let _gitDirResolved = false;
|
|||
|
||||
// ── Runtime helpers ──
|
||||
|
||||
export function resolvePythonExecutable(vaultPath: string, settings: PaperForgeSettings | null | undefined, _fs: any, _execFileSync: any): PythonResult {
|
||||
const f = _fs || fs;
|
||||
const execSync = _execFileSync || execFileSync;
|
||||
export function resolvePythonExecutable(
|
||||
vaultPath: string,
|
||||
settings: PaperForgeSettings | null | undefined,
|
||||
_fs: any,
|
||||
_execFileSync: any
|
||||
): PythonResult {
|
||||
const f = _fs || fs;
|
||||
const execSync = _execFileSync || execFileSync;
|
||||
|
||||
if (settings && settings.python_path && settings.python_path.trim()) {
|
||||
const manualPath = settings.python_path.trim();
|
||||
if (f.existsSync(manualPath)) {
|
||||
return { path: manualPath, source: "manual", extraArgs: [] };
|
||||
if (settings && settings.python_path && settings.python_path.trim()) {
|
||||
const manualPath = settings.python_path.trim();
|
||||
if (f.existsSync(manualPath)) {
|
||||
return { path: manualPath, source: "manual", extraArgs: [] };
|
||||
}
|
||||
}
|
||||
|
||||
const venvCandidates = [
|
||||
path.join(vaultPath, ".paperforge-test-venv", "Scripts", "python.exe"),
|
||||
path.join(vaultPath, ".venv", "Scripts", "python.exe"),
|
||||
path.join(vaultPath, "venv", "Scripts", "python.exe"),
|
||||
];
|
||||
for (const candidate of venvCandidates) {
|
||||
try {
|
||||
if (f.existsSync(candidate)) {
|
||||
return { path: candidate, source: "auto-detected", extraArgs: [] };
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
|
||||
const systemCandidates = [
|
||||
{ path: "py", extraArgs: ["-3"] },
|
||||
{ path: "python", extraArgs: [] },
|
||||
{ path: "python3", extraArgs: [] },
|
||||
];
|
||||
for (const candidate of systemCandidates) {
|
||||
try {
|
||||
const verOut = execSync(
|
||||
candidate.path,
|
||||
[...candidate.extraArgs, "--version"],
|
||||
{
|
||||
encoding: "utf-8",
|
||||
timeout: 5000,
|
||||
windowsHide: true,
|
||||
}
|
||||
}
|
||||
);
|
||||
if (verOut && verOut.toLowerCase().includes("python")) {
|
||||
return {
|
||||
path: candidate.path,
|
||||
source: "auto-detected",
|
||||
extraArgs: candidate.extraArgs,
|
||||
};
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
|
||||
const venvCandidates = [
|
||||
path.join(vaultPath, ".paperforge-test-venv", "Scripts", "python.exe"),
|
||||
path.join(vaultPath, ".venv", "Scripts", "python.exe"),
|
||||
path.join(vaultPath, "venv", "Scripts", "python.exe"),
|
||||
];
|
||||
for (const candidate of venvCandidates) {
|
||||
try {
|
||||
if (f.existsSync(candidate)) {
|
||||
return { path: candidate, source: "auto-detected", extraArgs: [] };
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
|
||||
const systemCandidates = [
|
||||
{ path: "py", extraArgs: ["-3"] },
|
||||
{ path: "python", extraArgs: [] },
|
||||
{ path: "python3", extraArgs: [] },
|
||||
];
|
||||
for (const candidate of systemCandidates) {
|
||||
try {
|
||||
const verOut = execSync(candidate.path, [...candidate.extraArgs, "--version"], {
|
||||
encoding: "utf-8", timeout: 5000, windowsHide: true,
|
||||
});
|
||||
if (verOut && verOut.toLowerCase().includes("python")) {
|
||||
return { path: candidate.path, source: "auto-detected", extraArgs: candidate.extraArgs };
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
|
||||
return { path: "python", source: "auto-detected", extraArgs: [] };
|
||||
return { path: "python", source: "auto-detected", extraArgs: [] };
|
||||
}
|
||||
|
||||
export function getPluginVersion(app: any): string | null {
|
||||
try {
|
||||
const manifest = app && app.plugins && app.plugins.plugins &&
|
||||
app.plugins.plugins["paperforge"] && app.plugins.plugins["paperforge"].manifest;
|
||||
return (manifest && manifest.version) || null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
const manifest =
|
||||
app &&
|
||||
app.plugins &&
|
||||
app.plugins.plugins &&
|
||||
app.plugins.plugins["paperforge"] &&
|
||||
app.plugins.plugins["paperforge"].manifest;
|
||||
return (manifest && manifest.version) || null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export function checkRuntimeVersion(pythonExe: string, pluginVersion: string | null, cwd: string, timeout: number | undefined, _execFile: any): Promise<{ status: string; pyVersion: string | null; pluginVersion: string | null; error: string | null }> {
|
||||
if (timeout === undefined) timeout = 10000;
|
||||
const exe = _execFile || execFile;
|
||||
export function checkRuntimeVersion(
|
||||
pythonExe: string,
|
||||
pluginVersion: string | null,
|
||||
cwd: string,
|
||||
timeout: number | undefined,
|
||||
_execFile: any
|
||||
): Promise<{
|
||||
status: string;
|
||||
pyVersion: string | null;
|
||||
pluginVersion: string | null;
|
||||
error: string | null;
|
||||
}> {
|
||||
if (timeout === undefined) timeout = 10000;
|
||||
const exe = _execFile || execFile;
|
||||
|
||||
return new Promise((resolve) => {
|
||||
exe(pythonExe, ["-c", "import paperforge; print(paperforge.__version__)"],
|
||||
{ cwd, timeout },
|
||||
(err: any, stdout: any) => {
|
||||
if (err) {
|
||||
resolve({ status: "not-installed", pyVersion: null, pluginVersion, error: err.message });
|
||||
return;
|
||||
}
|
||||
const pyVer = (stdout && stdout.trim()) || null;
|
||||
if (pyVer === pluginVersion) {
|
||||
resolve({ status: "match", pyVersion: pyVer, pluginVersion, error: null });
|
||||
} else {
|
||||
resolve({ status: "mismatch", pyVersion: pyVer, pluginVersion, error: null });
|
||||
}
|
||||
});
|
||||
});
|
||||
return new Promise((resolve) => {
|
||||
exe(
|
||||
pythonExe,
|
||||
["-c", "import paperforge; print(paperforge.__version__)"],
|
||||
{ cwd, timeout },
|
||||
(err: any, stdout: any) => {
|
||||
if (err) {
|
||||
resolve({
|
||||
status: "not-installed",
|
||||
pyVersion: null,
|
||||
pluginVersion,
|
||||
error: err.message,
|
||||
});
|
||||
return;
|
||||
}
|
||||
const pyVer = (stdout && stdout.trim()) || null;
|
||||
if (pyVer === pluginVersion) {
|
||||
resolve({
|
||||
status: "match",
|
||||
pyVersion: pyVer,
|
||||
pluginVersion,
|
||||
error: null,
|
||||
});
|
||||
} else {
|
||||
resolve({
|
||||
status: "mismatch",
|
||||
pyVersion: pyVer,
|
||||
pluginVersion,
|
||||
error: null,
|
||||
});
|
||||
}
|
||||
}
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
// ── Error helpers ──
|
||||
|
||||
export function classifyError(errorCode: string): ErrorClassification {
|
||||
const code = String(errorCode);
|
||||
const patterns: Record<string, ErrorClassification> = {
|
||||
ENOENT: { type: "python_missing", message: "Python executable not found", recoverable: true },
|
||||
"python-missing": { type: "python_missing", message: "Python executable not found", recoverable: true },
|
||||
MODULE_NOT_FOUND: { type: "import_failed", message: "PaperForge package not installed", recoverable: true },
|
||||
"import-failed": { type: "import_failed", message: "PaperForge package not installed", recoverable: true },
|
||||
"version-mismatch": { type: "version_mismatch", message: "Plugin and package versions differ", recoverable: true, action: "sync-runtime" },
|
||||
"pip-failed": { type: "pip_install_failure", message: "pip install command failed", recoverable: true },
|
||||
ETIMEDOUT: { type: "timeout", message: "Subprocess timed out", recoverable: true, action: "retry" },
|
||||
timeout: { type: "timeout", message: "Subprocess timed out", recoverable: true, action: "retry" },
|
||||
};
|
||||
const match = patterns[code];
|
||||
if (match) return { ...match };
|
||||
return { type: "unknown", message: String(errorCode), recoverable: false };
|
||||
const code = String(errorCode);
|
||||
const patterns: Record<string, ErrorClassification> = {
|
||||
ENOENT: {
|
||||
type: "python_missing",
|
||||
message: "Python executable not found",
|
||||
recoverable: true,
|
||||
},
|
||||
"python-missing": {
|
||||
type: "python_missing",
|
||||
message: "Python executable not found",
|
||||
recoverable: true,
|
||||
},
|
||||
MODULE_NOT_FOUND: {
|
||||
type: "import_failed",
|
||||
message: "PaperForge package not installed",
|
||||
recoverable: true,
|
||||
},
|
||||
"import-failed": {
|
||||
type: "import_failed",
|
||||
message: "PaperForge package not installed",
|
||||
recoverable: true,
|
||||
},
|
||||
"version-mismatch": {
|
||||
type: "version_mismatch",
|
||||
message: "Plugin and package versions differ",
|
||||
recoverable: true,
|
||||
action: "sync-runtime",
|
||||
},
|
||||
"pip-failed": {
|
||||
type: "pip_install_failure",
|
||||
message: "pip install command failed",
|
||||
recoverable: true,
|
||||
},
|
||||
ETIMEDOUT: {
|
||||
type: "timeout",
|
||||
message: "Subprocess timed out",
|
||||
recoverable: true,
|
||||
action: "retry",
|
||||
},
|
||||
timeout: {
|
||||
type: "timeout",
|
||||
message: "Subprocess timed out",
|
||||
recoverable: true,
|
||||
action: "retry",
|
||||
},
|
||||
NO_PYTHON: {
|
||||
type: "no_python",
|
||||
message: "Python executable not found",
|
||||
recoverable: true,
|
||||
action: "open-setup",
|
||||
},
|
||||
VECTOR_NOT_BUILT: {
|
||||
type: "vectors_not_built",
|
||||
message: "Vector index has not been built yet",
|
||||
recoverable: true,
|
||||
action: "open-vector-settings",
|
||||
},
|
||||
VECTOR_CORRUPTED: {
|
||||
type: "vectors_corrupted",
|
||||
message: "Vector index is corrupted",
|
||||
recoverable: true,
|
||||
action: "force-rebuild",
|
||||
},
|
||||
MODEL_CHANGED: {
|
||||
type: "model_changed",
|
||||
message: "Embedding model has changed since vectors were built",
|
||||
recoverable: true,
|
||||
action: "rebuild-vectors",
|
||||
},
|
||||
BACKEND_UNAVAILABLE: {
|
||||
type: "backend_unavailable",
|
||||
message: "Python CLI search backend is not responding",
|
||||
recoverable: true,
|
||||
action: "run-doctor",
|
||||
},
|
||||
TIMEOUT: {
|
||||
type: "timeout",
|
||||
message: "Search timed out",
|
||||
recoverable: true,
|
||||
action: "retry",
|
||||
},
|
||||
INTERNAL_ERROR: {
|
||||
type: "internal_error",
|
||||
message: "An internal error occurred",
|
||||
recoverable: false,
|
||||
},
|
||||
};
|
||||
const match = patterns[code];
|
||||
if (match) return { ...match };
|
||||
return { type: "unknown", message: String(errorCode), recoverable: false };
|
||||
}
|
||||
|
||||
export function buildRuntimeInstallCommand(pythonExe: string, version: string, extraArgs: string[]): InstallCommand {
|
||||
if (extraArgs === undefined) extraArgs = [];
|
||||
const pypiPkg = `paperforge==${version}`;
|
||||
const gitUrl = `git+https://github.com/LLLin000/PaperForge.git@${version}`;
|
||||
const pypiArgs = [...extraArgs, "-m", "pip", "install", "--upgrade", pypiPkg];
|
||||
const gitArgs = [...extraArgs, "-m", "pip", "install", "--upgrade", gitUrl];
|
||||
return { cmd: pythonExe, url: gitUrl, args: gitArgs, pypiArgs, gitArgs, timeout: 120000 };
|
||||
export function buildRuntimeInstallCommand(
|
||||
pythonExe: string,
|
||||
version: string,
|
||||
extraArgs: string[]
|
||||
): InstallCommand {
|
||||
if (extraArgs === undefined) extraArgs = [];
|
||||
const pypiPkg = `paperforge==${version}`;
|
||||
const gitUrl = `git+https://github.com/LLLin000/PaperForge.git@${version}`;
|
||||
const pypiArgs = [...extraArgs, "-m", "pip", "install", "--upgrade", pypiPkg];
|
||||
const gitArgs = [...extraArgs, "-m", "pip", "install", "--upgrade", gitUrl];
|
||||
return {
|
||||
cmd: pythonExe,
|
||||
url: gitUrl,
|
||||
args: gitArgs,
|
||||
pypiArgs,
|
||||
gitArgs,
|
||||
timeout: 120000,
|
||||
};
|
||||
}
|
||||
|
||||
export function parseRuntimeStatus(err: any, stdout: any, stderr: any): RuntimeStatus {
|
||||
if (!err && stdout) {
|
||||
return { status: "ok", version: stdout.trim() };
|
||||
}
|
||||
if (err && err.code === "ENOENT") {
|
||||
const classified = classifyError("ENOENT");
|
||||
return { status: "error", version: null, ...classified };
|
||||
}
|
||||
if (stderr && stderr.includes("No module named paperforge")) {
|
||||
const classified = classifyError("import-failed");
|
||||
return { status: "error", version: null, ...classified };
|
||||
}
|
||||
if (err && err.killed) {
|
||||
const classified = classifyError("timeout");
|
||||
return { status: "error", version: null, ...classified };
|
||||
}
|
||||
if (stderr && stderr.includes("ModuleNotFoundError")) {
|
||||
const classified = classifyError("import-failed");
|
||||
return { status: "error", version: null, ...classified };
|
||||
}
|
||||
return { status: "error", version: null, type: "unknown",
|
||||
message: err ? err.message : String(stderr), recoverable: false };
|
||||
export function parseRuntimeStatus(
|
||||
err: any,
|
||||
stdout: any,
|
||||
stderr: any
|
||||
): RuntimeStatus {
|
||||
if (!err && stdout) {
|
||||
return { status: "ok", version: stdout.trim() };
|
||||
}
|
||||
if (err && err.code === "ENOENT") {
|
||||
const classified = classifyError("ENOENT");
|
||||
return { status: "error", version: null, ...classified };
|
||||
}
|
||||
if (stderr && stderr.includes("No module named paperforge")) {
|
||||
const classified = classifyError("import-failed");
|
||||
return { status: "error", version: null, ...classified };
|
||||
}
|
||||
if (err && err.killed) {
|
||||
const classified = classifyError("timeout");
|
||||
return { status: "error", version: null, ...classified };
|
||||
}
|
||||
if (stderr && stderr.includes("ModuleNotFoundError")) {
|
||||
const classified = classifyError("import-failed");
|
||||
return { status: "error", version: null, ...classified };
|
||||
}
|
||||
return {
|
||||
status: "error",
|
||||
version: null,
|
||||
type: "unknown",
|
||||
message: err ? err.message : String(stderr),
|
||||
recoverable: false,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Action definitions ── (ACTIONS already in constants.ts)
|
||||
|
||||
export function buildCommandArgs(action: any, key: any, filter: any): string[] {
|
||||
const args = Array.isArray(action.args) ? [...action.args] : [];
|
||||
if (action.needsKey && key) args.push(key);
|
||||
if (action.needsFilter || filter) args.push("--all");
|
||||
return args;
|
||||
const args = Array.isArray(action.args) ? [...action.args] : [];
|
||||
if (action.needsKey && key) args.push(key);
|
||||
if (action.needsFilter || filter) args.push("--all");
|
||||
return args;
|
||||
}
|
||||
|
||||
export function runSubprocess(pythonExe: string, args: string[], cwd: string, timeout: number, _spawn: any, env?: any): Promise<SubprocessResult> {
|
||||
const sp = _spawn || spawn;
|
||||
export function runSubprocess(
|
||||
pythonExe: string,
|
||||
args: string[],
|
||||
cwd: string,
|
||||
timeout: number,
|
||||
_spawn: any,
|
||||
env?: any
|
||||
): Promise<SubprocessResult> {
|
||||
const sp = _spawn || spawn;
|
||||
|
||||
return new Promise((resolve) => {
|
||||
const startTime = Date.now();
|
||||
const opts: any = { cwd, timeout, windowsHide: true };
|
||||
if (env) opts.env = env;
|
||||
const child = sp(pythonExe, args, opts);
|
||||
const stdoutChunks: string[] = [];
|
||||
const stderrChunks: string[] = [];
|
||||
return new Promise((resolve) => {
|
||||
const startTime = Date.now();
|
||||
const opts: any = { cwd, timeout, windowsHide: true };
|
||||
if (env) opts.env = env;
|
||||
const child = sp(pythonExe, args, opts);
|
||||
const stdoutChunks: string[] = [];
|
||||
const stderrChunks: string[] = [];
|
||||
|
||||
child.stdout.on("data", (data: any) => { stdoutChunks.push(data.toString("utf-8")); });
|
||||
child.stderr.on("data", (data: any) => { stderrChunks.push(data.toString("utf-8")); });
|
||||
|
||||
child.on("close", (code: any) => {
|
||||
resolve({ stdout: stdoutChunks.join(""), stderr: stderrChunks.join(""),
|
||||
exitCode: code, elapsed: Date.now() - startTime });
|
||||
});
|
||||
|
||||
child.on("error", (err: any) => {
|
||||
resolve({ stdout: stdoutChunks.join(""),
|
||||
stderr: stderrChunks.join("") + "\n" + err.message,
|
||||
exitCode: -1, elapsed: Date.now() - startTime });
|
||||
});
|
||||
child.stdout.on("data", (data: any) => {
|
||||
stdoutChunks.push(data.toString("utf-8"));
|
||||
});
|
||||
child.stderr.on("data", (data: any) => {
|
||||
stderrChunks.push(data.toString("utf-8"));
|
||||
});
|
||||
|
||||
child.on("close", (code: any) => {
|
||||
resolve({
|
||||
stdout: stdoutChunks.join(""),
|
||||
stderr: stderrChunks.join(""),
|
||||
exitCode: code,
|
||||
elapsed: Date.now() - startTime,
|
||||
});
|
||||
});
|
||||
|
||||
child.on("error", (err: any) => {
|
||||
resolve({
|
||||
stdout: stdoutChunks.join(""),
|
||||
stderr: stderrChunks.join("") + "\n" + err.message,
|
||||
exitCode: -1,
|
||||
elapsed: Date.now() - startTime,
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
export function runQueryPlan(
|
||||
|
|
@ -229,198 +391,267 @@ export function runQueryPlan(
|
|||
query: string,
|
||||
intent: "discover" | "content" | "known-paper",
|
||||
timeout = 20000,
|
||||
_execFile?: any,
|
||||
_execFile?: any
|
||||
): Promise<QueryPlanResult> {
|
||||
const exe = _execFile || execFile;
|
||||
return new Promise((resolve) => {
|
||||
const args = [...extraArgs, "-m", "paperforge", "--vault", vaultPath, "query-plan", query, "--intent", intent, "--json"];
|
||||
exe(pythonExe, args, { cwd: vaultPath, timeout, windowsHide: true }, (err: any, stdout: any, stderr: any) => {
|
||||
if (err) {
|
||||
resolve({
|
||||
ok: false,
|
||||
command: "query-plan",
|
||||
version: "",
|
||||
data: null,
|
||||
error: { message: stderr || err.message || "query-plan failed" },
|
||||
});
|
||||
return;
|
||||
const args = [
|
||||
...extraArgs,
|
||||
"-m",
|
||||
"paperforge",
|
||||
"--vault",
|
||||
vaultPath,
|
||||
"query-plan",
|
||||
query,
|
||||
"--intent",
|
||||
intent,
|
||||
"--json",
|
||||
];
|
||||
exe(
|
||||
pythonExe,
|
||||
args,
|
||||
{ cwd: vaultPath, timeout, windowsHide: true },
|
||||
(err: any, stdout: any, stderr: any) => {
|
||||
if (err) {
|
||||
resolve({
|
||||
ok: false,
|
||||
command: "query-plan",
|
||||
version: "",
|
||||
data: null,
|
||||
error: { message: stderr || err.message || "query-plan failed" },
|
||||
});
|
||||
return;
|
||||
}
|
||||
try {
|
||||
resolve(JSON.parse(stdout));
|
||||
} catch (parseErr: any) {
|
||||
resolve({
|
||||
ok: false,
|
||||
command: "query-plan",
|
||||
version: "",
|
||||
data: null,
|
||||
error: { message: parseErr.message || "Invalid query-plan JSON" },
|
||||
});
|
||||
}
|
||||
}
|
||||
try {
|
||||
resolve(JSON.parse(stdout));
|
||||
} catch (parseErr: any) {
|
||||
resolve({
|
||||
ok: false,
|
||||
command: "query-plan",
|
||||
version: "",
|
||||
data: null,
|
||||
error: { message: parseErr.message || "Invalid query-plan JSON" },
|
||||
});
|
||||
}
|
||||
});
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
// ── Cross-platform Python and BBT detection (macOS/Linux) ──
|
||||
|
||||
export function resolveGitDir(): string | null {
|
||||
if (_gitDirResolved) return _gitDir;
|
||||
_gitDirResolved = true;
|
||||
try {
|
||||
let out: string;
|
||||
if (process.platform === 'win32') {
|
||||
const cmdExe = process.env.ComSpec || 'C:\\Windows\\System32\\cmd.exe';
|
||||
out = execFileSync(cmdExe, ['/c', 'where', 'git'], { timeout: 5000, windowsHide: true, encoding: 'utf-8' });
|
||||
} else {
|
||||
out = execFileSync('which', ['git'], { timeout: 5000, encoding: 'utf-8' });
|
||||
}
|
||||
if (out) {
|
||||
const line = out.split('\n')[0].trim();
|
||||
if (line) _gitDir = path.dirname(line);
|
||||
}
|
||||
} catch (_) {}
|
||||
return _gitDir;
|
||||
if (_gitDirResolved) return _gitDir;
|
||||
_gitDirResolved = true;
|
||||
try {
|
||||
let out: string;
|
||||
if (process.platform === "win32") {
|
||||
const cmdExe = process.env.ComSpec || "C:\\Windows\\System32\\cmd.exe";
|
||||
out = execFileSync(cmdExe, ["/c", "where", "git"], {
|
||||
timeout: 5000,
|
||||
windowsHide: true,
|
||||
encoding: "utf-8",
|
||||
});
|
||||
} else {
|
||||
out = execFileSync("which", ["git"], {
|
||||
timeout: 5000,
|
||||
encoding: "utf-8",
|
||||
});
|
||||
}
|
||||
if (out) {
|
||||
const line = out.split("\n")[0].trim();
|
||||
if (line) _gitDir = path.dirname(line);
|
||||
}
|
||||
} catch (_) {}
|
||||
return _gitDir;
|
||||
}
|
||||
|
||||
export function paperforgeEnrichedEnv(): Record<string, string | undefined> {
|
||||
const env: Record<string, string | undefined> = { ...process.env };
|
||||
const plat = process.platform;
|
||||
const home = os.homedir();
|
||||
const extras: string[] = [];
|
||||
const gitDir = resolveGitDir();
|
||||
if (gitDir) extras.push(gitDir);
|
||||
if (plat === 'darwin') {
|
||||
extras.push('/opt/homebrew/bin', '/usr/local/bin', '/usr/bin', `${home}/.local/bin`);
|
||||
} else if (plat === 'linux') {
|
||||
extras.push('/usr/local/bin', '/usr/bin', `${home}/.local/bin`);
|
||||
}
|
||||
const cur = env.PATH || '';
|
||||
env.PATH = [...extras, cur].filter(Boolean).join(path.delimiter);
|
||||
return env;
|
||||
const env: Record<string, string | undefined> = { ...process.env };
|
||||
const plat = process.platform;
|
||||
const home = os.homedir();
|
||||
const extras: string[] = [];
|
||||
const gitDir = resolveGitDir();
|
||||
if (gitDir) extras.push(gitDir);
|
||||
if (plat === "darwin") {
|
||||
extras.push(
|
||||
"/opt/homebrew/bin",
|
||||
"/usr/local/bin",
|
||||
"/usr/bin",
|
||||
`${home}/.local/bin`
|
||||
);
|
||||
} else if (plat === "linux") {
|
||||
extras.push("/usr/local/bin", "/usr/bin", `${home}/.local/bin`);
|
||||
}
|
||||
const cur = env.PATH || "";
|
||||
env.PATH = [...extras, cur].filter(Boolean).join(path.delimiter);
|
||||
return env;
|
||||
}
|
||||
|
||||
export function shellQuoteForExec(cmd: string): string {
|
||||
if (!cmd) return "''";
|
||||
if (/[\s'"\\]/.test(cmd)) return `'${cmd.replace(/'/g, "'\\''")}'`;
|
||||
return cmd;
|
||||
if (!cmd) return "''";
|
||||
if (/[\s'"\\]/.test(cmd)) return `'${cmd.replace(/'/g, "'\\''")}'`;
|
||||
return cmd;
|
||||
}
|
||||
|
||||
export function isLikelyAppleStubPython(resolvedAbsPath: string): boolean {
|
||||
const n = String(resolvedAbsPath).toLowerCase().replace(/\\/g, '/');
|
||||
return n.includes('commandlinetools') || n.includes('/library/developer/commandlinetools');
|
||||
const n = String(resolvedAbsPath).toLowerCase().replace(/\\/g, "/");
|
||||
return (
|
||||
n.includes("commandlinetools") ||
|
||||
n.includes("/library/developer/commandlinetools")
|
||||
);
|
||||
}
|
||||
|
||||
export function collectDarwinPythonCandidates(home: string): string[] {
|
||||
return [
|
||||
'/opt/homebrew/bin/python3',
|
||||
'/usr/local/bin/python3',
|
||||
path.join(home, '.local', 'bin', 'python3'),
|
||||
path.join(home, '.pyenv', 'shims', 'python3'),
|
||||
'/usr/bin/python3',
|
||||
];
|
||||
return [
|
||||
"/opt/homebrew/bin/python3",
|
||||
"/usr/local/bin/python3",
|
||||
path.join(home, ".local", "bin", "python3"),
|
||||
path.join(home, ".pyenv", "shims", "python3"),
|
||||
"/usr/bin/python3",
|
||||
];
|
||||
}
|
||||
|
||||
export function getPaperforgePythonCmd(): string {
|
||||
const plat = process.platform;
|
||||
const home = os.homedir();
|
||||
if (plat === 'darwin') {
|
||||
let stubFallback: string | null = null;
|
||||
for (const p of collectDarwinPythonCandidates(home)) {
|
||||
try {
|
||||
if (!p || !fs.existsSync(p)) continue;
|
||||
let resolved = p;
|
||||
try { resolved = fs.realpathSync(p); } catch (_) {}
|
||||
if (isLikelyAppleStubPython(resolved)) {
|
||||
if (!stubFallback) stubFallback = p;
|
||||
continue;
|
||||
}
|
||||
return p;
|
||||
} catch (_) {}
|
||||
const plat = process.platform;
|
||||
const home = os.homedir();
|
||||
if (plat === "darwin") {
|
||||
let stubFallback: string | null = null;
|
||||
for (const p of collectDarwinPythonCandidates(home)) {
|
||||
try {
|
||||
if (!p || !fs.existsSync(p)) continue;
|
||||
let resolved = p;
|
||||
try {
|
||||
resolved = fs.realpathSync(p);
|
||||
} catch (_) {}
|
||||
if (isLikelyAppleStubPython(resolved)) {
|
||||
if (!stubFallback) stubFallback = p;
|
||||
continue;
|
||||
}
|
||||
if (stubFallback) return stubFallback;
|
||||
return 'python3';
|
||||
return p;
|
||||
} catch (_) {}
|
||||
}
|
||||
const candidates: string[] = [];
|
||||
if (plat === 'linux') {
|
||||
candidates.push('/usr/bin/python3', '/usr/local/bin/python3', path.join(home, '.local', 'bin', 'python3'), path.join(home, '.pyenv', 'shims', 'python3'));
|
||||
}
|
||||
for (const p of candidates) {
|
||||
try { if (p && fs.existsSync(p)) return p; } catch (_) {}
|
||||
}
|
||||
if (plat === 'win32') return 'python';
|
||||
return 'python3';
|
||||
if (stubFallback) return stubFallback;
|
||||
return "python3";
|
||||
}
|
||||
const candidates: string[] = [];
|
||||
if (plat === "linux") {
|
||||
candidates.push(
|
||||
"/usr/bin/python3",
|
||||
"/usr/local/bin/python3",
|
||||
path.join(home, ".local", "bin", "python3"),
|
||||
path.join(home, ".pyenv", "shims", "python3")
|
||||
);
|
||||
}
|
||||
for (const p of candidates) {
|
||||
try {
|
||||
if (p && fs.existsSync(p)) return p;
|
||||
} catch (_) {}
|
||||
}
|
||||
if (plat === "win32") return "python";
|
||||
return "python3";
|
||||
}
|
||||
|
||||
export function paperforgePythonExecArgs(scriptTail: string): string {
|
||||
const py = shellQuoteForExec(getPaperforgePythonCmd());
|
||||
return `${py} ${scriptTail}`;
|
||||
const py = shellQuoteForExec(getPaperforgePythonCmd());
|
||||
return `${py} ${scriptTail}`;
|
||||
}
|
||||
|
||||
export function tryExecPythonVersion(callback: any): void {
|
||||
const plat = process.platform;
|
||||
const home = os.homedir();
|
||||
const tried = new Set<string>();
|
||||
const list: string[] = [];
|
||||
if (plat === 'darwin') {
|
||||
const nonStub: string[] = [], stub: string[] = [];
|
||||
for (const p of collectDarwinPythonCandidates(home)) {
|
||||
try {
|
||||
if (!fs.existsSync(p)) continue;
|
||||
let resolved = p;
|
||||
try { resolved = fs.realpathSync(p); } catch (_) {}
|
||||
if (isLikelyAppleStubPython(resolved)) stub.push(p);
|
||||
else nonStub.push(p);
|
||||
} catch (_) {}
|
||||
}
|
||||
list.push(...nonStub, ...stub);
|
||||
} else if (plat === 'linux') {
|
||||
list.push('/usr/bin/python3', '/usr/local/bin/python3', path.join(home, '.local', 'bin', 'python3'), path.join(home, '.pyenv', 'shims', 'python3'));
|
||||
const plat = process.platform;
|
||||
const home = os.homedir();
|
||||
const tried = new Set<string>();
|
||||
const list: string[] = [];
|
||||
if (plat === "darwin") {
|
||||
const nonStub: string[] = [],
|
||||
stub: string[] = [];
|
||||
for (const p of collectDarwinPythonCandidates(home)) {
|
||||
try {
|
||||
if (!fs.existsSync(p)) continue;
|
||||
let resolved = p;
|
||||
try {
|
||||
resolved = fs.realpathSync(p);
|
||||
} catch (_) {}
|
||||
if (isLikelyAppleStubPython(resolved)) stub.push(p);
|
||||
else nonStub.push(p);
|
||||
} catch (_) {}
|
||||
}
|
||||
list.push(plat === 'win32' ? 'python' : 'python3', 'python');
|
||||
const candidates = list.filter((c: string) => { if (!c || tried.has(c)) return false; tried.add(c); return true; });
|
||||
let i = 0;
|
||||
const next = () => {
|
||||
if (i >= candidates.length) { callback(new Error('Python not found'), '', null); return; }
|
||||
const py = candidates[i++];
|
||||
if (py.includes(path.sep) || py.startsWith('/')) {
|
||||
try { if (!fs.existsSync(py)) { next(); return; } } catch (_) { next(); return; }
|
||||
list.push(...nonStub, ...stub);
|
||||
} else if (plat === "linux") {
|
||||
list.push(
|
||||
"/usr/bin/python3",
|
||||
"/usr/local/bin/python3",
|
||||
path.join(home, ".local", "bin", "python3"),
|
||||
path.join(home, ".pyenv", "shims", "python3")
|
||||
);
|
||||
}
|
||||
list.push(plat === "win32" ? "python" : "python3", "python");
|
||||
const candidates = list.filter((c: string) => {
|
||||
if (!c || tried.has(c)) return false;
|
||||
tried.add(c);
|
||||
return true;
|
||||
});
|
||||
let i = 0;
|
||||
const next = () => {
|
||||
if (i >= candidates.length) {
|
||||
callback(new Error("Python not found"), "", null);
|
||||
return;
|
||||
}
|
||||
const py = candidates[i++];
|
||||
if (py.includes(path.sep) || py.startsWith("/")) {
|
||||
try {
|
||||
if (!fs.existsSync(py)) {
|
||||
next();
|
||||
return;
|
||||
}
|
||||
exec(`${shellQuoteForExec(py)} --version`, { timeout: 8000, env: paperforgeEnrichedEnv() as any }, (err: any, stdout: any) => {
|
||||
if (!err && stdout) callback(null, stdout.trim(), py);
|
||||
else next();
|
||||
});
|
||||
};
|
||||
next();
|
||||
} catch (_) {
|
||||
next();
|
||||
return;
|
||||
}
|
||||
}
|
||||
exec(
|
||||
`${shellQuoteForExec(py)} --version`,
|
||||
{ timeout: 8000, env: paperforgeEnrichedEnv() as any },
|
||||
(err: any, stdout: any) => {
|
||||
if (!err && stdout) callback(null, stdout.trim(), py);
|
||||
else next();
|
||||
}
|
||||
);
|
||||
};
|
||||
next();
|
||||
}
|
||||
|
||||
export function dirLooksLikeBetterBibtexFolder(entryName: string): boolean {
|
||||
const compact = String(entryName).toLowerCase().replace(/[^a-z0-9]/g, '');
|
||||
return compact.includes('betterbibtex');
|
||||
const compact = String(entryName)
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]/g, "");
|
||||
return compact.includes("betterbibtex");
|
||||
}
|
||||
|
||||
export function scanBbtDirectChildren(dir: string): boolean {
|
||||
if (!dir) return false;
|
||||
try {
|
||||
if (!fs.existsSync(dir)) return false;
|
||||
for (const entry of fs.readdirSync(dir)) {
|
||||
if (dirLooksLikeBetterBibtexFolder(entry)) return true;
|
||||
}
|
||||
} catch (_) {}
|
||||
return false;
|
||||
if (!dir) return false;
|
||||
try {
|
||||
if (!fs.existsSync(dir)) return false;
|
||||
for (const entry of fs.readdirSync(dir)) {
|
||||
if (dirLooksLikeBetterBibtexFolder(entry)) return true;
|
||||
}
|
||||
} catch (_) {}
|
||||
return false;
|
||||
}
|
||||
|
||||
export function scanBbtUnderProfiles(profilesDir: string): boolean {
|
||||
if (!profilesDir) return false;
|
||||
try {
|
||||
if (!fs.existsSync(profilesDir)) return false;
|
||||
for (const prof of fs.readdirSync(profilesDir)) {
|
||||
const extDir = path.join(profilesDir, prof, 'extensions');
|
||||
try {
|
||||
if (!fs.existsSync(extDir)) continue;
|
||||
for (const entry of fs.readdirSync(extDir)) {
|
||||
if (dirLooksLikeBetterBibtexFolder(entry)) return true;
|
||||
}
|
||||
} catch (_) {}
|
||||
if (!profilesDir) return false;
|
||||
try {
|
||||
if (!fs.existsSync(profilesDir)) return false;
|
||||
for (const prof of fs.readdirSync(profilesDir)) {
|
||||
const extDir = path.join(profilesDir, prof, "extensions");
|
||||
try {
|
||||
if (!fs.existsSync(extDir)) continue;
|
||||
for (const entry of fs.readdirSync(extDir)) {
|
||||
if (dirLooksLikeBetterBibtexFolder(entry)) return true;
|
||||
}
|
||||
} catch (_) {}
|
||||
return false;
|
||||
} catch (_) {}
|
||||
}
|
||||
} catch (_) {}
|
||||
return false;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -56,6 +56,8 @@ interface ISettingPlugin {
|
|||
_embedProcess?: unknown;
|
||||
_embedProgress?: { current: number; total: number; key: string };
|
||||
_embedStderr?: string;
|
||||
_embedPollInterval?: ReturnType<typeof setInterval> | null;
|
||||
_embedPolling?: boolean;
|
||||
}
|
||||
|
||||
export class PaperForgeSettingTab extends PluginSettingTab {
|
||||
|
|
@ -74,6 +76,12 @@ export class PaperForgeSettingTab extends PluginSettingTab {
|
|||
private _customPathDescEl: HTMLElement | null = null;
|
||||
private _checkEl: HTMLDivElement | null = null;
|
||||
activeTab = "setup";
|
||||
private _buildState: string = "idle";
|
||||
private _buildProgress: { current: number; total: number; key: string } = {
|
||||
current: 0,
|
||||
total: 0,
|
||||
key: "",
|
||||
};
|
||||
|
||||
constructor(app: App, plugin: ISettingPlugin) {
|
||||
super(app, plugin as any);
|
||||
|
|
@ -1146,7 +1154,7 @@ export class PaperForgeSettingTab extends PluginSettingTab {
|
|||
cls: "paperforge-embed-header",
|
||||
});
|
||||
embedHeader.createEl("span", {
|
||||
text: t("feat_rebuild_vectors"),
|
||||
text: t("retrieval_rebuild_vectors"),
|
||||
cls: "setting-item-name",
|
||||
});
|
||||
|
||||
|
|
@ -1156,12 +1164,19 @@ export class PaperForgeSettingTab extends PluginSettingTab {
|
|||
|
||||
const embedStatusText = embedSection.createEl("div", {
|
||||
cls: "paperforge-embed-status-text",
|
||||
attr: { "aria-live": "polite" },
|
||||
});
|
||||
|
||||
const renderEmbedUI = () => {
|
||||
embedControls.empty();
|
||||
embedStatusText.empty();
|
||||
const buildState: any = (getVectorRuntime(vp) || {}).build_state || {};
|
||||
|
||||
const vr = getVectorRuntime(vp);
|
||||
const bsRaw = vr?.build_state;
|
||||
const buildState: Record<string, unknown> =
|
||||
bsRaw && typeof bsRaw === "object" && !Array.isArray(bsRaw)
|
||||
? (bsRaw as Record<string, unknown>)
|
||||
: {};
|
||||
this.plugin._embedProgress = this.plugin._embedProgress || {
|
||||
current: 0,
|
||||
total: 0,
|
||||
|
|
@ -1170,168 +1185,391 @@ export class PaperForgeSettingTab extends PluginSettingTab {
|
|||
|
||||
if (!this.plugin._embedProcess && buildState.status === "running") {
|
||||
this.plugin._embedProgress = {
|
||||
current: buildState.current || 0,
|
||||
total: buildState.total || 1,
|
||||
key: buildState.paper_id || "",
|
||||
current:
|
||||
typeof buildState.current === "number" ? buildState.current : 0,
|
||||
total: typeof buildState.total === "number" ? buildState.total : 1,
|
||||
key:
|
||||
typeof buildState.paper_id === "string" ? buildState.paper_id : "",
|
||||
};
|
||||
}
|
||||
|
||||
const { current, total, key } = this.plugin._embedProgress;
|
||||
const isRunning =
|
||||
!!this.plugin._embedProcess || buildState.status === "running";
|
||||
|
||||
if (isRunning) {
|
||||
const track = embedControls.createEl("div", {
|
||||
cls: "paperforge-progress-track",
|
||||
});
|
||||
track.style.cssText = "flex:1;";
|
||||
const pct = total > 0 ? ((current / total) * 100).toFixed(1) : "0";
|
||||
const doneSeg = track.createEl("div", {
|
||||
cls: "paperforge-progress-seg done",
|
||||
});
|
||||
doneSeg.style.cssText = `width:${pct}%; min-width:${current > 0 ? "2px" : "0"};`;
|
||||
if (current < total) {
|
||||
const pendingSeg = track.createEl("div", {
|
||||
cls: "paperforge-progress-seg pending",
|
||||
});
|
||||
pendingSeg.style.cssText = `width:${(100 - parseFloat(pct)).toFixed(1)}%;`;
|
||||
}
|
||||
embedStatusText.createEl("span", {
|
||||
cls: "paperforge-embed-progress-text",
|
||||
text: `${current}/${total} papers`,
|
||||
});
|
||||
if (key) {
|
||||
embedStatusText.createEl("span", {
|
||||
cls: "paperforge-embed-progress-key",
|
||||
text: ` (${key})`,
|
||||
});
|
||||
}
|
||||
// Safely access fields from VectorRuntime index signature
|
||||
const bodyChunkCount =
|
||||
typeof vr?.body_chunk_count === "number" ? vr.body_chunk_count : 0;
|
||||
const objectChunkCount =
|
||||
typeof vr?.object_chunk_count === "number" ? vr.object_chunk_count : 0;
|
||||
const chunkCount =
|
||||
typeof vr?.chunk_count === "number" ? vr.chunk_count : 0;
|
||||
const totalChunks = chunkCount + bodyChunkCount + objectChunkCount;
|
||||
const hasChunks = totalChunks > 0;
|
||||
const isCorrupted =
|
||||
vr !== null && typeof vr.corrupted === "boolean" && vr.corrupted;
|
||||
const isBuilding = !!this.plugin._embedProcess;
|
||||
const isStale =
|
||||
!this.plugin._embedProcess && buildState.status === "running";
|
||||
// deps_installed is a defined boolean? property on VectorRuntime
|
||||
const depsInstalled =
|
||||
vr?.deps_installed !== undefined ? !!vr.deps_installed : true;
|
||||
|
||||
const stopBtn = embedControls.createEl("button");
|
||||
stopBtn.setText("Stop");
|
||||
stopBtn.className = "mod-warning";
|
||||
stopBtn.addEventListener("click", () => {
|
||||
this._callPython(["embed", "stop", "--json"], { timeout: 8000 });
|
||||
if (this.plugin._embedProcess) {
|
||||
(this.plugin._embedProcess as any).kill();
|
||||
this.plugin._embedProcess = null;
|
||||
}
|
||||
this.display();
|
||||
});
|
||||
} else {
|
||||
const embedInfo = getVectorRuntime(vp);
|
||||
const embedChunks =
|
||||
(embedInfo?.chunk_count ?? 0) +
|
||||
((embedInfo?.body_chunk_count as number) ?? 0) +
|
||||
((embedInfo?.object_chunk_count as number) ?? 0);
|
||||
const hasChunks = embedChunks > 0;
|
||||
const isCorrupted = embedInfo ? !!embedInfo.corrupted : false;
|
||||
const status =
|
||||
typeof buildState.status === "string" ? buildState.status : "";
|
||||
const buildMessage =
|
||||
typeof buildState.message === "string" ? buildState.message : "";
|
||||
|
||||
const startBuild = (flag: string) => {
|
||||
const py = getCachedPython(vp, this.plugin.settings);
|
||||
if (!py.path) {
|
||||
new Notice(t("feat_no_python"));
|
||||
return;
|
||||
}
|
||||
const env = Object.assign({}, process.env, {
|
||||
PYTHONIOENCODING: "utf-8",
|
||||
PYTHONUTF8: "1",
|
||||
VECTOR_DB_API_KEY: this.plugin.settings.vector_db_api_key || "",
|
||||
VECTOR_DB_API_BASE: this.plugin.settings.vector_db_api_base || "",
|
||||
VECTOR_DB_API_MODEL: this.plugin.settings.vector_db_api_model || "",
|
||||
});
|
||||
this.plugin._embedStderr = "";
|
||||
this.plugin._embedProgress = { current: 0, total: 0, key: "" };
|
||||
this.plugin._embedProcess = this._callPython(
|
||||
["embed", "build", flag],
|
||||
{
|
||||
stream: true,
|
||||
env: env,
|
||||
onData: (data: any) => {
|
||||
const lines = data.toString("utf-8").split("\n");
|
||||
for (const line of lines) {
|
||||
if (line.startsWith("EMBED_START:")) {
|
||||
this.plugin._embedProgress!.total =
|
||||
parseInt(line.split(":")[1]) || 0;
|
||||
} else if (line.startsWith("EMBED_PROGRESS:")) {
|
||||
const parts = line.split(":");
|
||||
this.plugin._embedProgress!.current =
|
||||
parseInt(parts[1]) || 0;
|
||||
this.plugin._embedProgress!.key = parts[3] || "";
|
||||
} else if (line.startsWith("EMBED_DONE")) {
|
||||
this.plugin._embedProcess = null;
|
||||
this.plugin._embedProgress!.current =
|
||||
this.plugin._embedProgress!.total;
|
||||
}
|
||||
}
|
||||
this.display();
|
||||
},
|
||||
onStderr: (data: any) => {
|
||||
if (!this.plugin._embedStderr) this.plugin._embedStderr = "";
|
||||
this.plugin._embedStderr += data.toString("utf-8");
|
||||
},
|
||||
onError: (err: any) => {
|
||||
this.plugin._embedProcess = null;
|
||||
new Notice(
|
||||
t("feat_build_failed") + ": " + (err.message || err)
|
||||
);
|
||||
this.display();
|
||||
},
|
||||
onClose: (code: number | null) => {
|
||||
this.plugin._embedProcess = null;
|
||||
if (code === 0) {
|
||||
this.plugin._embedProgress!.current =
|
||||
this.plugin._embedProgress!.total;
|
||||
this.plugin.saveSettings();
|
||||
this._embedStatusText = getVectorStatusText(vp);
|
||||
new Notice(t("feat_build_complete"));
|
||||
} else {
|
||||
this._embedStatusText = null;
|
||||
const errMsg = (this.plugin._embedStderr || "").slice(0, 200);
|
||||
new Notice(
|
||||
t("feat_build_failed") + (errMsg ? ": " + errMsg : ""),
|
||||
8000
|
||||
);
|
||||
}
|
||||
this.plugin._embedStderr = "";
|
||||
this.display();
|
||||
this._refreshSnapshots(vp);
|
||||
},
|
||||
}
|
||||
const startBuild = (flag: string) => {
|
||||
// ── Destructive warnings ──
|
||||
if (flag === "--resume" && hasChunks && !isCorrupted) {
|
||||
const msg = t("retrieval_rebuild_warning").replace(
|
||||
"{n}",
|
||||
String(totalChunks)
|
||||
);
|
||||
if (!confirm(msg)) return;
|
||||
}
|
||||
if (flag === "--force" && hasChunks && !isCorrupted) {
|
||||
const msg =
|
||||
"Force rebuild will replace " +
|
||||
totalChunks +
|
||||
" existing chunk(s). Continue?";
|
||||
if (!confirm(msg)) return;
|
||||
}
|
||||
|
||||
this.display();
|
||||
};
|
||||
const py = getCachedPython(vp, this.plugin.settings);
|
||||
if (!py.path) {
|
||||
new Notice(t("retrieval_no_python"));
|
||||
return;
|
||||
}
|
||||
const env = Object.assign({}, process.env, {
|
||||
PYTHONIOENCODING: "utf-8",
|
||||
PYTHONUTF8: "1",
|
||||
VECTOR_DB_API_KEY: this.plugin.settings.vector_db_api_key || "",
|
||||
VECTOR_DB_API_BASE: this.plugin.settings.vector_db_api_base || "",
|
||||
VECTOR_DB_API_MODEL: this.plugin.settings.vector_db_api_model || "",
|
||||
});
|
||||
this.plugin._embedStderr = "";
|
||||
this.plugin._embedProgress = { current: 0, total: 0, key: "" };
|
||||
this.plugin._embedProcess = this._callPython(["embed", "build", flag], {
|
||||
stream: true,
|
||||
env: env,
|
||||
onData: (data: unknown) => {
|
||||
// Node stream emits Buffer; data can also be string
|
||||
const text =
|
||||
typeof data === "string"
|
||||
? data
|
||||
: Buffer.isBuffer(data)
|
||||
? data.toString("utf-8")
|
||||
: String(data);
|
||||
const lines = text.split("\n");
|
||||
for (const line of lines) {
|
||||
if (line.startsWith("EMBED_START:")) {
|
||||
this.plugin._embedProgress!.total =
|
||||
parseInt(line.split(":")[1]) || 0;
|
||||
} else if (line.startsWith("EMBED_PROGRESS:")) {
|
||||
const parts = line.split(":");
|
||||
this.plugin._embedProgress!.current = parseInt(parts[1]) || 0;
|
||||
this.plugin._embedProgress!.key = parts[3] || "";
|
||||
} else if (line.startsWith("EMBED_DONE")) {
|
||||
this.plugin._embedProcess = null;
|
||||
this.plugin._embedProgress!.current =
|
||||
this.plugin._embedProgress!.total;
|
||||
}
|
||||
}
|
||||
this.display();
|
||||
},
|
||||
onStderr: (data: unknown) => {
|
||||
if (!this.plugin._embedStderr) this.plugin._embedStderr = "";
|
||||
this.plugin._embedStderr += String(data);
|
||||
},
|
||||
onError: (err: Error) => {
|
||||
this.plugin._embedProcess = null;
|
||||
new Notice(t("feat_build_failed") + ": " + (err.message || err));
|
||||
this.display();
|
||||
},
|
||||
onClose: (code: number | null) => {
|
||||
clearInterval(this.plugin._embedPollInterval ?? undefined);
|
||||
this.plugin._embedPollInterval = null;
|
||||
this.plugin._embedProcess = null;
|
||||
if (code === 0) {
|
||||
this.plugin._embedProgress!.current =
|
||||
this.plugin._embedProgress!.total;
|
||||
this.plugin.saveSettings();
|
||||
this._embedStatusText = getVectorStatusText(vp);
|
||||
new Notice(t("feat_build_complete"));
|
||||
} else {
|
||||
this._embedStatusText = null;
|
||||
const errMsg = (this.plugin._embedStderr || "").slice(0, 200);
|
||||
new Notice(
|
||||
t("feat_build_failed") + (errMsg ? ": " + errMsg : ""),
|
||||
8000
|
||||
);
|
||||
}
|
||||
this.plugin._embedStderr = "";
|
||||
this.display();
|
||||
this._refreshSnapshots(vp);
|
||||
},
|
||||
});
|
||||
|
||||
if (isCorrupted) {
|
||||
const warnEl = embedSection.createEl("div");
|
||||
warnEl.style.cssText =
|
||||
"padding:8px 12px; margin:8px 0; background:var(--background-modifier-warning); border-radius:4px; font-size:12px; display:flex; align-items:center; justify-content:space-between;";
|
||||
warnEl.createEl("span", { text: t("feat_vector_corrupted") });
|
||||
const forceBtn = warnEl.createEl("button", {
|
||||
text: t("feat_vector_rebuild_force_btn"),
|
||||
// Poll embed status every 2s during build for live state
|
||||
clearInterval(this.plugin._embedPollInterval ?? undefined);
|
||||
this.plugin._embedPollInterval = setInterval(() => {
|
||||
if (this.plugin._embedPolling) return;
|
||||
this.plugin._embedPolling = true;
|
||||
this._callPython(["embed", "status", "--json"], {
|
||||
timeout: 5000,
|
||||
onClose: (_code: number | null, stdout: string) => {
|
||||
this.plugin._embedPolling = false;
|
||||
if (_code === 0 && stdout) {
|
||||
try {
|
||||
const result = JSON.parse(stdout);
|
||||
const data = result.data;
|
||||
if (data && data.build_state) {
|
||||
const bs = data.build_state;
|
||||
if (bs.status === "stopping" || bs.status === "idle") {
|
||||
if (this.plugin._embedProcess) {
|
||||
this.plugin._embedProcess = null;
|
||||
clearInterval(
|
||||
this.plugin._embedPollInterval ?? undefined
|
||||
);
|
||||
this.plugin._embedPollInterval = null;
|
||||
this.display();
|
||||
}
|
||||
}
|
||||
if (bs.current !== undefined && bs.total !== undefined) {
|
||||
this.plugin._embedProgress!.current = bs.current;
|
||||
this.plugin._embedProgress!.total = bs.total || 1;
|
||||
this.plugin._embedProgress!.key = bs.paper_id || "";
|
||||
}
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
},
|
||||
});
|
||||
}, 2000);
|
||||
|
||||
this.display();
|
||||
};
|
||||
|
||||
// Detect runtime version mismatch from health data
|
||||
const health = getRuntimeHealth(vp);
|
||||
let runtimeMismatch = false;
|
||||
if (
|
||||
health &&
|
||||
typeof health.summary === "object" &&
|
||||
health.summary !== null &&
|
||||
"status" in health.summary
|
||||
) {
|
||||
runtimeMismatch = health.summary.status === "version_mismatch";
|
||||
}
|
||||
|
||||
// ── State determination (priority order) ──
|
||||
let uiState: string;
|
||||
if (!depsInstalled) {
|
||||
uiState = "deps-missing";
|
||||
} else if (runtimeMismatch) {
|
||||
uiState = "runtime-mismatch";
|
||||
} else if (status === "stopping") {
|
||||
uiState = "stopping";
|
||||
} else if (isBuilding && status === "running") {
|
||||
uiState = "building";
|
||||
} else if (status === "failed") {
|
||||
uiState = "failed";
|
||||
} else if (status === "stopped") {
|
||||
uiState = "stopped";
|
||||
} else if (isStale) {
|
||||
uiState = "stale";
|
||||
} else if (isCorrupted) {
|
||||
uiState = "corrupted";
|
||||
} else if (hasChunks) {
|
||||
uiState = "ready";
|
||||
} else {
|
||||
uiState = "idle";
|
||||
}
|
||||
|
||||
// ── State rendering ──
|
||||
switch (uiState) {
|
||||
case "building": {
|
||||
const track = embedControls.createEl("div", {
|
||||
cls: "paperforge-progress-track",
|
||||
});
|
||||
track.style.cssText = "flex:1;";
|
||||
const pct = total > 0 ? ((current / total) * 100).toFixed(1) : "0";
|
||||
const doneSeg = track.createEl("div", {
|
||||
cls: "paperforge-progress-seg done",
|
||||
});
|
||||
doneSeg.style.cssText = `width:${pct}%; min-width:${current > 0 ? "2px" : "0"};`;
|
||||
if (current < total) {
|
||||
const pendingSeg = track.createEl("div", {
|
||||
cls: "paperforge-progress-seg pending",
|
||||
});
|
||||
pendingSeg.style.cssText = `width:${(100 - parseFloat(pct)).toFixed(1)}%;`;
|
||||
}
|
||||
embedStatusText.createEl("span", {
|
||||
cls: "paperforge-embed-progress-text",
|
||||
text: `${current}/${total} papers`,
|
||||
});
|
||||
if (key) {
|
||||
embedStatusText.createEl("span", {
|
||||
cls: "paperforge-embed-progress-key",
|
||||
text: ` (${key})`,
|
||||
});
|
||||
}
|
||||
// Warning button: Stop
|
||||
const stopBtn = embedControls.createEl("button");
|
||||
stopBtn.setText(t("retrieval_stop"));
|
||||
stopBtn.className = "mod-warning";
|
||||
stopBtn.addEventListener("click", () => {
|
||||
this._callPython(["embed", "stop", "--json"], {
|
||||
timeout: 8000,
|
||||
});
|
||||
this.display();
|
||||
});
|
||||
break;
|
||||
}
|
||||
|
||||
case "stopping": {
|
||||
const track = embedControls.createEl("div", {
|
||||
cls: "paperforge-progress-track",
|
||||
});
|
||||
track.style.cssText = "flex:1; opacity:0.5;";
|
||||
const pct = total > 0 ? ((current / total) * 100).toFixed(1) : "0";
|
||||
const doneSeg = track.createEl("div", {
|
||||
cls: "paperforge-progress-seg done",
|
||||
});
|
||||
doneSeg.style.cssText = `width:${pct}%; min-width:${current > 0 ? "2px" : "0"};`;
|
||||
if (current < total) {
|
||||
const pendingSeg = track.createEl("div", {
|
||||
cls: "paperforge-progress-seg pending",
|
||||
});
|
||||
pendingSeg.style.cssText = `width:${(100 - parseFloat(pct)).toFixed(1)}%;`;
|
||||
}
|
||||
embedStatusText.createEl("span", {
|
||||
text: t("retrieval_build_stopping"),
|
||||
});
|
||||
const stopBtn = embedControls.createEl("button");
|
||||
stopBtn.setText(t("retrieval_stop"));
|
||||
stopBtn.className = "mod-warning";
|
||||
stopBtn.setAttr("disabled", "");
|
||||
break;
|
||||
}
|
||||
|
||||
case "failed": {
|
||||
embedStatusText.createEl("div", {
|
||||
cls: "paperforge-desc-box",
|
||||
text:
|
||||
t("retrieval_build_failed") +
|
||||
(buildMessage ? ": " + buildMessage : ""),
|
||||
attr: { style: "color:var(--text-error);" },
|
||||
});
|
||||
// Primary CTA: Retry
|
||||
const retryBtn = embedControls.createEl("button");
|
||||
retryBtn.setText(t("retrieval_retry"));
|
||||
retryBtn.className = "mod-cta";
|
||||
retryBtn.addEventListener("click", () => startBuild("--resume"));
|
||||
// Secondary: Force Rebuild
|
||||
const forceBtn = embedControls.createEl("button");
|
||||
forceBtn.setText(t("retrieval_force_rebuild"));
|
||||
forceBtn.style.marginLeft = "6px";
|
||||
forceBtn.addEventListener("click", () => startBuild("--force"));
|
||||
break;
|
||||
}
|
||||
|
||||
case "stopped": {
|
||||
embedStatusText.setText(t("retrieval_build_stopped"));
|
||||
// Primary CTA: Resume
|
||||
const resumeBtn = embedControls.createEl("button");
|
||||
resumeBtn.setText(t("retrieval_retry"));
|
||||
resumeBtn.className = "mod-cta";
|
||||
resumeBtn.addEventListener("click", () => startBuild("--resume"));
|
||||
break;
|
||||
}
|
||||
|
||||
case "corrupted": {
|
||||
embedStatusText.createEl("div", {
|
||||
cls: "paperforge-desc-box",
|
||||
text: t("feat_vector_corrupted"),
|
||||
attr: {
|
||||
style: "background:var(--background-modifier-warning);",
|
||||
},
|
||||
});
|
||||
// Primary CTA: Force Rebuild (no destructive warning on corrupted)
|
||||
const forceBtn = embedControls.createEl("button");
|
||||
forceBtn.setText(t("retrieval_force_rebuild"));
|
||||
forceBtn.className = "mod-cta";
|
||||
forceBtn.addEventListener("click", () => startBuild("--force"));
|
||||
break;
|
||||
}
|
||||
|
||||
if (hasChunks && !isCorrupted) {
|
||||
case "stale": {
|
||||
embedStatusText.createEl("div", {
|
||||
cls: "paperforge-desc-box",
|
||||
text: t("retrieval_build_stale"),
|
||||
attr: { style: "color:var(--text-warning);" },
|
||||
});
|
||||
// Primary CTA: Rebuild
|
||||
const rebuildBtn = embedControls.createEl("button");
|
||||
rebuildBtn.setText(t("retrieval_rebuild_vectors"));
|
||||
rebuildBtn.className = "mod-cta";
|
||||
rebuildBtn.addEventListener("click", () => startBuild("--resume"));
|
||||
break;
|
||||
}
|
||||
|
||||
case "ready": {
|
||||
embedControls.createEl("span", {
|
||||
text: embedChunks + " chunks embedded",
|
||||
text: totalChunks + " chunks embedded",
|
||||
cls: "setting-item-description",
|
||||
});
|
||||
// Primary CTA: Rebuild Vectors
|
||||
const rebuildBtn = embedControls.createEl("button");
|
||||
rebuildBtn.setText(t("retrieval_rebuild_vectors"));
|
||||
rebuildBtn.className = "mod-cta";
|
||||
rebuildBtn.addEventListener("click", () => startBuild("--resume"));
|
||||
// Secondary: Force Rebuild
|
||||
const forceBtn = embedControls.createEl("button");
|
||||
forceBtn.setText(t("retrieval_force_rebuild"));
|
||||
forceBtn.style.marginLeft = "6px";
|
||||
forceBtn.addEventListener("click", () => startBuild("--force"));
|
||||
break;
|
||||
}
|
||||
const buildBtn = embedControls.createEl("button");
|
||||
buildBtn.setText(
|
||||
hasChunks ? t("feat_rebuild_btn") : t("feat_build_btn")
|
||||
);
|
||||
buildBtn.addClass("mod-cta");
|
||||
buildBtn.addEventListener("click", () => startBuild("--resume"));
|
||||
if (!isCorrupted && hasChunks) {
|
||||
const forceBtn2 = embedControls.createEl("button");
|
||||
forceBtn2.setText(t("feat_vector_rebuild_force_btn"));
|
||||
forceBtn2.style.marginLeft = "6px";
|
||||
forceBtn2.addEventListener("click", () => startBuild("--force"));
|
||||
|
||||
case "deps-missing": {
|
||||
embedStatusText.setText(t("retrieval_build_deps_missing"));
|
||||
// Link-style: Install Dependencies redirects to full settings display
|
||||
const installBtn = embedControls.createEl("a");
|
||||
installBtn.setText(t("feat_install_deps"));
|
||||
installBtn.style.cssText =
|
||||
"cursor:pointer; text-decoration:underline;";
|
||||
installBtn.addEventListener("click", () => {
|
||||
this.display();
|
||||
});
|
||||
break;
|
||||
}
|
||||
|
||||
case "runtime-mismatch": {
|
||||
embedStatusText.createEl("div", {
|
||||
cls: "paperforge-desc-box",
|
||||
text: t("retrieval_build_runtime_mismatch"),
|
||||
attr: { style: "color:var(--text-warning);" },
|
||||
});
|
||||
// Link-style: Sync Runtime navigates to Runtime Health section
|
||||
const syncLink = embedControls.createEl("a");
|
||||
syncLink.setText(t("runtime_health_sync"));
|
||||
syncLink.style.cssText = "cursor:pointer; text-decoration:underline;";
|
||||
syncLink.addEventListener("click", () => {
|
||||
this.display();
|
||||
});
|
||||
break;
|
||||
}
|
||||
|
||||
case "idle":
|
||||
default: {
|
||||
embedStatusText.setText(t("retrieval_build_idle"));
|
||||
// Primary CTA: Build
|
||||
const buildBtn = embedControls.createEl("button");
|
||||
buildBtn.setText(t("feat_build_btn"));
|
||||
buildBtn.className = "mod-cta";
|
||||
buildBtn.addEventListener("click", () => startBuild("--resume"));
|
||||
break;
|
||||
}
|
||||
}
|
||||
};
|
||||
|
|
|
|||
|
|
@ -51,11 +51,6 @@ import {
|
|||
restoreVersion,
|
||||
compareVersions,
|
||||
} from "../services/version-history";
|
||||
import {
|
||||
initDatabase,
|
||||
searchMetadata,
|
||||
type SearchResultItem,
|
||||
} from "../services/db";
|
||||
|
||||
// ── Interface for plugin ref used by static open ──
|
||||
|
||||
|
|
@ -101,8 +96,11 @@ export class PaperForgeStatusView extends ItemView {
|
|||
_searchInput: HTMLInputElement | null = null;
|
||||
_searchResultsEl: HTMLElement | null = null;
|
||||
_searchTimer: ReturnType<typeof setTimeout> | undefined = undefined;
|
||||
_sqlJsInitialized: boolean = false;
|
||||
_sqlJsFailed: boolean = false;
|
||||
_searchState: string = "idle";
|
||||
_searchMode: "M" | "@" = "M";
|
||||
_searchResults: unknown[] | null = null;
|
||||
_searchActiveIndex: number = -1;
|
||||
_onKeyDown: ((e: KeyboardEvent) => void) | null = null;
|
||||
|
||||
constructor(leaf: WorkspaceLeaf) {
|
||||
super(leaf);
|
||||
|
|
@ -134,8 +132,20 @@ export class PaperForgeStatusView extends ItemView {
|
|||
this._setupEventSubscriptions();
|
||||
this._fetchVersion();
|
||||
this._detectAndSwitch();
|
||||
}
|
||||
|
||||
// Global "/" keyboard shortcut to focus search input
|
||||
this._onKeyDown = (e: KeyboardEvent) => {
|
||||
if (e.key === "/" && !e.ctrlKey && !e.metaKey && !e.altKey) {
|
||||
// Only focus search when not already in an input/textarea
|
||||
const tag = (e.target as HTMLElement)?.tagName?.toLowerCase();
|
||||
if (tag !== "input" && tag !== "textarea") {
|
||||
e.preventDefault();
|
||||
this._searchInput?.focus();
|
||||
}
|
||||
}
|
||||
};
|
||||
document.addEventListener("keydown", this._onKeyDown);
|
||||
}
|
||||
async onClose() {
|
||||
if (this._modeSubscribers && this._modeSubscribers.length > 0) {
|
||||
for (const sub of this._modeSubscribers) {
|
||||
|
|
@ -151,6 +161,16 @@ export class PaperForgeStatusView extends ItemView {
|
|||
clearTimeout(this._leafChangeTimer);
|
||||
this._leafChangeTimer = null;
|
||||
}
|
||||
// Remove global keydown listener
|
||||
if (this._onKeyDown) {
|
||||
document.removeEventListener("keydown", this._onKeyDown);
|
||||
this._onKeyDown = null;
|
||||
}
|
||||
// Reset search state
|
||||
this._searchState = "idle";
|
||||
this._searchResults = null;
|
||||
this._searchActiveIndex = -1;
|
||||
this._searchTimer = undefined;
|
||||
this._cachedItems = null;
|
||||
this._cachedStats = null;
|
||||
}
|
||||
|
|
@ -187,7 +207,10 @@ export class PaperForgeStatusView extends ItemView {
|
|||
this._invalidateIndex();
|
||||
this._detectAndSwitch();
|
||||
});
|
||||
this._messageEl = root.createEl("div", { cls: "paperforge-message" });
|
||||
this._messageEl = root.createEl("div", {
|
||||
cls: "paperforge-message",
|
||||
attr: { "aria-live": "polite" },
|
||||
});
|
||||
this._contentEl = root.createEl("div", { cls: "paperforge-content-area" });
|
||||
}
|
||||
|
||||
|
|
@ -2396,51 +2419,361 @@ export class PaperForgeStatusView extends ItemView {
|
|||
cls: "paperforge-search-results",
|
||||
});
|
||||
|
||||
// Detect @ mode prefix + debounced metadata search via sql.js
|
||||
// Update placeholder from i18n on init
|
||||
this._searchInput.placeholder = t("retrieval_search_placeholder");
|
||||
|
||||
// Detect @ mode prefix + debounced metadata search
|
||||
this._searchInput.addEventListener("input", () => {
|
||||
const val = this._searchInput?.value || "";
|
||||
if (val.startsWith("@") && !val.startsWith("@ ")) {
|
||||
this._searchMode = "@";
|
||||
modeBadge.setText("@");
|
||||
modeBadge.addClass("deep");
|
||||
if (this._searchInput)
|
||||
this._searchInput.placeholder = t(
|
||||
"retrieval_search_placeholder_deep"
|
||||
);
|
||||
} else {
|
||||
this._searchMode = "M";
|
||||
modeBadge.setText("M");
|
||||
modeBadge.removeClass("deep");
|
||||
if (this._searchInput)
|
||||
this._searchInput.placeholder = t("retrieval_search_placeholder");
|
||||
}
|
||||
// Cancel pending debounce
|
||||
clearTimeout(this._searchTimer);
|
||||
// Reset results & show idle when input cleared
|
||||
if (!val.trim()) {
|
||||
this._searchState = "idle";
|
||||
this._searchResults = null;
|
||||
this._searchActiveIndex = -1;
|
||||
this._renderSearchState();
|
||||
return;
|
||||
}
|
||||
// Debounced metadata search for non-@ queries
|
||||
if (!val.startsWith("@") && val.trim()) {
|
||||
if (!val.startsWith("@")) {
|
||||
this._searchTimer = setTimeout(() => {
|
||||
this.executeSearch({ source: "sqljs" });
|
||||
this.executeSearch();
|
||||
}, 200);
|
||||
}
|
||||
});
|
||||
|
||||
// Enter triggers CLI search (full search for @ or explicit)
|
||||
// Keyboard: Enter triggers search, Escape clears, arrows navigate, Ctrl+Enter forces CLI
|
||||
this._searchInput.addEventListener("keydown", (e: KeyboardEvent) => {
|
||||
if (e.key === "Escape") {
|
||||
e.preventDefault();
|
||||
if (this._searchInput) {
|
||||
this._searchInput.value = "";
|
||||
this._searchInput.blur();
|
||||
}
|
||||
this._searchState = "idle";
|
||||
this._searchResults = null;
|
||||
this._searchActiveIndex = -1;
|
||||
this._renderSearchState();
|
||||
return;
|
||||
}
|
||||
if (e.key === "ArrowDown" || e.key === "ArrowUp") {
|
||||
if (this._searchState !== "results" || !this._searchResults?.length)
|
||||
return;
|
||||
e.preventDefault();
|
||||
const max = this._searchResults.length;
|
||||
if (e.key === "ArrowDown") {
|
||||
this._searchActiveIndex = Math.min(
|
||||
this._searchActiveIndex + 1,
|
||||
max - 1
|
||||
);
|
||||
} else {
|
||||
this._searchActiveIndex = Math.max(this._searchActiveIndex - 1, -1);
|
||||
}
|
||||
// Update aria-selected on result cards
|
||||
const cards = this._searchResultsEl?.querySelectorAll(
|
||||
".paperforge-search-result-card"
|
||||
);
|
||||
if (cards) {
|
||||
cards.forEach((c, i) => {
|
||||
if (i === this._searchActiveIndex) {
|
||||
c.setAttribute("aria-selected", "true");
|
||||
c.classList.add("active");
|
||||
} else {
|
||||
c.setAttribute("aria-selected", "false");
|
||||
c.classList.remove("active");
|
||||
}
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (e.key === "Enter" && e.ctrlKey) {
|
||||
// Ctrl+Enter forces CLI deep search regardless of mode
|
||||
e.preventDefault();
|
||||
if (this._searchTimer) {
|
||||
clearTimeout(this._searchTimer);
|
||||
this._searchTimer = undefined;
|
||||
}
|
||||
// Override mode to @ for this search
|
||||
const originalMode = this._searchMode;
|
||||
this._searchMode = "@";
|
||||
this.executeSearch();
|
||||
this._searchMode = originalMode;
|
||||
return;
|
||||
}
|
||||
if (e.key === "Enter") {
|
||||
e.preventDefault();
|
||||
if (this._searchTimer) {
|
||||
clearTimeout(this._searchTimer);
|
||||
this._searchTimer = undefined;
|
||||
}
|
||||
this.executeSearch({ source: "cli" });
|
||||
this.executeSearch();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
async executeSearch(options: { source?: "auto" | "cli" | "sqljs" } = {}) {
|
||||
_renderSearchState() {
|
||||
if (!this._searchResultsEl) return;
|
||||
this._searchResultsEl.empty();
|
||||
this._searchResultsEl.removeAttribute("role");
|
||||
this._searchResultsEl.removeAttribute("aria-live");
|
||||
// Re-enable input on any state change
|
||||
if (this._searchInput) this._searchInput.disabled = false;
|
||||
const state = this._searchState;
|
||||
|
||||
switch (state) {
|
||||
case "idle": {
|
||||
// Show placeholder hint in the input's placeholder — already set by input handler
|
||||
// Keep results area empty
|
||||
break;
|
||||
}
|
||||
case "searching": {
|
||||
const isDeep = this._searchMode === "@";
|
||||
this._searchResultsEl.createEl("div", {
|
||||
cls: "paperforge-search-loading",
|
||||
text: isDeep
|
||||
? t("retrieval_searching_deep")
|
||||
: t("retrieval_searching_metadata"),
|
||||
});
|
||||
this._searchResultsEl.setAttr("aria-live", "polite");
|
||||
// Disable input during deep search
|
||||
if (isDeep && this._searchInput) {
|
||||
this._searchInput.disabled = true;
|
||||
}
|
||||
break;
|
||||
}
|
||||
case "results": {
|
||||
this._searchResultsEl.setAttr("role", "listbox");
|
||||
this._searchResultsEl.setAttr("aria-live", "polite");
|
||||
if (this._searchResults) {
|
||||
this._renderSearchResultsList(
|
||||
this._searchResults,
|
||||
this._searchMode === "@"
|
||||
);
|
||||
}
|
||||
// Focus management: focus first result after a short delay
|
||||
setTimeout(() => {
|
||||
const firstCard = this._searchResultsEl?.querySelector(
|
||||
".paperforge-search-result-card"
|
||||
);
|
||||
if (firstCard && firstCard instanceof HTMLElement) {
|
||||
firstCard.focus();
|
||||
}
|
||||
}, 100);
|
||||
break;
|
||||
}
|
||||
case "empty": {
|
||||
const emptyEl = this._searchResultsEl.createEl("div", {
|
||||
cls: "paperforge-search-empty",
|
||||
});
|
||||
emptyEl.setAttr("role", "alert");
|
||||
emptyEl.createEl("div", {
|
||||
text: t("retrieval_empty"),
|
||||
});
|
||||
emptyEl.createEl("div", {
|
||||
cls: "paperforge-search-empty-tips",
|
||||
text: t("retrieval_empty_tips"),
|
||||
});
|
||||
break;
|
||||
}
|
||||
case "vectors-not-built": {
|
||||
const warn = this._searchResultsEl.createEl("div", {
|
||||
cls: "paperforge-search-state-card",
|
||||
attr: { role: "alert" },
|
||||
});
|
||||
warn.addClass("warning-soft");
|
||||
warn.createEl("div", {
|
||||
cls: "paperforge-search-state-title",
|
||||
text: t("retrieval_vectors_not_built"),
|
||||
});
|
||||
warn.createEl("div", {
|
||||
cls: "paperforge-search-state-desc",
|
||||
text: t("retrieval_vectors_not_built_desc"),
|
||||
});
|
||||
const btn = warn.createEl("button", {
|
||||
cls: "pf-btn-link",
|
||||
text: t("retrieval_open_vector_settings"),
|
||||
});
|
||||
btn.addEventListener("click", () => {
|
||||
const setting = (this.app as unknown as Record<string, unknown>)[
|
||||
"setting"
|
||||
];
|
||||
if (setting && typeof setting === "object") {
|
||||
const openTab = (setting as Record<string, unknown>)["openTab"];
|
||||
if (typeof openTab === "function") {
|
||||
openTab.call(setting, "paperforge");
|
||||
}
|
||||
}
|
||||
});
|
||||
// Focus management: focus the action button
|
||||
setTimeout(() => {
|
||||
btn.focus();
|
||||
}, 100);
|
||||
break;
|
||||
}
|
||||
case "backend-unavailable": {
|
||||
const err = this._searchResultsEl.createEl("div", {
|
||||
cls: "paperforge-search-state-card",
|
||||
attr: { role: "alert" },
|
||||
});
|
||||
err.addClass("error-soft");
|
||||
err.createEl("div", {
|
||||
cls: "paperforge-search-state-title",
|
||||
text: t("retrieval_backend_unavailable"),
|
||||
});
|
||||
err.createEl("div", {
|
||||
cls: "paperforge-search-state-desc",
|
||||
text: t("retrieval_backend_unavailable_desc"),
|
||||
});
|
||||
const actions = err.createEl("div", {
|
||||
cls: "paperforge-search-state-actions",
|
||||
});
|
||||
const doctorBtn = actions.createEl("button", {
|
||||
cls: "pf-btn-primary",
|
||||
text: t("retrieval_run_doctor"),
|
||||
});
|
||||
doctorBtn.addEventListener("click", () => {
|
||||
const vp = (
|
||||
this.app.vault.adapter as unknown as Record<string, unknown>
|
||||
)["basePath"];
|
||||
if (typeof vp === "string") {
|
||||
const { path: pyExe, extraArgs = [] } = resolvePythonExecutable(
|
||||
vp,
|
||||
null,
|
||||
undefined,
|
||||
undefined
|
||||
);
|
||||
spawn(pyExe, [...extraArgs, "-m", "paperforge", "doctor"], {
|
||||
cwd: vp,
|
||||
stdio: "inherit",
|
||||
});
|
||||
}
|
||||
});
|
||||
const retryBtn = actions.createEl("button", {
|
||||
cls: "pf-btn-secondary",
|
||||
text: t("retrieval_retry"),
|
||||
});
|
||||
retryBtn.addEventListener("click", () => {
|
||||
this.executeSearch();
|
||||
});
|
||||
// Focus: primary action
|
||||
setTimeout(() => {
|
||||
doctorBtn.focus();
|
||||
}, 100);
|
||||
break;
|
||||
}
|
||||
case "timeout": {
|
||||
const err = this._searchResultsEl.createEl("div", {
|
||||
cls: "paperforge-search-state-card",
|
||||
attr: { role: "alert" },
|
||||
});
|
||||
err.addClass("warning-soft");
|
||||
err.createEl("div", {
|
||||
cls: "paperforge-search-state-title",
|
||||
text: t("retrieval_timeout_title"),
|
||||
});
|
||||
err.createEl("div", {
|
||||
cls: "paperforge-search-state-desc",
|
||||
text: t("retrieval_timeout_desc"),
|
||||
});
|
||||
const retryBtn = err.createEl("button", {
|
||||
cls: "pf-btn-primary",
|
||||
text: t("retrieval_retry"),
|
||||
});
|
||||
retryBtn.addEventListener("click", () => {
|
||||
this.executeSearch();
|
||||
});
|
||||
// Focus: retry button
|
||||
setTimeout(() => {
|
||||
retryBtn.focus();
|
||||
}, 100);
|
||||
break;
|
||||
}
|
||||
case "model-changed": {
|
||||
const warn = this._searchResultsEl.createEl("div", {
|
||||
cls: "paperforge-search-state-card",
|
||||
attr: { role: "alert" },
|
||||
});
|
||||
warn.addClass("warning-soft");
|
||||
warn.createEl("div", {
|
||||
cls: "paperforge-search-state-title",
|
||||
text: t("retrieval_model_changed"),
|
||||
});
|
||||
warn.createEl("div", {
|
||||
cls: "paperforge-search-state-desc",
|
||||
text: t("retrieval_model_changed_desc"),
|
||||
});
|
||||
const rebuildBtn = warn.createEl("button", {
|
||||
cls: "pf-btn-primary",
|
||||
text: t("retrieval_rebuild_vectors"),
|
||||
});
|
||||
rebuildBtn.addEventListener("click", () => {
|
||||
// Trigger rebuild from settings
|
||||
const setting = (this.app as unknown as Record<string, unknown>)[
|
||||
"setting"
|
||||
];
|
||||
if (setting && typeof setting === "object") {
|
||||
const openTab = (setting as Record<string, unknown>)["openTab"];
|
||||
if (typeof openTab === "function") {
|
||||
openTab.call(setting, "paperforge");
|
||||
}
|
||||
}
|
||||
});
|
||||
// Focus: rebuild button
|
||||
setTimeout(() => {
|
||||
rebuildBtn.focus();
|
||||
}, 100);
|
||||
break;
|
||||
}
|
||||
default: {
|
||||
// Fallback: show internal error for unknown states
|
||||
const err = this._searchResultsEl.createEl("div", {
|
||||
cls: "paperforge-search-state-card",
|
||||
text: t("retrieval_internal_error"),
|
||||
attr: { role: "alert" },
|
||||
});
|
||||
err.addClass("error-soft");
|
||||
setTimeout(() => {
|
||||
if (this._searchInput) this._searchInput.focus();
|
||||
}, 100);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async executeSearch() {
|
||||
if (!this._searchInput || !this._searchResultsEl) return;
|
||||
const raw = this._searchInput.value.trim();
|
||||
if (!raw) return;
|
||||
|
||||
const isDeep = raw.startsWith("@");
|
||||
const query = isDeep ? raw.slice(1).trim() : raw;
|
||||
const isDeep = this._searchMode === "@" || raw.startsWith("@");
|
||||
const query = isDeep ? raw.replace(/^@\s*/, "").trim() : raw;
|
||||
if (!query) return;
|
||||
|
||||
const mode = isDeep ? "retrieve" : "search";
|
||||
|
||||
// Resolve vault path (used by both sql.js and CLI paths)
|
||||
// Set searching state immediately
|
||||
this._searchState = "searching";
|
||||
this._searchResults = null;
|
||||
this._searchActiveIndex = -1;
|
||||
this._renderSearchState();
|
||||
|
||||
// Resolve vault path for CLI search
|
||||
const adapter = this.app.vault.adapter;
|
||||
let vaultPath = "";
|
||||
if (adapter && typeof adapter === "object" && "basePath" in adapter) {
|
||||
|
|
@ -2448,42 +2781,9 @@ export class PaperForgeStatusView extends ItemView {
|
|||
vaultPath = typeof bp === "string" ? bp : "";
|
||||
}
|
||||
|
||||
this._searchResultsEl.empty();
|
||||
|
||||
// ── sql.js metadata search path (non-@ queries only) ──
|
||||
if (
|
||||
mode === "search" &&
|
||||
(options.source === "auto" || options.source === "sqljs")
|
||||
) {
|
||||
if (vaultPath) {
|
||||
try {
|
||||
if (!this._sqlJsInitialized && !this._sqlJsFailed) {
|
||||
await initDatabase(vaultPath);
|
||||
this._sqlJsInitialized = true;
|
||||
}
|
||||
if (this._sqlJsInitialized) {
|
||||
const results = searchMetadata(query, 20);
|
||||
if (results !== null) {
|
||||
this.renderSearchResults(results, false);
|
||||
return;
|
||||
}
|
||||
}
|
||||
} catch (e) {
|
||||
console.error("PaperForge sql.js search failed:", e);
|
||||
this._sqlJsFailed = true;
|
||||
}
|
||||
}
|
||||
// sql.js unavailable — fall through to CLI
|
||||
}
|
||||
|
||||
// ── CLI search path (deep search or sql.js fallback) ──
|
||||
this._searchResultsEl.createEl("div", {
|
||||
cls: "paperforge-search-loading",
|
||||
text: "Searching...",
|
||||
});
|
||||
|
||||
if (!vaultPath) {
|
||||
this._renderSearchError("Could not determine vault path");
|
||||
this._searchState = "backend-unavailable";
|
||||
this._renderSearchState();
|
||||
return;
|
||||
}
|
||||
|
||||
|
|
@ -2515,10 +2815,20 @@ export class PaperForgeStatusView extends ItemView {
|
|||
undefined,
|
||||
undefined
|
||||
);
|
||||
|
||||
const deepFlag = mode === "retrieve" ? ["--deep"] : [];
|
||||
const child = spawn(
|
||||
pythonExe,
|
||||
[...pyExtra, "-m", "paperforge", mode, query, "--json"],
|
||||
[
|
||||
...pyExtra,
|
||||
"-m",
|
||||
"paperforge",
|
||||
"--vault",
|
||||
vaultPath,
|
||||
mode,
|
||||
query,
|
||||
...deepFlag,
|
||||
"--json",
|
||||
],
|
||||
{ cwd: vaultPath, timeout: 30000 }
|
||||
);
|
||||
|
||||
|
|
@ -2531,7 +2841,10 @@ export class PaperForgeStatusView extends ItemView {
|
|||
});
|
||||
child.on("close", (code: number | null) => {
|
||||
if (code !== 0) {
|
||||
this._renderSearchError(`Search failed (exit ${code})`);
|
||||
// Map exit code to error state
|
||||
const classification = classifyError(String(code));
|
||||
this._searchState = this._mapErrorToSearchState(classification.type);
|
||||
this._renderSearchState();
|
||||
return;
|
||||
}
|
||||
const rawOutput = chunks.join("");
|
||||
|
|
@ -2550,7 +2863,8 @@ export class PaperForgeStatusView extends ItemView {
|
|||
}
|
||||
}
|
||||
if (!jsonStr) {
|
||||
this._renderSearchError("No JSON output from CLI");
|
||||
this._searchState = "internal-error";
|
||||
this._renderSearchState();
|
||||
return;
|
||||
}
|
||||
try {
|
||||
|
|
@ -2561,29 +2875,64 @@ export class PaperForgeStatusView extends ItemView {
|
|||
const d = (parsed as Record<string, unknown>).data;
|
||||
if (d && typeof d === "object") {
|
||||
const dd = d as Record<string, unknown>;
|
||||
// search output: data.matches
|
||||
// Unified PFResult v1: data.matches
|
||||
if ("matches" in dd && Array.isArray(dd.matches)) {
|
||||
results = dd.matches as unknown[];
|
||||
} else if ("results" in dd && Array.isArray(dd.results)) {
|
||||
results = dd.results as unknown[];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
this.renderSearchResults(results, isDeep);
|
||||
this._searchResults = results;
|
||||
this._searchState = results.length > 0 ? "results" : "empty";
|
||||
this._renderSearchState();
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
this._renderSearchError("Failed to parse results: " + msg);
|
||||
this._searchState = "internal-error";
|
||||
this._renderSearchState();
|
||||
}
|
||||
});
|
||||
child.on("error", (err: Error) => {
|
||||
this._renderSearchError("Process error: " + err.message);
|
||||
// Check for structured error in stderr
|
||||
const errCode = (err as unknown as Record<string, unknown>)["code"];
|
||||
if (typeof errCode === "string") {
|
||||
const classification = classifyError(errCode);
|
||||
this._searchState = this._mapErrorToSearchState(classification.type);
|
||||
} else {
|
||||
this._searchState = "backend-unavailable";
|
||||
}
|
||||
this._renderSearchState();
|
||||
});
|
||||
}
|
||||
|
||||
renderSearchResults(results: unknown[], isDeep: boolean) {
|
||||
/** Map classifyError types to _searchState values */
|
||||
_mapErrorToSearchState(errorType: string): string {
|
||||
switch (errorType) {
|
||||
case "vectors_not_built":
|
||||
return "vectors-not-built";
|
||||
case "vectors_corrupted":
|
||||
return "vectors-not-built";
|
||||
case "backend_unavailable":
|
||||
return "backend-unavailable";
|
||||
case "model_changed":
|
||||
return "model-changed";
|
||||
case "timeout":
|
||||
return "timeout";
|
||||
case "no_python":
|
||||
case "python_missing":
|
||||
case "import_failed":
|
||||
case "version_mismatch":
|
||||
return "backend-unavailable";
|
||||
default:
|
||||
return "backend-unavailable";
|
||||
}
|
||||
}
|
||||
|
||||
_renderSearchResultsList(results: unknown[], isDeep: boolean) {
|
||||
if (!this._searchResultsEl) return;
|
||||
this._searchResultsEl.empty();
|
||||
|
||||
// Container already has role="listbox" set by _renderSearchState
|
||||
// Add aria-live polite for live updates
|
||||
this._searchResultsEl.setAttr("aria-live", "polite");
|
||||
|
||||
if (results.length === 0) {
|
||||
this._searchResultsEl.createEl("div", {
|
||||
|
|
@ -2596,21 +2945,33 @@ export class PaperForgeStatusView extends ItemView {
|
|||
const header = this._searchResultsEl.createEl("div", {
|
||||
cls: "paperforge-search-results-header",
|
||||
});
|
||||
header.createEl("span", {
|
||||
text: `${results.length} result${results.length !== 1 ? "s" : ""}`,
|
||||
const countSpan = header.createEl("span", {
|
||||
text: t("retrieval_results_count")
|
||||
.replace("{n}", String(results.length))
|
||||
.replace("{s}", results.length !== 1 ? "s" : ""),
|
||||
});
|
||||
countSpan.setAttr("aria-live", "polite");
|
||||
header.createEl("span", {
|
||||
cls: "paperforge-search-mode",
|
||||
text: isDeep ? "@" : "M",
|
||||
});
|
||||
|
||||
for (const r of results) {
|
||||
for (let i = 0; i < results.length; i++) {
|
||||
const r = results[i];
|
||||
if (!r || typeof r !== "object") continue;
|
||||
const rec = r as Record<string, unknown>;
|
||||
const isActive = i === this._searchActiveIndex;
|
||||
const card = this._searchResultsEl.createEl("div", {
|
||||
cls: "paperforge-search-result-card",
|
||||
attr: { role: "button" },
|
||||
attr: {
|
||||
role: "option",
|
||||
tabindex: "0",
|
||||
"aria-selected": isActive ? "true" : "false",
|
||||
"aria-posinset": String(i + 1),
|
||||
"aria-setsize": String(results.length),
|
||||
},
|
||||
});
|
||||
if (isActive) card.addClass("active");
|
||||
|
||||
// Title
|
||||
const titleText =
|
||||
|
|
@ -2669,26 +3030,24 @@ export class PaperForgeStatusView extends ItemView {
|
|||
});
|
||||
}
|
||||
|
||||
// Enter key on card opens the note
|
||||
card.addEventListener("keydown", (e: KeyboardEvent) => {
|
||||
if (e.key === "Enter" && resolvedPath) {
|
||||
e.preventDefault();
|
||||
const newLeaf = e.ctrlKey || e.metaKey;
|
||||
this.app.workspace.openLinkText(resolvedPath, "", newLeaf);
|
||||
}
|
||||
});
|
||||
|
||||
// Meta row
|
||||
const meta = card.createEl("div", {
|
||||
cls: "paperforge-search-result-meta",
|
||||
});
|
||||
|
||||
if (typeof rec["authors"] === "string") {
|
||||
if (typeof rec["first_author"] === "string" && rec["first_author"]) {
|
||||
meta.createEl("span", {
|
||||
cls: "paperforge-search-result-author",
|
||||
text: rec["authors"],
|
||||
});
|
||||
} else if (Array.isArray(rec["authors"])) {
|
||||
meta.createEl("span", {
|
||||
cls: "paperforge-search-result-author",
|
||||
text: (rec["authors"] as string[]).slice(0, 3).join("; "),
|
||||
});
|
||||
}
|
||||
if (typeof rec["year"] === "number" || typeof rec["year"] === "string") {
|
||||
meta.createEl("span", {
|
||||
cls: "paperforge-search-result-year",
|
||||
text: String(rec["year"]),
|
||||
text: rec["first_author"],
|
||||
});
|
||||
}
|
||||
if (typeof rec["journal"] === "string" && rec["journal"]) {
|
||||
|
|
@ -2725,12 +3084,8 @@ export class PaperForgeStatusView extends ItemView {
|
|||
}
|
||||
|
||||
// @ mode: matched text
|
||||
if (
|
||||
isDeep &&
|
||||
typeof rec["matched_text"] === "string" &&
|
||||
rec["matched_text"]
|
||||
) {
|
||||
const mt = rec["matched_text"] as string;
|
||||
if (isDeep && typeof rec["text"] === "string" && rec["text"]) {
|
||||
const mt = rec["text"] as string;
|
||||
card.createEl("div", {
|
||||
cls: "paperforge-search-result-source",
|
||||
text: mt.length > 300 ? mt.slice(0, 300) + "..." : mt,
|
||||
|
|
|
|||
|
|
@ -1,91 +1,143 @@
|
|||
import { describe, it, expect } from 'vitest';
|
||||
import { describe, it, expect } from "vitest";
|
||||
|
||||
import {
|
||||
classifyError,
|
||||
parseRuntimeStatus,
|
||||
classifyError,
|
||||
parseRuntimeStatus,
|
||||
} from "../src/services/python-bridge";
|
||||
|
||||
describe('classifyError', () => {
|
||||
it('classifies ENOENT as python_missing', () => {
|
||||
const result = classifyError('ENOENT');
|
||||
expect(result.type).toBe('python_missing');
|
||||
expect(result.recoverable).toBe(true);
|
||||
expect(result.message).toContain('Python');
|
||||
});
|
||||
describe("classifyError", () => {
|
||||
it("classifies ENOENT as python_missing", () => {
|
||||
const result = classifyError("ENOENT");
|
||||
expect(result.type).toBe("python_missing");
|
||||
expect(result.recoverable).toBe(true);
|
||||
expect(result.message).toContain("Python");
|
||||
});
|
||||
|
||||
it('classifies MODULE_NOT_FOUND as import_failed', () => {
|
||||
const result = classifyError('MODULE_NOT_FOUND');
|
||||
expect(result.type).toBe('import_failed');
|
||||
expect(result.recoverable).toBe(true);
|
||||
});
|
||||
it("classifies MODULE_NOT_FOUND as import_failed", () => {
|
||||
const result = classifyError("MODULE_NOT_FOUND");
|
||||
expect(result.type).toBe("import_failed");
|
||||
expect(result.recoverable).toBe(true);
|
||||
});
|
||||
|
||||
it('classifies version-mismatch with sync-runtime action', () => {
|
||||
const result = classifyError('version-mismatch');
|
||||
expect(result.type).toBe('version_mismatch');
|
||||
expect(result.recoverable).toBe(true);
|
||||
expect(result.action).toBe('sync-runtime');
|
||||
});
|
||||
it("classifies version-mismatch with sync-runtime action", () => {
|
||||
const result = classifyError("version-mismatch");
|
||||
expect(result.type).toBe("version_mismatch");
|
||||
expect(result.recoverable).toBe(true);
|
||||
expect(result.action).toBe("sync-runtime");
|
||||
});
|
||||
|
||||
it('classifies pip-failed as pip_install_failure', () => {
|
||||
const result = classifyError('pip-failed');
|
||||
expect(result.type).toBe('pip_install_failure');
|
||||
expect(result.recoverable).toBe(true);
|
||||
});
|
||||
it("classifies pip-failed as pip_install_failure", () => {
|
||||
const result = classifyError("pip-failed");
|
||||
expect(result.type).toBe("pip_install_failure");
|
||||
expect(result.recoverable).toBe(true);
|
||||
});
|
||||
|
||||
it('classifies ETIMEDOUT as timeout with retry action', () => {
|
||||
const result = classifyError('ETIMEDOUT');
|
||||
expect(result.type).toBe('timeout');
|
||||
expect(result.recoverable).toBe(true);
|
||||
expect(result.action).toBe('retry');
|
||||
});
|
||||
it("classifies ETIMEDOUT as timeout with retry action", () => {
|
||||
const result = classifyError("ETIMEDOUT");
|
||||
expect(result.type).toBe("timeout");
|
||||
expect(result.recoverable).toBe(true);
|
||||
expect(result.action).toBe("retry");
|
||||
});
|
||||
|
||||
it('classifies unknown error strings as unknown', () => {
|
||||
const result = classifyError('SOME_RANDOM_ERROR');
|
||||
expect(result.type).toBe('unknown');
|
||||
expect(result.recoverable).toBe(false);
|
||||
expect(result.message).toBe('SOME_RANDOM_ERROR');
|
||||
});
|
||||
it("classifies NO_PYTHON as no_python with open-setup action", () => {
|
||||
const result = classifyError("NO_PYTHON");
|
||||
expect(result.type).toBe("no_python");
|
||||
expect(result.recoverable).toBe(true);
|
||||
expect(result.action).toBe("open-setup");
|
||||
});
|
||||
|
||||
it('classifies numeric exit codes as unknown', () => {
|
||||
const result = classifyError(1);
|
||||
expect(result.type).toBe('unknown');
|
||||
expect(result.recoverable).toBe(false);
|
||||
});
|
||||
it("classifies VECTOR_NOT_BUILT as vectors_not_built with open-vector-settings action", () => {
|
||||
const result = classifyError("VECTOR_NOT_BUILT");
|
||||
expect(result.type).toBe("vectors_not_built");
|
||||
expect(result.recoverable).toBe(true);
|
||||
expect(result.action).toBe("open-vector-settings");
|
||||
});
|
||||
|
||||
it("classifies VECTOR_CORRUPTED as vectors_corrupted with force-rebuild action", () => {
|
||||
const result = classifyError("VECTOR_CORRUPTED");
|
||||
expect(result.type).toBe("vectors_corrupted");
|
||||
expect(result.recoverable).toBe(true);
|
||||
expect(result.action).toBe("force-rebuild");
|
||||
});
|
||||
|
||||
it("classifies MODEL_CHANGED as model_changed with rebuild-vectors action", () => {
|
||||
const result = classifyError("MODEL_CHANGED");
|
||||
expect(result.type).toBe("model_changed");
|
||||
expect(result.recoverable).toBe(true);
|
||||
expect(result.action).toBe("rebuild-vectors");
|
||||
});
|
||||
|
||||
it("classifies BACKEND_UNAVAILABLE as backend_unavailable with run-doctor action", () => {
|
||||
const result = classifyError("BACKEND_UNAVAILABLE");
|
||||
expect(result.type).toBe("backend_unavailable");
|
||||
expect(result.recoverable).toBe(true);
|
||||
expect(result.action).toBe("run-doctor");
|
||||
});
|
||||
|
||||
it("classifies TIMEOUT as timeout with retry action", () => {
|
||||
const result = classifyError("TIMEOUT");
|
||||
expect(result.type).toBe("timeout");
|
||||
expect(result.recoverable).toBe(true);
|
||||
expect(result.action).toBe("retry");
|
||||
expect(result.message).toContain("Search");
|
||||
});
|
||||
|
||||
it("classifies INTERNAL_ERROR as internal_error not recoverable", () => {
|
||||
const result = classifyError("INTERNAL_ERROR");
|
||||
expect(result.type).toBe("internal_error");
|
||||
expect(result.recoverable).toBe(false);
|
||||
});
|
||||
it("classifies unknown error strings as unknown", () => {
|
||||
const result = classifyError("SOME_RANDOM_ERROR");
|
||||
expect(result.type).toBe("unknown");
|
||||
expect(result.recoverable).toBe(false);
|
||||
expect(result.message).toBe("SOME_RANDOM_ERROR");
|
||||
});
|
||||
|
||||
it("classifies numeric exit codes as unknown", () => {
|
||||
const result = classifyError(1);
|
||||
expect(result.type).toBe("unknown");
|
||||
expect(result.recoverable).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseRuntimeStatus', () => {
|
||||
it('returns ok with version on success', () => {
|
||||
const result = parseRuntimeStatus(null, '1.4.17rc3\n', '');
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.version).toBe('1.4.17rc3');
|
||||
});
|
||||
describe("parseRuntimeStatus", () => {
|
||||
it("returns ok with version on success", () => {
|
||||
const result = parseRuntimeStatus(null, "1.4.17rc3\n", "");
|
||||
expect(result.status).toBe("ok");
|
||||
expect(result.version).toBe("1.4.17rc3");
|
||||
});
|
||||
|
||||
it('classifies ENOENT as python_missing', () => {
|
||||
const err = new Error('spawn ENOENT');
|
||||
err.code = 'ENOENT';
|
||||
const result = parseRuntimeStatus(err, null, '');
|
||||
expect(result.status).toBe('error');
|
||||
expect(result.type).toBe('python_missing');
|
||||
});
|
||||
it("classifies ENOENT as python_missing", () => {
|
||||
const err = new Error("spawn ENOENT");
|
||||
err.code = "ENOENT";
|
||||
const result = parseRuntimeStatus(err, null, "");
|
||||
expect(result.status).toBe("error");
|
||||
expect(result.type).toBe("python_missing");
|
||||
});
|
||||
|
||||
it('classifies ModuleNotFoundError in stderr as import_failed', () => {
|
||||
const result = parseRuntimeStatus(new Error('fail'), null, 'No module named paperforge');
|
||||
expect(result.status).toBe('error');
|
||||
expect(result.type).toBe('import_failed');
|
||||
});
|
||||
it("classifies ModuleNotFoundError in stderr as import_failed", () => {
|
||||
const result = parseRuntimeStatus(
|
||||
new Error("fail"),
|
||||
null,
|
||||
"No module named paperforge"
|
||||
);
|
||||
expect(result.status).toBe("error");
|
||||
expect(result.type).toBe("import_failed");
|
||||
});
|
||||
|
||||
it('classifies killed/timeout subprocess as timeout', () => {
|
||||
const err = new Error('timeout');
|
||||
err.killed = true;
|
||||
const result = parseRuntimeStatus(err, null, '');
|
||||
expect(result.status).toBe('error');
|
||||
expect(result.type).toBe('timeout');
|
||||
});
|
||||
it("classifies killed/timeout subprocess as timeout", () => {
|
||||
const err = new Error("timeout");
|
||||
err.killed = true;
|
||||
const result = parseRuntimeStatus(err, null, "");
|
||||
expect(result.status).toBe("error");
|
||||
expect(result.type).toBe("timeout");
|
||||
});
|
||||
|
||||
it('classifies generic error as unknown', () => {
|
||||
const err = new Error('Something went wrong');
|
||||
const result = parseRuntimeStatus(err, null, 'some output');
|
||||
expect(result.status).toBe('error');
|
||||
expect(result.type).toBe('unknown');
|
||||
});
|
||||
it("classifies generic error as unknown", () => {
|
||||
const err = new Error("Something went wrong");
|
||||
const result = parseRuntimeStatus(err, null, "some output");
|
||||
expect(result.status).toBe("error");
|
||||
expect(result.type).toBe("unknown");
|
||||
});
|
||||
});
|
||||
|
|
|
|||
138
tests/integration/test_health_probe.py
Normal file
138
tests/integration/test_health_probe.py
Normal file
|
|
@ -0,0 +1,138 @@
|
|||
"""Integration tests for embed status vec0 health probe.
|
||||
|
||||
Verifies that embed status --json returns healthy=false when a vec0 table
|
||||
with meta rows is dropped (simulating corruption).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
PYTHON = Path(r"D:\L\OB\Literature-hub\.venv\Scripts\python.exe")
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
|
||||
|
||||
def _build_vault(tmp_path: Path) -> Path:
|
||||
"""Build a minimal vault with paperforge.json and a seeded memory DB."""
|
||||
vault = tmp_path / "vault"
|
||||
vault.mkdir(parents=True)
|
||||
(vault / "paperforge.json").write_text(
|
||||
'{"vault_config":{"system_dir":"System"}}', encoding="utf-8"
|
||||
)
|
||||
(vault / "System" / "PaperForge").mkdir(parents=True)
|
||||
return vault
|
||||
|
||||
|
||||
def _run_embed_status(vault: Path) -> dict:
|
||||
"""Run embed status --json and return the parsed data dict."""
|
||||
result = subprocess.run(
|
||||
[
|
||||
str(PYTHON),
|
||||
"-m",
|
||||
"paperforge",
|
||||
"--vault",
|
||||
str(vault),
|
||||
"embed",
|
||||
"status",
|
||||
"--json",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=str(REPO_ROOT),
|
||||
timeout=30,
|
||||
)
|
||||
assert result.returncode == 0, f"embed status failed:\n{result.stderr}"
|
||||
parsed = json.loads(result.stdout)
|
||||
assert parsed["ok"] is True
|
||||
return parsed["data"]
|
||||
|
||||
|
||||
def _db_path(vault: Path) -> Path:
|
||||
"""Resolve paperforge.db path using the same logic as get_memory_db_path."""
|
||||
from paperforge.memory.db import get_memory_db_path
|
||||
return get_memory_db_path(vault)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def vault_with_vec0(tmp_path: Path) -> Path:
|
||||
"""Create a vault with a seeded paperforge.db containing vec0 tables + meta rows."""
|
||||
vault = _build_vault(tmp_path)
|
||||
|
||||
# Create DB schema explicitly (embed status doesn't auto-create)
|
||||
from paperforge.memory.db import get_connection, get_memory_db_path, ensure_vec_extension
|
||||
from paperforge.memory.schema import ensure_schema
|
||||
|
||||
db_path = get_memory_db_path(vault)
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = get_connection(db_path)
|
||||
ensure_vec_extension(conn)
|
||||
ensure_schema(conn)
|
||||
conn.close()
|
||||
|
||||
# Insert data via a separate connection with vec0 loaded
|
||||
import sqlite3
|
||||
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
try:
|
||||
conn.enable_load_extension(True)
|
||||
import sqlite_vec
|
||||
sqlite_vec.load(conn)
|
||||
conn.enable_load_extension(False)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Insert meta row into vec_fulltext_meta so total_meta > 0
|
||||
conn.execute(
|
||||
"INSERT INTO vec_fulltext_meta (rowid, paper_id, chunk_index, text, source) "
|
||||
"VALUES (1, 'TESTKEY', 0, 'test', 'fulltext')"
|
||||
)
|
||||
|
||||
# Insert a zero vector into vec_fulltext so the k-NN probe works
|
||||
zero_vec = [0.0] * 1536
|
||||
vec_json = json.dumps(zero_vec)
|
||||
conn.execute(
|
||||
"INSERT INTO vec_fulltext (rowid, embedding) VALUES (1, ?)",
|
||||
(vec_json,),
|
||||
)
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return vault
|
||||
|
||||
|
||||
class TestHealthProbe:
|
||||
"""embed status --json runs vec0 k-NN probe and reports healthy=false when vec0 broken."""
|
||||
|
||||
def test_healthy_when_vec0_ok(self, vault_with_vec0: Path):
|
||||
"""When vec0 tables are intact, health probe passes."""
|
||||
data = _run_embed_status(vault_with_vec0)
|
||||
assert data["healthy"] is True, f"Expected healthy=True, got {data}"
|
||||
assert data["corrupted"] is False
|
||||
assert data["chunk_count"] >= 1
|
||||
|
||||
def test_unhealthy_when_vec0_table_dropped(self, vault_with_vec0: Path):
|
||||
"""When a vec0 virtual table is dropped but meta rows exist, healthy=false."""
|
||||
db_path = _db_path(vault_with_vec0)
|
||||
import sqlite3
|
||||
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
try:
|
||||
conn.enable_load_extension(True)
|
||||
import sqlite_vec
|
||||
sqlite_vec.load(conn)
|
||||
conn.enable_load_extension(False)
|
||||
except Exception:
|
||||
pass
|
||||
conn.execute("DROP TABLE IF EXISTS vec_fulltext")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
data = _run_embed_status(vault_with_vec0)
|
||||
assert data["healthy"] is False, f"Expected healthy=False after dropping vec table, got {data}"
|
||||
assert data["corrupted"] is True
|
||||
err = data.get("error", "").lower()
|
||||
assert "vec0 probe failed" in err or "no such table" in err, f"Unexpected error: {data.get('error')}"
|
||||
154
tests/integration/test_stop_idle_resume.py
Normal file
154
tests/integration/test_stop_idle_resume.py
Normal file
|
|
@ -0,0 +1,154 @@
|
|||
"""Integration tests for the build lifecycle stop → idle → resume cycle.
|
||||
|
||||
Verifies that embed stop --json marks build as stopping, settles to idle,
|
||||
and that resume can continue a build from progress.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
PYTHON = Path(r"D:\L\OB\Literature-hub\.venv\Scripts\python.exe")
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
|
||||
|
||||
def _build_vault(tmp_path: Path) -> Path:
|
||||
"""Build a minimal vault with paperforge.json."""
|
||||
vault = tmp_path / "vault"
|
||||
vault.mkdir(parents=True)
|
||||
(vault / "paperforge.json").write_text(
|
||||
'{"vault_config":{"system_dir":"System"}}', encoding="utf-8"
|
||||
)
|
||||
(vault / "System" / "PaperForge").mkdir(parents=True)
|
||||
return vault
|
||||
|
||||
|
||||
def _run_embed_stop(vault: Path) -> dict:
|
||||
"""Run embed stop --json and return parsed data."""
|
||||
result = subprocess.run(
|
||||
[
|
||||
str(PYTHON),
|
||||
"-m",
|
||||
"paperforge",
|
||||
"--vault",
|
||||
str(vault),
|
||||
"embed",
|
||||
"stop",
|
||||
"--json",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=str(REPO_ROOT),
|
||||
timeout=30,
|
||||
)
|
||||
parsed = json.loads(result.stdout)
|
||||
return parsed
|
||||
|
||||
|
||||
def _run_embed_status(vault: Path) -> dict:
|
||||
"""Run embed status --json and return parsed data."""
|
||||
result = subprocess.run(
|
||||
[
|
||||
str(PYTHON),
|
||||
"-m",
|
||||
"paperforge",
|
||||
"--vault",
|
||||
str(vault),
|
||||
"embed",
|
||||
"status",
|
||||
"--json",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=str(REPO_ROOT),
|
||||
timeout=30,
|
||||
)
|
||||
assert result.returncode == 0
|
||||
parsed = json.loads(result.stdout)
|
||||
assert parsed["ok"] is True
|
||||
return parsed["data"]
|
||||
|
||||
|
||||
def _mark_running(vault: Path, pid: int, current: int = 5, total: int = 100) -> None:
|
||||
"""Set build_state to 'running' with the given PID."""
|
||||
from paperforge.embedding.build_state import mark_vector_build_state
|
||||
|
||||
mark_vector_build_state(
|
||||
vault,
|
||||
status="running",
|
||||
pid=pid,
|
||||
current=current,
|
||||
total=total,
|
||||
paper_id="TESTKEY",
|
||||
model="test-model",
|
||||
mode="api",
|
||||
)
|
||||
|
||||
|
||||
def _read_build_state(vault: Path) -> dict:
|
||||
"""Read build_state table."""
|
||||
from paperforge.embedding.build_state import read_vector_build_state
|
||||
|
||||
return read_vector_build_state(vault)
|
||||
|
||||
|
||||
class TestStopIdleResume:
|
||||
"""Stop → idle → resume lifecycle."""
|
||||
|
||||
def test_stop_returns_stopped_and_idle(self, tmp_path: Path):
|
||||
"""embed stop --json returns {state: stopped} when a build is running, and settles to idle."""
|
||||
vault = _build_vault(tmp_path)
|
||||
|
||||
# Set build to running with a dead PID (not 0 — 0 is falsy in Python)
|
||||
_mark_running(vault, pid=99999, current=5, total=100)
|
||||
|
||||
# Stop
|
||||
result = _run_embed_stop(vault)
|
||||
assert result["ok"] is True, f"Stop returned error: {result}"
|
||||
assert result["data"]["state"] == "stopped", f"Expected stopped, got {result['data']['state']}"
|
||||
|
||||
# Verify build state settled to idle
|
||||
bs = _read_build_state(vault)
|
||||
assert bs["status"] == "idle", f"Expected idle, got {bs}"
|
||||
assert bs["pid"] == 0
|
||||
|
||||
def test_stop_when_idle_stays_idle(self, tmp_path: Path):
|
||||
"""embed stop --json on idle build returns {state: idle}."""
|
||||
vault = _build_vault(tmp_path)
|
||||
|
||||
result = _run_embed_stop(vault)
|
||||
assert result["ok"] is True
|
||||
assert result["data"]["state"] == "idle"
|
||||
|
||||
bs = _read_build_state(vault)
|
||||
assert bs["status"] == "idle"
|
||||
|
||||
def test_resume_after_stop_reads_progress(self, tmp_path: Path):
|
||||
"""After stop, status shows the idle state with progress, and resume mode continues from there."""
|
||||
vault = _build_vault(tmp_path)
|
||||
|
||||
_mark_running(vault, pid=99998, current=42, total=200)
|
||||
|
||||
# Stop
|
||||
_run_embed_stop(vault)
|
||||
|
||||
# Status should report progress and idle state
|
||||
status = _run_embed_status(vault)
|
||||
bs = status.get("build_state", {})
|
||||
assert bs.get("status") == "idle"
|
||||
assert bs.get("current", 0) == 42
|
||||
assert bs.get("total", 0) == 200
|
||||
|
||||
# Resume mode: verify that a subsequent --resume build picks up from idle
|
||||
# (we can't do a full build without a real vault, so we check that
|
||||
# the resume path isn't blocked by the idle state)
|
||||
from paperforge.embedding.build_state import read_vector_build_state
|
||||
|
||||
bs_read = read_vector_build_state(vault)
|
||||
assert bs_read["status"] == "idle"
|
||||
# A resume build call would check this and start fresh or continue
|
||||
# The state is correctly idle which allows resume
|
||||
Loading…
Reference in a new issue