Commit graph

1845 commits

Author SHA1 Message Date
LLLin000
85a7ae30a2 feat: add OCR rebuild UX and capability tracer 2026-07-15 20:36:15 +08:00
LLLin000
e56a5faf44 docs: record control-center PRD and setup migration 2026-07-15 02:30:05 +08:00
LLLin000
906b3cafc0 fixup: headless via SetupPlan only, ConfigWriter always schema_version 2
- Removed headless_setup call from CLI dispatch entirely. --headless,
  --modular, and bare all construct the same SetupPlan with identical
  _cfg/zotero_path/agent_type/skip_checks. Only the deprecation notice
  differs per entry point.
- Added skip_checks parameter to SetupPlan (forwarded to CLI arg).
- ConfigWriter always writes schema_version '2' unconditionally (never
  preserves a v1 value from existing config).

RED commands:
  python -m pytest tests/test_setup_plan_v2.py::TestConfigWriterSchemaVersion -v
  python -m pytest tests/test_setup_plan_v2.py::TestCliHeadlessViaSetupPlan -v

GREEN commands (same, after fixes): 61/61 passed.
2026-07-15 02:23:46 +08:00
LLLin000
7b7474237a fixup: address #75 review — headless canonical, zotero forwarding, visible deprecation, all legacy path keys
- --headless now delegates to SetupPlan for canonical config, then runs
  headless_setup for full deployment; canonical ConfigWriter pass ensures
  v2 vault_config format.
- --zotero-data is forwarded through SetupPlan.zotero_path for both
  --modular and bare paths.
- Deprecation warnings use print(stderr) instead of filtered
  DeprecationWarning so users always see them.
- ConfigWriter.PATH_KEYS extended to converge all seven legacy path keys
  (system_dir, resources_dir, literature_dir, control_dir, base_dir,
   skill_dir, command_dir) into vault_config, preserving non-path metadata.
- Added CLI-level tests: deprecation on stderr, config canonicalization,
  --literature-dir forwarding, --zotero-data plumbing.
2026-07-15 02:17:04 +08:00
LLLin000
af84969972 feat: canonicalize setup and configuration migration (closes #75)
ConfigWriter now writes v2 canonical format (nested vault_config with
schema_version). On rerun it merges with existing config rather than
overwriting, making repeated runs idempotent.

load_vault_config reverses the v1 precedence: vault_config block now
wins over legacy top-level path keys, which are treated as a read-only
fallback with a UserWarning. This is the N+1 transition — old v1
configs remain readable with a warning; all writes produce v2 only.

Bare and --headless setup now delegate to the canonical SetupPlan engine
(the --modular path) with a DeprecationWarning.

Closes #75.
2026-07-15 02:07:13 +08:00
LLLin000
b97e82ab7c docs: close #71/#72 prototypes, set #73 as active frontier
Record the completed six-module control-center and maintenance-inbox
prototype wave. Both prototype pairs passed independent Critical/Important
PASS review and direct browser verification at 768px without production
code changes. Update executive summary, active queue, decision log,
session timeline, and milestone checkpoint.

- #71 resolved: six-module control-center HTML prototype
  (5 scenarios x 6 modules, plain-button switcher, responsive 768px)
- #72 resolved: actionable-only maintenance inbox prototype
  (single-action rows, inline issue-draft, confirmation-first report)
- #73 promoted to active frontier: integrate accepted model into
  production control-center plugin
2026-07-15 00:41:28 +08:00
LLLin000
c68b671e0c fixup: align fix row 33 commit hash with final commit 2026-07-15 00:03:26 +08:00
LLLin000
d7b0a52780 closeout: record #69/#70 resolution, capability model, runtime architecture, maintenance regression fixes
- Update PROJECT-MANAGEMENT.md for #69/#70 resolved, #71/#72 as live frontier
- Add orthogonal capability/activity/attention model and managed-runtime immutable-slot
  architecture decisions to decision log and timeline
- Record OCR maintenance regression: canonical per-row action routing (display_action->verb),
  redo confirmation gate, cache manifest preservation (19/19 tests pass)
- Add research reports: docs/research/2026-07-14-capability-state-action-contract.md,
  docs/research/2026-07-14-managed-runtime-architecture.md
- Update project/current/ocr-v2-active-queue.md with resolved items and new frontier
2026-07-15 00:03:16 +08:00
LLLin000
29a0e27077 docs: research control center recovery patterns 2026-07-14 23:08:26 +08:00
LLLin000
68f342ddf9 docs: audit control center contracts 2026-07-14 19:58:30 +08:00
LLLin000
92fe2b7847 docs: chart control center UX wayfinder 2026-07-14 19:04:20 +08:00
LLLin000
b7e80c84d6 docs: record OCR rebuild UX completion 2026-07-14 18:33:37 +08:00
LLLin000
e556c8bab0 fix: complete final OCR batch item normally 2026-07-14 18:32:48 +08:00
LLLin000
3a516add86 fix: align OCR maintenance selection semantics 2026-07-14 18:27:53 +08:00
LLLin000
349d4f6dcb feat: add streaming OCR rebuild progress 2026-07-14 18:05:49 +08:00
LLLin000
3a0d171e31 docs: update PROJECT-MANAGEMENT and ocr-v2-active-queue with retrieval recovery Wayfinder checkpoint 2026-07-11 01:30:58 +08:00
LLLin000
78c6d7af65 docs: add Source Corpus / Retrieval Experience / Retrieval Artifact glossary terms 2026-07-11 01:30:39 +08:00
LLLin000
b4ac277438 Merge branch 'feat/retrieval-recovery'
# Conflicts:
#	paperforge/commands/embed.py
2026-07-11 01:26:37 +08:00
LLLin000
f158c9ff80 fix: force-rebuild uses ensure_vec_tables for correct embedding dimension 2026-07-11 00:52:30 +08:00
LLLin000
460412d423 fix: isVectorReady checks total_chunks instead of fulltext-only chunk_count 2026-07-11 00:39:25 +08:00
LLLin000
537e8e36ed fix: add --vault flag to search/retrieve spawn 2026-07-11 00:31:42 +08:00
LLLin000
2dab3d11ad fix: health probe uses detect_embedding_dim instead of hardcoded 1536 2026-07-11 00:28:49 +08:00
LLLin000
1eb4d80f8d fix: create vec0 companion meta tables unconditionally in ensure_schema 2026-07-11 00:16:38 +08:00
LLLin000
3cb3fd75a4 fix: force rebuild drops tables before ensure_schema recreates them 2026-07-11 00:15:09 +08:00
LLLin000
9f8f73ba17 fix: remove shadowing local imports of get_memory_db_path and get_connection 2026-07-11 00:13:46 +08:00
LLLin000
0fd5227355 chore: fix ruff violations — unused imports, Path, dead code, formatting 2026-07-11 00:10:32 +08:00
LLLin000
4c5d981fff fix: delete old vectors before writing new ones in build loop
The previous ordering wrote encoded payloads first, then deleted ALL
vectors for that paper (including the just-written ones). Swap the
ordering so old vectors are deleted first and new ones survive.
2026-07-10 23:57:34 +08:00
LLLin000
11eb2946d9 fix: --force drops vec0 tables instead of ChromaDB directory
Replace shutil.rmtree on the ChromaDB path with DROP TABLE IF EXISTS
on vec0 virtual tables and their companion meta tables. Force rebuild
now opens paperforge.db and drops vec_fulltext, vec_body, vec_objects,
vec_fulltext_meta, vec_body_meta, and vec_objects_meta.

Also replaced the remaining get_vector_db_path reference in the error
handler with get_memory_db_path. Removed get_vector_db_path import
from embed.py since it is no longer used in active control flow.
2026-07-10 23:57:34 +08:00
LLLin000
59581c72bf fix: --resume checks vec0 meta tables, not ChromaDB path
Replace get_vector_db_path() directory existence check with vec0
meta table row counts via SQLite. Resume now checks whether any
rows exist in vec_fulltext_meta, vec_body_meta, or vec_objects_meta.
If none exist, starts a fresh build. This removes the last ChromaDB
path dependency from the resume control flow.
2026-07-10 23:57:34 +08:00
LLLin000
e7a030d057 refactor: remove ChromaDB dependency gating from status and preflight
Embed status no longer checks for chromadb import — vec0 runs in
paperforge.db and doesn't need the ChromaDB library. The preflight
check also no longer requires chromadb since vec0 backends do not
depend on it. The chroma_backend module remains for legacy test
compatibility but no longer gates any production build/status path.
2026-07-10 23:57:33 +08:00
LLLin000
51018943ad feat: implement 8 search domain states with keyboard shortcuts and a11y in dashboard
Adds _renderSearchState() rendering 8 distinguishable states:
idle, searching (M/@), results, empty, vectors-not-built,
backend-unavailable, timeout, model-changed.

Keyboard: / focuses search, Escape cancels, ArrowUp/Down navigates
results, Enter opens note, Ctrl+Enter forces deep mode.

Accessibility: role=alert on errors, aria-live=polite on results
and progress, role=listbox with aria-selected/aria-posinset on cards,
aria-setsize, tabindex=0 on result cards, focus management on
state transitions.

Focus management: Stop button on build start, first result on
search complete, error banner on failure.
2026-07-10 23:56:31 +08:00
LLLin000
6e4b446ca2 feat: add retrieval error code routing in classifyError
Maps 7 retrieval error codes (VECTOR_NOT_BUILT, VECTOR_CORRUPTED,
MODEL_CHANGED, BACKEND_UNAVAILABLE, TIMEOUT, INTERNAL_ERROR, NO_PYTHON)
to distinct UI states with type, message, recoverable flag, and action.

Adds 7 new test cases for the error classification routing.
2026-07-10 23:56:25 +08:00
LLLin000
6b53de8135 feat: add 33 retrieval_* i18n keys for error distinction and recovery UI
Adds all retrieval state labels, button text, warning messages, and
descriptive copy in English and Chinese under the retrieval_* prefix.

Prepares i18n foundation for search domain (8 states) and build domain
(10 states) per PRD #57 and issue #62.
2026-07-10 23:56:20 +08:00
LLLin000
ec810b10a9 feat: plugin reads embed status --json live, polls during build, stop without kill
getVectorRuntime now calls embed status --json synchronously with
a 2s time-based cache before falling back to the JSON snapshot.

During build, a 2s polling interval reads embed status --json for
live build state and detects remote stop.

Stop button calls embed stop --json only (no direct .kill() on the
child process), letting the CLI handle cooperative stop.

ISettingPlugin interface extended with _embedPollInterval and
_embedPolling properties.
2026-07-10 23:55:26 +08:00
LLLin000
0bc054e7b0 fix: cooperative stop with state settlement and build-loop cancellation
embed stop now sets status=stopping, waits for the build PID to exit
(8s timeout), force-kills as last resort, then settles to idle.

Build loop checks the cancellation flag (read_vector_build_state)
between papers in the main loop and drains in-flight work before
cleanly exiting without marking 'completed'.

Fixes pre-existing UnboundLocalError from a local import shadowing
the global mark_vector_build_state symbol in the run() function.
2026-07-10 23:55:21 +08:00
LLLin000
40326d6759 fix: add vec0 k-NN health probe to embed status
embed status now runs a zero-vector k-NN query on each vec0 table
that has companion meta rows. If the probe fails (table dropped,
extension unavailable, index corrupted), healthy=false and
corrupted=true are reported even when meta row counts > 0.

Includes integration test that drops vec_fulltext and verifies
healthy=false.
2026-07-10 23:55:16 +08:00
LLLin000
aa9f1de278 fix: add missing CREATE_VEC_FULLTEXT_META to vec0 schema
The vec_fulltext_meta table DDL was referenced by ensure_schema()
but never defined. ADD the missing CREATE_VEC_FULLTEXT_META variable
so the meta table is created alongside vec_body_meta and vec_objects_meta.
2026-07-10 23:55:11 +08:00
LLLin000
66c2db427e fix: add missing vec_fulltext_meta DDL and embed lifecycle tests
schema.py: CREATE_VEC_FULLTEXT_META was referenced at line 328 but
never defined. Adds the missing DDL constant so the v5 schema migration
does not raise NameError on fresh databases.

tests: add embedding test suite with four tests covering vec0 resume
detection, empty-table fallback, force drop, and delete-then-write
ordering verification.
2026-07-10 23:39:10 +08:00
LLLin000
733640413a fix: --force drops vec0 tables instead of ChromaDB directory
Replace shutil.rmtree on the ChromaDB path with DROP TABLE IF EXISTS
on vec0 virtual tables and their companion meta tables. Force rebuild
now opens paperforge.db and drops vec_fulltext, vec_body, vec_objects,
vec_fulltext_meta, vec_body_meta, and vec_objects_meta.

Also replaced the remaining get_vector_db_path reference in the error
handler with get_memory_db_path. Removed get_vector_db_path import
from embed.py since it is no longer used in active control flow.
2026-07-10 23:36:55 +08:00
LLLin000
9095a42a13 fix: --resume checks vec0 meta tables, not ChromaDB path
Replace get_vector_db_path() directory existence check with vec0
meta table row counts via SQLite. Resume now checks whether any
rows exist in vec_fulltext_meta, vec_body_meta, or vec_objects_meta.
If none exist, starts a fresh build. This removes the last ChromaDB
path dependency from the resume control flow.
2026-07-10 23:36:14 +08:00
LLLin000
85b7205827 refactor: remove ChromaDB dependency gating from status and preflight
Embed status no longer checks for chromadb import — vec0 runs in
paperforge.db and doesn't need the ChromaDB library. The preflight
check also no longer requires chromadb since vec0 backends do not
depend on it. The chroma_backend module remains for legacy test
compatibility but no longer gates any production build/status path.
2026-07-10 23:35:26 +08:00
LLLin000
8e6b2dfc2c fix: delete old vectors before writing new ones in build loop
The previous ordering wrote encoded payloads first, then deleted ALL
vectors for that paper (including the just-written ones). Swap the
ordering so old vectors are deleted first and new ones survive.
2026-07-10 23:34:50 +08:00
LLLin000
fc1dca429c Unify PFResult envelope: data.matches with consistent field names, --deep flag
- search.py: normalize FTS results to unified field set (score from rank,
  add text/heading/source keys). Result key unchanged (already 'matches').
- retrieve.py: rename data.chunks to data.matches in standard and deep paths.
  Map field names: paper_id->zotero_key, chunk_text->text, section_path->heading.
  Expand enrichment SQL to fetch journal and domain.
  --deep flag already wired in CLI parser.
- tests/cli/test_json_contracts.py: add contract tests verifying PFResult
  envelope shape, data.matches key presence, unified field names,
  --deep flag acceptance, and field type correctness.
2026-07-10 23:15:16 +08:00
LLLin000
984dfc6114 Unify PFResult envelope: data.matches with consistent field names, --deep flag
- search.py: normalize FTS results to unified field set (score from rank,
  add text/heading/source keys). Result key unchanged (already 'matches').
- retrieve.py: rename data.chunks to data.matches in standard and deep paths.
  Map field names: paper_id->zotero_key, chunk_text->text, section_path->heading.
  Expand enrichment SQL to fetch journal and domain.
  --deep flag already wired in CLI parser.
- tests/cli/test_json_contracts.py: add contract tests verifying PFResult
  envelope shape, data.matches key presence, unified field names,
  --deep flag acceptance, and field type correctness.
2026-07-10 23:10:36 +08:00
LLLin000
32e69a6c7b Delete sql.js: remove db.ts, wasm, dependency, and all sql.js code paths
- Remove paperforge/plugin/src/services/db.ts (sql.js wrapper, initDatabase/searchMetadata/SearchResultItem)
- Remove paperforge/plugin/sql-wasm.wasm (644KB binary)
- Remove sql.js and @types/sql.js from package.json; npm install to update lockfile
- dashboard.ts: remove import of db.ts, remove _sqlJsInitialized/_sqlJsFailed state,
  remove the sql.js search path block entirely — all searches go straight to CLI spawn.
  Clean up stale comments.
- Bundle drops from 346KB to 167KB; 6 test files / 58 tests still pass
2026-07-10 23:07:26 +08:00
LLLin000
f52124bcf7 docs: retrieval recovery design — Approach A (Python CLI sole owner) recommended 2026-07-10 23:01:57 +08:00
LLLin000
e3c41cc07d chore: add sql.js WASM, DB service, and PRD document 2026-07-10 02:49:49 +08:00
LLLin000
bf29966a86 feat: real-time metadata search via sql.js with 200ms debounce
Replace CLI spawn for metadata search with direct sql.js FTS5 queries.
Input debounced at 200ms. Falls back to CLI when sql.js unavailable.
@ Deep Search stays on CLI.
2026-07-10 02:49:31 +08:00
LLLin000
d6c1d1539b chore: rebuild plugin main.js with sql.js dependency
Bundles sql.js for direct FTS5 access to paperforge.db.
2026-07-10 02:49:06 +08:00
LLLin000
65311fc1c5 fix(status): handle read-only DB during concurrent build
get_embed_status opens a read-only connection that cannot load the
vec extension when another process has the DB locked. ensure_vec_extension
failure is now non-fatal — the status query proceeds with vec0 meta
table reads, which work in read-only mode.
2026-07-10 02:48:51 +08:00