Commit graph

107 commits

Author SHA1 Message Date
Research Assistant
766ed1edfa feat(08-deep-helper-deployment): add rollback to prepare_deep_reading
- Wrap figure-map, chart-type-map, and scaffold generation in try/except
- Track written files for cleanup on failure
- Save original formal note content before modification
- On exception: delete partial files and restore original note
- Add test_prepare_rollback.py with 3 tests:
  - figure-map failure triggers rollback
  - scaffold failure triggers rollback
  - success path leaves all files intact
- Fix zotero_key quote stripping in scan_deep_reading_queue
2026-04-24 14:37:13 +08:00
Research Assistant
802d34bff6 test(08-deep-helper-deployment): add smoke tests and fixtures
- Add conftest.py with test_vault fixture for deterministic test vault
- Add test_smoke.py with 14 regression tests covering:
  - setup wizard pip install step
  - doctor importability and env name checks
  - ld_deep.py import from deployed location
  - prepare produces scaffold with figure-map and chart-type-map
  - queue shows ready papers with correct ocr_status
  - doc commands extractable and executable
  - metadata fields (first_author, journal) populated
  - BBT PDF path resolution
- Fix zotero_key quote stripping in scan_deep_reading_queue
2026-04-24 14:34:10 +08:00
Research Assistant
afa19eb5b2 feat(08-deep-helper-deployment): create OCR-complete fixtures for TSTONE001
- Add synthetic 骨科.json export with TSTONE001 metadata
- Add generate_ocr_fixture.py script for deterministic fixture generation
- Generate fulltext.md, figure-map.json, chart-type-map.json, meta.json
- Fixtures represent a realistic biomechanical paper with 4 figures, 2 tables
2026-04-24 13:13:45 +08:00
Research Assistant
fa51cf70ac fix(08-deep-helper-deployment): fix ld_deep.py importability and doctor checks
- Upgrade Agent script check from directory existence to actual importability
- Fix doctor env var check: prefer PADDLEOCR_API_TOKEN over old names
- Support per-domain JSON export validation (not just library.json)
- Remove blanket tests/ exclusion from .gitignore
2026-04-24 12:55:57 +08:00
Research Assistant
79d8f3758d docs(08): add phase 8 implementation plan 2026-04-24 12:50:15 +08:00
Research Assistant
4d625a5156 docs(08): capture phase 8 context 2026-04-24 12:45:20 +08:00
Research Assistant
c7a73b047f chore: exclude tests/ from git, update planning docs for Phase 7 2026-04-24 01:22:17 +08:00
Research Assistant
bef53fc61f docs(agents): add paperforge repair command reference 2026-04-24 01:10:06 +08:00
Research Assistant
1f817a5519 Add repair subcommand to CLI
- Add 'repair' subcommand with --verbose and --fix flags
- Wire to run_repair() from literature_pipeline worker
- Follow existing dispatch pattern (like deep-reading)
2026-04-24 01:07:13 +08:00
Research Assistant
ff1251522f fix: initialize validated_status and validated_error before try block
Both variables were assigned inside the try block but used after it.
If the try block threw before assignment, validated_error would be unbound
when line 2909 referenced it with done_incomplete status.

Also initialized validated_status = None for consistency since it could
also be unset if the try block fails before assignment.
2026-04-24 00:59:01 +08:00
Research Assistant
d0ba1da715 fix(run_repair): case 2 repair updates all three + case 3 handles None; add test_fix_case4 2026-04-24 00:42:44 +08:00
Research Assistant
6052c823e6 Add run_repair() function for three-way OCR state divergence repair 2026-04-24 00:02:32 +08:00
Research Assistant
981f5fa8ba fix(run_deep_reading): call validate_ocr_meta() before using ocr_status
- Replace direct meta.get('ocr_status') read with validate_ocr_meta() call
  in run_deep_reading() to properly validate OCR completion status
- Fixes bug where meta.json with ocr_status=done but missing files
  incorrectly marked paper as ready for deep reading
- validate_ocr_meta() checks 7 conditions before returning 'done':
  file existence, sizes, page markers, etc.
- done_incomplete status is correctly treated as blocked (not ready)
  by existing blocked logic at line 2810
- Add regression test: test_smoke_deep_reading_done_incomplete_is_blocked
2026-04-23 23:42:09 +08:00
Research Assistant
187af3b379 fix(cli): lazy pipeline import with idempotent guard; remove paperforge_lite deployment from wizard
Root cause: paperforge.exe pointed to vault's PaperForge/paperforge_lite/ (deployed by
wizard), which lacked the 'pipeline/' package, causing ModuleNotFoundError.

Fix:
- cli.py: deferred sys.path resolution + idempotent worker import (each global
  is imported only once; tests can patch stubs and guards skip if already bound)
- wizard: remove paperforge_lite copytree deployment (pip install -e . handles it)
- .gitignore: add tests/sandbox/00_TestVault/
2026-04-23 21:31:51 +08:00
Research Assistant
1d8478ce9c fix(wizard): auto-run pip install -e . and improve UX flow
- setup_wizard.py: _deploy() now calls pip install -e <repo_root> automatically
- DoneStep: removes manual 'pip install -e .' instruction (wizard handles it)
- README.md: removes separate pip install -r requirements.txt step
- docs/INSTALLATION.md: same pip install simplification
- pyproject.toml: add pytest ignore for nested sandbox vault
- tests/sandbox/: remove old vault/ (replaced by 00_TestVault/)
- tests/sandbox/README.md: update flow, wizard handles pip install
2026-04-23 20:49:10 +08:00
Research Assistant
31cce820e2 docs(sandbox): add pip install step to README 2026-04-23 20:34:13 +08:00
Research Assistant
e3151a6368 feat(tests/sandbox): add generate_sandbox.py with 5 mock papers across 2 domains 2026-04-23 20:27:26 +08:00
Research Assistant
dda7886d41 feat(tests): add sealed sandbox vault for paperforge testing 2026-04-23 20:18:27 +08:00
Research Assistant
b9f06ccad3 feat: complete PaperForge Lite v1.0 release hardening (5 phases, 28 reqs, 145 tests) 2026-04-23 20:15:20 +08:00
Research Assistant
08ddebbc0b docs(state): mark Phase 5 complete 2026-04-23 20:11:20 +08:00
Research Assistant
e79ff8cbc6 docs(05-01): complete plan 05-01 with OCR state machine tests and command docs verification 2026-04-23 20:10:03 +08:00
Research Assistant
935d948ede feat(05-01): add OCR state machine tests covering job transitions
- Test pending->queued transition on job submission
- Test processing->done transition when polling returns success
- Test processing->error transition on API error response
- Test processing->blocked transition via HTTPError 401 (classify_error maps to blocked)
- Test sync_ocr_queue skips done/blocked items from existing queue
- Test cleanup_blocked_ocr_dirs removes empty dirs, preserves dirs with payload
- Test all 7 OCR states (pending, queued, running, done, error, blocked, nopdf) don't crash

Key: patch requests.post with side_effect=HTTPError 401 to trigger
classify_error path, since registry token is always present in test env.
2026-04-23 19:37:15 +08:00
Research Assistant
b6df4ae510 docs(05-02): complete 05-02 plan metadata
- 05-02-SUMMARY.md created with 13 smoke tests documented
- STATE.md: Phase 5 progress table updated with 05-01/05-02 done
- ROADMAP.md: Phase 5 plans checked off, phase status updated to COMPLETE
- REQUIREMENTS.md: REL-02 marked complete (05-02)
2026-04-23 18:48:45 +08:00
Research Assistant
405fc3f127 test(05-02): add end-to-end smoke test suite covering full pipeline
Smoke tests (13 total):
- Setup validation (run_doctor returns 0/1, output checks)
- Selection sync (returns 0, scans exports, creates records)
- Index refresh (returns 0, generates formal Literature notes)
- OCR doctor L1-L3 (returns int code, detects missing token)
- Deep-reading queue (returns 0, produces 3-state output)
- CLI main entry (doctor/status return int, not sys.exit)

All tests use tmp_path fixtures, no real vault, no live network calls.
2026-04-23 18:46:28 +08:00
Research Assistant
1704395d34 test(05-02): add fixture vault factory for smoke testing
- fixture_vault: creates complete vault structure with paperforge.json
- fixture_library_records: 3 minimal library-record .md files
- fixture_bbt_json: Better BibTeX JSON export in exports/
- fixture_with_pdf: vault with real temp PDF for PDF-resolver tests
- All use pathlib.Path, Windows-compatible
2026-04-23 18:46:13 +08:00
Research Assistant
dc11e10c37 docs(state): record phase 5 context session 2026-04-23 18:26:49 +08:00
Research Assistant
f8b523baf9 docs(05): capture phase context 2026-04-23 18:26:18 +08:00
Research Assistant
6bbdb3d1d1 docs(02-03): complete OCR Doctor plan with summary, state, and roadmap updates
- Add 02-03-SUMMARY.md with execution summary and deviations
- Update STATE.md: mark 02-03 done, add decisions, update next action
- Update ROADMAP.md: mark 02-03 and 01-04 complete
2026-04-23 13:02:18 +08:00
Research Assistant
2a3295ed38 feat(02-03): add ocr doctor CLI dispatch, tests, and docs
- Convert ocr parser to sub-subcommands (run, doctor)
- Add _cmd_ocr_doctor() with formatted tiered report output
- Update dispatch logic to route ocr doctor subcommand
- Add CLI dispatch test for doctor command
- Update command/lp-ocr.md with doctor documentation
2026-04-23 12:59:04 +08:00
Research Assistant
3bd1c5ea69 feat(02-03): implement ocr_doctor() with L1-L4 diagnostics
- Add paperforge_lite/ocr_diagnostics.py with tiered L1-L4 checks
- L1: token presence, L2: URL reachability, L3: API schema validation
- L4: optional live PDF round-trip test with polling
- Add 7 mocked unit tests for all levels
- Include blank.pdf test fixture
2026-04-23 12:58:54 +08:00
Research Assistant
24e3d9fb30 test(02-01): add failing tests for pdf_resolver path resolution
- 8 test cases covering absolute, vault-relative, junction, storage-relative
- Missing file, has_pdf=False, empty path, whitespace path scenarios
- is_valid_pdf and resolve_junction helper coverage
2026-04-23 12:54:38 +08:00
Research Assistant
97a8fd6772 docs(01-03): complete plan 01-03 2026-04-23 12:07:00 +08:00
Research Assistant
500d268db7 feat(01-03): wire validate_setup.py to shared resolver
- load_config now tries paperforge_lite.config.load_vault_config first,
  falls back to legacy JSON parsing for pre-01-03 installs
- resolve_vault_for_validate checks PAPERFORGE_VAULT before VAULT_PATH
  before cwd (matching the shared resolver's precedence)
- Preserves all existing validation checks
2026-04-23 12:03:08 +08:00
Research Assistant
e0acb64e8d feat(01-03): deploy paperforge_lite package through setup wizard
- setup_wizard.py now copies the paperforge_lite/ package alongside
  the worker and ld_deep scripts during deployment
- Two copy targets: <pf_path>/worker/paperforge_lite/ for
  literature_pipeline.py and <skill_dir>/literature-qa/paperforge_lite/
  for ld_deep.py
- Existing copy targets for literature_pipeline.py and ld_deep.py
  remain unchanged
- Warning emitted if paperforge_lite source not found
2026-04-23 12:01:36 +08:00
Research Assistant
1c1e6d7c0d feat(01-03): replace duplicated resolver logic with shared wrappers
- literature_pipeline: load_vault_config and pipeline_paths now
  delegate to paperforge_lite.config, preserving public names
- ld_deep: _load_vault_config and _paperforge_paths now delegate
  to shared resolver, preserving public names
- Both scripts require paperforge_lite package to be available
  (via pip install or copied install in deployed vaults)
- Existing load_simple_env preserved for .env loading before dispatch
2026-04-23 12:00:24 +08:00
Research Assistant
fec25c2d3e feat(01-03): wire worker and /LD-deep to shared resolver
- Replace literature_pipeline.load_vault_config body with
  paperforge_lite.config.load_vault_config wrapper
- Rebuild pipeline_paths() from shared paperforge_paths() plus
  10 worker-only keys (pipeline, candidates, search_*, harvest_root,
  records, review, config, queue, log, bridge_config*, index, ocr_queue)
- Replace ld_deep._load_vault_config and _paperforge_paths with
  shared resolver wrappers returning ocr, records, literature
- Fix subprocess test to set PYTHONPATH so paperforge_lite is
  importable when worker runs as a standalone script
- 34 tests pass (22 config + 8 worker compat + 4 ld_deep compat)
2026-04-23 12:00:11 +08:00
Research Assistant
618227eb9b test(01-03): add failing tests for worker and /LD-deep resolver integration
- Test literature_pipeline.load_vault_config matches paperforge_lite.config
- Test PAPERFORGE_SYSTEM_DIR env var override in worker
- Test pipeline_paths returns all 17 expected keys (shared + worker-only)
- Test ld_deep._load_vault_config and _paperforge_paths match shared resolver
- Test direct worker status subprocess smoke test (CMD-02)
- Tests fail against current code: worker has 5 config keys vs shared's 7,
  env overrides not implemented, and pipeline_paths lacks shared resolver keys
2026-04-23 11:55:24 +08:00
Research Assistant
767548d2fe docs(01-02): complete 01-02 plan
- paperforge CLI launcher with argparse subcommands
- pyproject.toml packaging entry point
- load_simple_env added to config.py
- 9 CLI tests (3 path + 6 dispatch) passing
- 31 total tests passing
- CONF-02, CMD-01, CMD-03 requirements completed
2026-04-23 11:48:26 +08:00
Research Assistant
9496382e81 feat(01-02): implement CLI launcher, packaging entry point, and shared config resolver
- paperforge_lite/cli.py: argparse CLI with global --vault, subcommands paths/status/selection-sync/index-refresh/deep-reading/ocr (ocr defaults to run action)
- paperforge_lite/__main__.py: python -m paperforge_lite fallback entry point
- paperforge_lite/config.py: shared resolver (load_vault_config, resolve_vault, paperforge_paths, paths_as_strings, load_simple_env) matching Plan 01 contract
- pyproject.toml: setuptools build, paperforge-lite package, Python >=3.10, dependencies from requirements.txt, [project.scripts] paperforge=paperforge_lite.cli:main
- paperforge_lite/__init__.py: package init (version 1.0.0)

CLI loads .env from vault root and <system_dir>/PaperForge/.env before worker dispatch, matching legacy pipeline behavior.
2026-04-23 11:44:26 +08:00
Research Assistant
216ea4e8ac test(01-02): add failing test for CLI paths output and worker dispatch
- Tests main() with --vault vault paths --json emits valid JSON with vault, worker_script, ld_deep_script keys
- Tests main() with --vault vault paths emits text paths without unresolved <system_dir> or <resources_dir> tokens
- Tests ocr and ocr run both dispatch to run_ocr
- Tests status/selection-sync/index-refresh/deep-reading each dispatch to correct worker
- Uses monkeypatch to isolate from real workers and filesystem
2026-04-23 11:44:09 +08:00
Research Assistant
6dc62ac9f1 docs(01-01): complete 01-01 plan
- SUMMARY: 22 passing tests, 3 files created, 2 task commits (test + feat)
- STATE: phase 1 progress updated, decisions logged
- ROADMAP: 01-01 marked complete
2026-04-23 11:43:37 +08:00
Research Assistant
034542bcd2 docs(01-04): complete plan 04 - stable command documentation
- SUMMARY.md: phase 1 plan 4 complete, all tasks committed
- STATE.md: updated with Phase 1 progress (4/4 plans complete)
- ROADMAP.md: marked all 4 Phase 1 plans as done

Commits:
- baa2d77: test(01-04): add failing tests for stable command docs
- 51d35d6: feat(01-04): replace unresolved path-token commands with stable paperforge commands
2026-04-23 11:42:23 +08:00
Research Assistant
a6c0cb0e70 feat(01-01): implement shared config resolver
- DEFAULT_CONFIG: 99_System, 03_Resources, Literature, LiteratureControl, 05_Bases, .opencode/skills, .opencode/command
- ENV_KEYS: all 8 PAPERFORGE_* overrides mapped
- read_paperforge_json: supports both nested vault_config and top-level legacy keys
- resolve_vault: cli > PAPERFORGE_VAULT > json-search > cwd precedence
- load_vault_config: layered merge with locked D-Configuration Hierarchy order
- paperforge_paths: returns exactly 13 required user-facing path keys
- paths_as_strings: JSON-serializable string output
- No os.environ mutation, no OCR secrets, stdlib only
2026-04-23 11:41:10 +08:00
Research Assistant
51d35d6999 feat(01-04): replace unresolved path-token commands with stable paperforge commands
- command/lp-*.md: replace python <system_dir>/... with paperforge status|selection-sync|index-refresh|ocr run; add fallback legacy command section
- command/ld-deep.md: update queue preflight to 'paperforge deep-reading'; add paperforge paths --json for variable resolution
- README.md: update core commands section to show paperforge first, legacy as fallback
- docs/INSTALLATION.md: add paperforge paths/status as primary verification
- docs/setup-guide.md: replace python pipeline/worker/... with paperforge selection-sync/index-refresh
- setup_wizard.py DoneStep: add pip install -e . and paperforge commands as primary next steps

All 15 tests in tests/test_command_docs.py pass.
2026-04-23 11:40:24 +08:00
Research Assistant
baa2d77bbd test(01-04): add failing tests for stable command docs
- Verify lp-* docs contain paperforge status|selection-sync|index-refresh|ocr run
- Verify ld-deep.md mentions paperforge deep-reading and paths --json
- Verify no legacy python <system_dir>/.../literature_pipeline.py in user examples
- Scope exclusion: AGENTS.md frontmatter examples, architecture diagrams

RED phase: all 15 assertions fail before implementation.
2026-04-23 11:37:40 +08:00
Research Assistant
6b0b16b1dd test(01-01): add failing test for config resolver contract
- Tests DEFAULT_CONFIG defaults (99_System, 03_Resources, Literature, etc.)
- Tests ENV_KEYS has all required PAPERFORGE_* overrides
- Tests load_vault_config precedence: overrides > env > JSON > defaults
- Tests nested vault_config honored per CONF-04
- Tests top-level keys override nested for backward compat per CONF-04
- Tests paperforge_paths returns exactly required user-facing keys
- Tests paths are absolute Path objects
- Tests worker_script and ld_deep_script keys present
- Tests paths_as_strings returns JSON-serializable dict[str,str]
- Tests resolve_vault precedence: cli > env > json search > cwd

RED phase: module not yet implemented, all tests fail as expected.
2026-04-23 11:35:52 +08:00
Research Assistant
8097ec6ff3 docs(01): create phase plan 2026-04-23 11:18:05 +08:00
Research Assistant
881457dd92 Fix configurable Lite setup paths 2026-04-23 01:44:13 +08:00
Research Assistant
e0881f7d2c restore ASCII logo in WelcomeStep 2026-04-23 01:17:46 +08:00
Research Assistant
6fcd1132fa feat: read system_dir/resources_dir from paperforge.json instead of hardcoding 99_System/03_Resources 2026-04-23 01:07:50 +08:00