Commit graph

116 commits

Author SHA1 Message Date
Research Assistant
45de80da5d refactor(phase-9): update all Python imports to paperforge
- Updated 35 files: from paperforge_lite.* → from paperforge.*
- Verified: pytest passes (172 passed, 2 skipped)
2026-04-24 16:15:30 +08:00
Research Assistant
59457d15c7 refactor(phase-9): rename paperforge_lite package to paperforge
- Rename directory: paperforge_lite/ → paperforge/
- Update pyproject.toml: package name, entry point, includes
- Verified: python -m paperforge --help works
2026-04-24 16:15:13 +08:00
Research Assistant
fb0e7a9469 plan(phase-9): command unification and CLI simplification
- Add 09-CONTEXT.md with architecture decisions
- Add 09-DISCUSSION-LOG.md documenting 4 gray area decisions:
  - Aggressive migration (no aliases)
  - Default full sync with --dry-run
  - Merge ocr run + doctor into single ocr command
  - Unified command modules (paperforge/commands/)
- Add 09-PLAN.md with 7 tasks (0-6):
  - Task 0: Rename package (paperforge_lite -> paperforge)
  - Task 1: Create shared command modules
  - Task 2: Refactor CLI to use modules
  - Task 3: Create /pf-* agent command docs
  - Task 4: Remove old /LD-* and /lp-* docs
  - Task 5: Update AGENTS.md and tests
  - Task 6: Verification and cleanup
- Update STATE.md with Phase 9 decisions
2026-04-24 16:03:48 +08:00
Research Assistant
00f361109c docs(v1.2): add requirements, roadmap, and milestone planning artifacts
- Add REQUIREMENTS-v1.2.md with 7 systematization requirements
- Add ROADMAP-v1.2.md with Phase 9-10 plan
- Update master ROADMAP.md with v1.2 phases
- Update MILESTONES.md marking v1.2 In Progress
- Architecture research from get-shit-done-main incorporated
2026-04-24 15:33:08 +08:00
Research Assistant
e8f666adb3 docs(milestone): initiate v1.2 systematization - update PROJECT, STATE, MILESTONES 2026-04-24 15:22:46 +08:00
Research Assistant
2f04380b9b docs(milestone): complete v1.1 milestone archive
- Archive milestone to .planning/milestones/v1.1.md with full stats:
  - 110+ commits, 24 requirements (21 Complete, 3 Partial)
  - Phase 6, 7, 8 accomplishments and test coverage
  - Known limitations and v1.2 candidate roadmap
- Evolve PROJECT.md:
  - Move sandbox audit items from Context to completed checklist
  - Mark 'Use sandbox audit as v1.1 release gate' decision as Complete
  - Add remaining BBT bare path normalization to Active
  - Update Out of Scope with v1.1 deferrals
- Reorganize ROADMAP.md:
  - Mark v1.1 phases as Done
  - Add v1.2 candidate table
  - Include all-time phase summary

Milestone v1.1 Sandbox Onboarding Hardening is complete.
2026-04-24 15:09:16 +08:00
Research Assistant
b863bf0fa1 docs(phase-7): add missing SUMMARY.md and update requirements traceability
- Create 07-SUMMARY.md documenting actual accomplishments:
  - Tasks 2-5 completed (OCR meta validation, run_repair, CLI, docs)
  - Task 1 (BBT PDF normalization) honestly marked as deferred
  - Requirements mapping: META-01/02 and STATE-01/02/03/04 = Complete
  - ZPATH-01/02/03 = Partial (bare path normalization not implemented)
- Update REQUIREMENTS.md traceability table:
  - Phase 6: 10/10 Complete
  - Phase 7: 6/9 Complete, 3 Partial
  - Phase 8: 6/6 Complete
- Update STATE.md to include Phase 7 in milestone summary table

Addresses milestone completion prerequisite: Phase 7 documentation gap.
2026-04-24 15:02:57 +08:00
Research Assistant
3b929b1d12 docs(phase-8): verify, update roadmap/state/project after phase completion 2026-04-24 14:44:53 +08:00
Research Assistant
b1cfe4f5cd docs(08-deep-helper-deployment): complete plan summary and state updates
- Add 08-SUMMARY.md with full execution record
- Update STATE.md: Phase 8 complete, 3 new decisions logged
- Update ROADMAP.md: Phase 8 marked Done
- 17/17 tests passing
2026-04-24 14:39:34 +08:00
Research Assistant
766ed1edfa feat(08-deep-helper-deployment): add rollback to prepare_deep_reading
- Wrap figure-map, chart-type-map, and scaffold generation in try/except
- Track written files for cleanup on failure
- Save original formal note content before modification
- On exception: delete partial files and restore original note
- Add test_prepare_rollback.py with 3 tests:
  - figure-map failure triggers rollback
  - scaffold failure triggers rollback
  - success path leaves all files intact
- Fix zotero_key quote stripping in scan_deep_reading_queue
2026-04-24 14:37:13 +08:00
Research Assistant
802d34bff6 test(08-deep-helper-deployment): add smoke tests and fixtures
- Add conftest.py with test_vault fixture for deterministic test vault
- Add test_smoke.py with 14 regression tests covering:
  - setup wizard pip install step
  - doctor importability and env name checks
  - ld_deep.py import from deployed location
  - prepare produces scaffold with figure-map and chart-type-map
  - queue shows ready papers with correct ocr_status
  - doc commands extractable and executable
  - metadata fields (first_author, journal) populated
  - BBT PDF path resolution
- Fix zotero_key quote stripping in scan_deep_reading_queue
2026-04-24 14:34:10 +08:00
Research Assistant
afa19eb5b2 feat(08-deep-helper-deployment): create OCR-complete fixtures for TSTONE001
- Add synthetic 骨科.json export with TSTONE001 metadata
- Add generate_ocr_fixture.py script for deterministic fixture generation
- Generate fulltext.md, figure-map.json, chart-type-map.json, meta.json
- Fixtures represent a realistic biomechanical paper with 4 figures, 2 tables
2026-04-24 13:13:45 +08:00
Research Assistant
fa51cf70ac fix(08-deep-helper-deployment): fix ld_deep.py importability and doctor checks
- Upgrade Agent script check from directory existence to actual importability
- Fix doctor env var check: prefer PADDLEOCR_API_TOKEN over old names
- Support per-domain JSON export validation (not just library.json)
- Remove blanket tests/ exclusion from .gitignore
2026-04-24 12:55:57 +08:00
Research Assistant
79d8f3758d docs(08): add phase 8 implementation plan 2026-04-24 12:50:15 +08:00
Research Assistant
4d625a5156 docs(08): capture phase 8 context 2026-04-24 12:45:20 +08:00
Research Assistant
c7a73b047f chore: exclude tests/ from git, update planning docs for Phase 7 2026-04-24 01:22:17 +08:00
Research Assistant
bef53fc61f docs(agents): add paperforge repair command reference 2026-04-24 01:10:06 +08:00
Research Assistant
1f817a5519 Add repair subcommand to CLI
- Add 'repair' subcommand with --verbose and --fix flags
- Wire to run_repair() from literature_pipeline worker
- Follow existing dispatch pattern (like deep-reading)
2026-04-24 01:07:13 +08:00
Research Assistant
ff1251522f fix: initialize validated_status and validated_error before try block
Both variables were assigned inside the try block but used after it.
If the try block threw before assignment, validated_error would be unbound
when line 2909 referenced it with done_incomplete status.

Also initialized validated_status = None for consistency since it could
also be unset if the try block fails before assignment.
2026-04-24 00:59:01 +08:00
Research Assistant
d0ba1da715 fix(run_repair): case 2 repair updates all three + case 3 handles None; add test_fix_case4 2026-04-24 00:42:44 +08:00
Research Assistant
6052c823e6 Add run_repair() function for three-way OCR state divergence repair 2026-04-24 00:02:32 +08:00
Research Assistant
981f5fa8ba fix(run_deep_reading): call validate_ocr_meta() before using ocr_status
- Replace direct meta.get('ocr_status') read with validate_ocr_meta() call
  in run_deep_reading() to properly validate OCR completion status
- Fixes bug where meta.json with ocr_status=done but missing files
  incorrectly marked paper as ready for deep reading
- validate_ocr_meta() checks 7 conditions before returning 'done':
  file existence, sizes, page markers, etc.
- done_incomplete status is correctly treated as blocked (not ready)
  by existing blocked logic at line 2810
- Add regression test: test_smoke_deep_reading_done_incomplete_is_blocked
2026-04-23 23:42:09 +08:00
Research Assistant
187af3b379 fix(cli): lazy pipeline import with idempotent guard; remove paperforge_lite deployment from wizard
Root cause: paperforge.exe pointed to vault's PaperForge/paperforge_lite/ (deployed by
wizard), which lacked the 'pipeline/' package, causing ModuleNotFoundError.

Fix:
- cli.py: deferred sys.path resolution + idempotent worker import (each global
  is imported only once; tests can patch stubs and guards skip if already bound)
- wizard: remove paperforge_lite copytree deployment (pip install -e . handles it)
- .gitignore: add tests/sandbox/00_TestVault/
2026-04-23 21:31:51 +08:00
Research Assistant
1d8478ce9c fix(wizard): auto-run pip install -e . and improve UX flow
- setup_wizard.py: _deploy() now calls pip install -e <repo_root> automatically
- DoneStep: removes manual 'pip install -e .' instruction (wizard handles it)
- README.md: removes separate pip install -r requirements.txt step
- docs/INSTALLATION.md: same pip install simplification
- pyproject.toml: add pytest ignore for nested sandbox vault
- tests/sandbox/: remove old vault/ (replaced by 00_TestVault/)
- tests/sandbox/README.md: update flow, wizard handles pip install
2026-04-23 20:49:10 +08:00
Research Assistant
31cce820e2 docs(sandbox): add pip install step to README 2026-04-23 20:34:13 +08:00
Research Assistant
e3151a6368 feat(tests/sandbox): add generate_sandbox.py with 5 mock papers across 2 domains 2026-04-23 20:27:26 +08:00
Research Assistant
dda7886d41 feat(tests): add sealed sandbox vault for paperforge testing 2026-04-23 20:18:27 +08:00
Research Assistant
b9f06ccad3 feat: complete PaperForge Lite v1.0 release hardening (5 phases, 28 reqs, 145 tests) 2026-04-23 20:15:20 +08:00
Research Assistant
08ddebbc0b docs(state): mark Phase 5 complete 2026-04-23 20:11:20 +08:00
Research Assistant
e79ff8cbc6 docs(05-01): complete plan 05-01 with OCR state machine tests and command docs verification 2026-04-23 20:10:03 +08:00
Research Assistant
935d948ede feat(05-01): add OCR state machine tests covering job transitions
- Test pending->queued transition on job submission
- Test processing->done transition when polling returns success
- Test processing->error transition on API error response
- Test processing->blocked transition via HTTPError 401 (classify_error maps to blocked)
- Test sync_ocr_queue skips done/blocked items from existing queue
- Test cleanup_blocked_ocr_dirs removes empty dirs, preserves dirs with payload
- Test all 7 OCR states (pending, queued, running, done, error, blocked, nopdf) don't crash

Key: patch requests.post with side_effect=HTTPError 401 to trigger
classify_error path, since registry token is always present in test env.
2026-04-23 19:37:15 +08:00
Research Assistant
b6df4ae510 docs(05-02): complete 05-02 plan metadata
- 05-02-SUMMARY.md created with 13 smoke tests documented
- STATE.md: Phase 5 progress table updated with 05-01/05-02 done
- ROADMAP.md: Phase 5 plans checked off, phase status updated to COMPLETE
- REQUIREMENTS.md: REL-02 marked complete (05-02)
2026-04-23 18:48:45 +08:00
Research Assistant
405fc3f127 test(05-02): add end-to-end smoke test suite covering full pipeline
Smoke tests (13 total):
- Setup validation (run_doctor returns 0/1, output checks)
- Selection sync (returns 0, scans exports, creates records)
- Index refresh (returns 0, generates formal Literature notes)
- OCR doctor L1-L3 (returns int code, detects missing token)
- Deep-reading queue (returns 0, produces 3-state output)
- CLI main entry (doctor/status return int, not sys.exit)

All tests use tmp_path fixtures, no real vault, no live network calls.
2026-04-23 18:46:28 +08:00
Research Assistant
1704395d34 test(05-02): add fixture vault factory for smoke testing
- fixture_vault: creates complete vault structure with paperforge.json
- fixture_library_records: 3 minimal library-record .md files
- fixture_bbt_json: Better BibTeX JSON export in exports/
- fixture_with_pdf: vault with real temp PDF for PDF-resolver tests
- All use pathlib.Path, Windows-compatible
2026-04-23 18:46:13 +08:00
Research Assistant
dc11e10c37 docs(state): record phase 5 context session 2026-04-23 18:26:49 +08:00
Research Assistant
f8b523baf9 docs(05): capture phase context 2026-04-23 18:26:18 +08:00
Research Assistant
6bbdb3d1d1 docs(02-03): complete OCR Doctor plan with summary, state, and roadmap updates
- Add 02-03-SUMMARY.md with execution summary and deviations
- Update STATE.md: mark 02-03 done, add decisions, update next action
- Update ROADMAP.md: mark 02-03 and 01-04 complete
2026-04-23 13:02:18 +08:00
Research Assistant
2a3295ed38 feat(02-03): add ocr doctor CLI dispatch, tests, and docs
- Convert ocr parser to sub-subcommands (run, doctor)
- Add _cmd_ocr_doctor() with formatted tiered report output
- Update dispatch logic to route ocr doctor subcommand
- Add CLI dispatch test for doctor command
- Update command/lp-ocr.md with doctor documentation
2026-04-23 12:59:04 +08:00
Research Assistant
3bd1c5ea69 feat(02-03): implement ocr_doctor() with L1-L4 diagnostics
- Add paperforge_lite/ocr_diagnostics.py with tiered L1-L4 checks
- L1: token presence, L2: URL reachability, L3: API schema validation
- L4: optional live PDF round-trip test with polling
- Add 7 mocked unit tests for all levels
- Include blank.pdf test fixture
2026-04-23 12:58:54 +08:00
Research Assistant
24e3d9fb30 test(02-01): add failing tests for pdf_resolver path resolution
- 8 test cases covering absolute, vault-relative, junction, storage-relative
- Missing file, has_pdf=False, empty path, whitespace path scenarios
- is_valid_pdf and resolve_junction helper coverage
2026-04-23 12:54:38 +08:00
Research Assistant
97a8fd6772 docs(01-03): complete plan 01-03 2026-04-23 12:07:00 +08:00
Research Assistant
500d268db7 feat(01-03): wire validate_setup.py to shared resolver
- load_config now tries paperforge_lite.config.load_vault_config first,
  falls back to legacy JSON parsing for pre-01-03 installs
- resolve_vault_for_validate checks PAPERFORGE_VAULT before VAULT_PATH
  before cwd (matching the shared resolver's precedence)
- Preserves all existing validation checks
2026-04-23 12:03:08 +08:00
Research Assistant
e0acb64e8d feat(01-03): deploy paperforge_lite package through setup wizard
- setup_wizard.py now copies the paperforge_lite/ package alongside
  the worker and ld_deep scripts during deployment
- Two copy targets: <pf_path>/worker/paperforge_lite/ for
  literature_pipeline.py and <skill_dir>/literature-qa/paperforge_lite/
  for ld_deep.py
- Existing copy targets for literature_pipeline.py and ld_deep.py
  remain unchanged
- Warning emitted if paperforge_lite source not found
2026-04-23 12:01:36 +08:00
Research Assistant
1c1e6d7c0d feat(01-03): replace duplicated resolver logic with shared wrappers
- literature_pipeline: load_vault_config and pipeline_paths now
  delegate to paperforge_lite.config, preserving public names
- ld_deep: _load_vault_config and _paperforge_paths now delegate
  to shared resolver, preserving public names
- Both scripts require paperforge_lite package to be available
  (via pip install or copied install in deployed vaults)
- Existing load_simple_env preserved for .env loading before dispatch
2026-04-23 12:00:24 +08:00
Research Assistant
fec25c2d3e feat(01-03): wire worker and /LD-deep to shared resolver
- Replace literature_pipeline.load_vault_config body with
  paperforge_lite.config.load_vault_config wrapper
- Rebuild pipeline_paths() from shared paperforge_paths() plus
  10 worker-only keys (pipeline, candidates, search_*, harvest_root,
  records, review, config, queue, log, bridge_config*, index, ocr_queue)
- Replace ld_deep._load_vault_config and _paperforge_paths with
  shared resolver wrappers returning ocr, records, literature
- Fix subprocess test to set PYTHONPATH so paperforge_lite is
  importable when worker runs as a standalone script
- 34 tests pass (22 config + 8 worker compat + 4 ld_deep compat)
2026-04-23 12:00:11 +08:00
Research Assistant
618227eb9b test(01-03): add failing tests for worker and /LD-deep resolver integration
- Test literature_pipeline.load_vault_config matches paperforge_lite.config
- Test PAPERFORGE_SYSTEM_DIR env var override in worker
- Test pipeline_paths returns all 17 expected keys (shared + worker-only)
- Test ld_deep._load_vault_config and _paperforge_paths match shared resolver
- Test direct worker status subprocess smoke test (CMD-02)
- Tests fail against current code: worker has 5 config keys vs shared's 7,
  env overrides not implemented, and pipeline_paths lacks shared resolver keys
2026-04-23 11:55:24 +08:00
Research Assistant
767548d2fe docs(01-02): complete 01-02 plan
- paperforge CLI launcher with argparse subcommands
- pyproject.toml packaging entry point
- load_simple_env added to config.py
- 9 CLI tests (3 path + 6 dispatch) passing
- 31 total tests passing
- CONF-02, CMD-01, CMD-03 requirements completed
2026-04-23 11:48:26 +08:00
Research Assistant
9496382e81 feat(01-02): implement CLI launcher, packaging entry point, and shared config resolver
- paperforge_lite/cli.py: argparse CLI with global --vault, subcommands paths/status/selection-sync/index-refresh/deep-reading/ocr (ocr defaults to run action)
- paperforge_lite/__main__.py: python -m paperforge_lite fallback entry point
- paperforge_lite/config.py: shared resolver (load_vault_config, resolve_vault, paperforge_paths, paths_as_strings, load_simple_env) matching Plan 01 contract
- pyproject.toml: setuptools build, paperforge-lite package, Python >=3.10, dependencies from requirements.txt, [project.scripts] paperforge=paperforge_lite.cli:main
- paperforge_lite/__init__.py: package init (version 1.0.0)

CLI loads .env from vault root and <system_dir>/PaperForge/.env before worker dispatch, matching legacy pipeline behavior.
2026-04-23 11:44:26 +08:00
Research Assistant
216ea4e8ac test(01-02): add failing test for CLI paths output and worker dispatch
- Tests main() with --vault vault paths --json emits valid JSON with vault, worker_script, ld_deep_script keys
- Tests main() with --vault vault paths emits text paths without unresolved <system_dir> or <resources_dir> tokens
- Tests ocr and ocr run both dispatch to run_ocr
- Tests status/selection-sync/index-refresh/deep-reading each dispatch to correct worker
- Uses monkeypatch to isolate from real workers and filesystem
2026-04-23 11:44:09 +08:00
Research Assistant
6dc62ac9f1 docs(01-01): complete 01-01 plan
- SUMMARY: 22 passing tests, 3 files created, 2 task commits (test + feat)
- STATE: phase 1 progress updated, decisions logged
- ROADMAP: 01-01 marked complete
2026-04-23 11:43:37 +08:00