PaperForge is a local-first literature workflow that bridges Zotero (reference management) and Obsidian (knowledge management) for medical researchers. The system is intentionally split into two distinct layers: the **Worker layer** and the **Agent layer**. This separation is the defining architectural choice of the project.
The **Worker layer** (`literature_pipeline.py` and the `paperforge/commands/` package) handles all automated, mechanical tasks: detecting new literature from Zotero via Better BibTeX JSON export, generating library records and formal notes, running OCR through the PaddleOCR API, and maintaining state consistency across the system. Workers are deterministic, idempotent where possible, and designed to run without human intervention. They are triggered by CLI commands such as `paperforge sync` or `paperforge ocr`.
The **Agent layer** (OpenCode Agent skills like `/pf-deep` and `/pf-paper`) handles interactive, cognitive tasks: deep reading, critical analysis, figure interpretation, and synthesis writing. Agents require human direction — they are triggered by explicit user commands and operate on data prepared by the Worker layer. An Agent never triggers a Worker automatically, and a Worker never triggers an Agent automatically.
This separation matters for three reasons. First, it keeps the automation layer simple and testable — workers are plain Python functions with clear inputs and outputs. Second, it respects user agency — deep reading is a deliberate act, not a background process. Third, it isolates failure domains: a bug in OCR handling cannot corrupt a user's carefully written analysis, and an Agent hallucination cannot damage the underlying library index.
---
## Data Flow
The complete pipeline flows from Zotero to Obsidian through six stages, with file formats and state transitions at each step.
PaperForge uses **5 core directories** under the Obsidian vault root. All paths are configurable via `paperforge.json` and resolved through the shared config resolver (see [ADR-001](#adr-001-config-precedence)).
| `<system_dir>/Zotero/` | Junction or symlink to Zotero data directory | User (installation step) | Read-only for PaperForge |
The separation between `<resources_dir>/` (user-facing, should be backed up) and `<system_dir>/` (system-generated, can be rebuilt) is intentional. If a vault is lost, `<resources_dir>/` contains the valuable intellectual output; `<system_dir>/PaperForge/ocr/` and `<system_dir>/PaperForge/exports/` can be regenerated by re-running workers.
---
## Commands Package
### Why `paperforge/commands/`?
In v1.1 and earlier, CLI commands were implemented directly in `cli.py` or invoked through the legacy `literature_pipeline.py` script. This created three problems:
1.**Code duplication**: The same sync logic existed in the CLI path and the worker script path.
2.**Test fragility**: CLI tests had to patch `sys.path` and mock imports differently from worker tests.
3.**Agent divergence**: Agent commands (`/pf-deep`) could not reuse CLI logic because the CLI was tightly coupled to `argparse`.
In Phase 9 (v1.2), we extracted shared command logic into `paperforge/commands/` — a package where each module implements a single command as a pure function taking an `args` namespace. Both the CLI (`cli.py`) and Agent skills import from this package.
### Registry Pattern
The `commands/__init__.py` exposes a registry for dynamic dispatch:
```python
# paperforge/commands/__init__.py
_COMMAND_REGISTRY: dict[str, str] = {
"sync": "paperforge.commands.sync",
"ocr": "paperforge.commands.ocr",
"deep": "paperforge.commands.deep",
"repair": "paperforge.commands.repair",
"status": "paperforge.commands.status",
}
def get_command_module(name: str):
"""Dynamically import a command module by name."""
import importlib
module_path = _COMMAND_REGISTRY.get(name)
if module_path is None:
raise ValueError(f"Unknown command: {name}")
return importlib.import_module(module_path)
```
### Command Module Structure
Each command module follows a uniform contract:
```python
# paperforge/commands/sync.py
import argparse
def run(args: argparse.Namespace) -> int:
"""Execute the command. Returns exit code (0 = success)."""
`cli.py` dispatches to command modules after resolving the vault and loading config:
```python
# paperforge/cli.py (excerpt)
if args.command == "sync":
from paperforge.commands import sync
return sync.run(args)
if args.command == "status":
from paperforge.commands import status
return status.run(args)
if args.command == "deep-reading":
from paperforge.commands import deep
return deep.run(args)
```
### Agent Integration
Agent skills (e.g., `skills/literature-qa/scripts/ld_deep.py`) reuse the same resolver logic but call `paperforge.commands` functions directly rather than going through the CLI argument parser.
### Backward Compatibility
Old command names (`selection-sync`, `index-refresh`, `ocr run`) are mapped to the new unified commands in `cli.py`:
**Context:** PaperForge needs to support multiple configuration sources: built-in defaults, JSON config files, environment variables, and explicit function arguments. Users need predictable override behavior.
The `load_vault_config()` function in `paperforge/config.py` merges sources in this exact order. No source can accidentally override a higher-precedence source.
**Context:** The original prototype mixed automated tasks (sync, OCR) with interactive tasks (deep reading) in a single script. This made it unclear what ran automatically vs. what required human judgment.
Workers never trigger Agents, and Agents never trigger Workers. The handoff point is the `library-record` frontmatter: users set `do_ocr: true` or `analyze: true`, then run the appropriate worker or agent command.
**Context:** PaddleOCR API calls are asynchronous and may take minutes for large PDFs. The system must handle pending, processing, completed, and failed states without blocking the user.
The `paperforge ocr` command scans all library-records for `do_ocr: true` and `ocr_status != done`, uploads PDFs, and updates both persistence layers. The `paperforge deep-reading` command checks `ocr_status == done` before listing a paper as "ready."
**Context:** Obsidian Base views (`.base` files) provide a database-like UI for library-records. Hardcoded paths in Base views break when users customize their directory structure.
**Decision:** Generate Base views from Jinja2 templates at setup time, injecting resolved paths from `paperforge_paths()`. Templates live in `paperforge/templates/bases/` and are rendered into `<base_dir>/` with user-configured directory names substituted.
**Context:** Deep reading of academic papers is a skill. The system needs a structured method that ensures comprehensive understanding without overwhelming the user.
**Context:** Integration tests for a literature pipeline require realistic inputs: Zotero JSON, PDFs, and Obsidian vault structures. Generating these dynamically in every test run is slow and non-deterministic.
**Context:** Early versions had path resolution logic duplicated between `literature_pipeline.py` and the CLI wrapper. Changes to one often broke the other.
**Decision:** Extract shared path resolution and config loading into `paperforge/config.py`. Both the CLI and worker scripts import from this module. The `paperforge_paths()` function returns a dict of resolved `Path` objects used consistently across the system.
**Context:** Over time, three sources of truth for a paper's status can diverge: the formal note frontmatter, the library-record frontmatter, and the actual OCR output directory. Users may manually edit one without updating the others.
**Context:** The `prepare_deep_reading()` function in `ld_deep.py` writes multiple files (figure-map.json, chart-type-map.json) and modifies the formal note before the Agent begins reading. If any step fails, partial writes leave the vault in an inconsistent state.
**Context:** v1.1 had fragmented command namespaces: CLI used `selection-sync`/`index-refresh`, Agent used `/LD-deep`/`/LD-paper`, and legacy scripts used `/lp-*` prefixes. This confused users and complicated documentation.
**Context:** Better BibTeX (BBT) exports attachment paths in three different formats depending on export settings and platform: absolute Windows paths (`D:\Zotero\storage\KEY\file.pdf`), `storage:` prefixed paths (`storage:KEY/file.pdf`), and bare relative paths (`KEY/file.pdf`). PaperForge needs to normalize these to a single internal representation and generate valid Obsidian wikilinks that work across Windows and macOS/Linux. Additionally, users may keep their Zotero data directory outside the Obsidian vault, requiring junction/symlink resolution.
**Decision:** Implement a three-stage normalization pipeline with explicit decisions for each ambiguity:
1.**D-01 — Normalize all paths to `storage:KEY/filename.pdf`:**`_normalize_attachment_path()` handles all three BBT formats and converts them to a unified `storage:` representation with forward slashes. Absolute paths outside Zotero storage get an `absolute:` prefix for explicit handling.
2.**D-02 — Hybrid main PDF selection:**`_identify_main_pdf()` uses a three-priority strategy: (1) attachment with `title == "PDF"`, (2) largest file by size (or shortest title if sizes are equal/unavailable), (3) first PDF in the list. All other PDFs become `supplementary`.
3.**D-03 — Store raw BBT path for debugging:** Every attachment carries `bbt_path_raw` preserving the original export string, enabling diagnosis when normalization produces unexpected results.
4.**D-04 — Extract 8-character storage key:**`zotero_storage_key` is extracted from absolute paths (`.../storage/8CHARKEY/...`) or `storage:8CHARKEY/...` prefix. This key is used for OCR directory naming and cross-referencing.
5.**D-05 — Resolve junctions before computing relative paths:**`absolutize_vault_path()` gains a `resolve_junction=True` parameter. When enabled, Windows junctions/symlinks in the path are resolved to their targets before `relative_to(vault)` is computed, ensuring wikilinks point to the true file location.
6.**D-06 — Track path errors explicitly:** A `path_error` frontmatter field (`not_found`, `invalid`, `permission_denied`) is set when PDF resolution fails. `paperforge repair --fix-paths` can re-run normalization and clear the error if the issue is resolved.
7.**D-07 — Doctor detects Zotero location and recommends junction:**`paperforge doctor` checks whether Zotero is inside the vault, outside with a junction, or missing. When outside without junction, it prints the exact `mklink /J` command needed.
8.**D-08 — Wikilinks use forward slashes exclusively:** All generated `pdf_path` values use `[[relative/path/file.pdf]]` format with `/` separators, even on Windows. `Path.as_posix()` is used instead of string replacement for robustness.
**Consequences:**
- (+) All real-world BBT export formats are handled without user configuration changes.
- (+) Wikilinks are vault-relative and work on all platforms.
- (+) Junctions are transparently resolved; users don't need to know the real Zotero data dir path.
- (+) Path errors are visible, diagnosable, and auto-repairable.
- (-) `storage:` prefix semantics are overloaded: it means "relative to Zotero data dir" in `pdf_resolver.py`, but the actual Zotero structure has an intermediate `storage/` directory. This requires `zotero_dir` configuration to point to the correct level.
- (-) One extra frontmatter field (`path_error`) adds noise to library-records; it is omitted when empty.
**Context:** Approximately 1,610 lines of utility code were copy-pasted across 7 worker modules; changes to one copy often missed others; bug fixes had to be applied N times. Each worker module independently defined `read_json`, `write_json`, `yaml_quote`, `slugify_filename`, `lookup_impact_factor`, and other helpers — with identical logic but subtle behavioral drifts over time.
**Decision:** Create `paperforge/worker/_utils.py` as a pure leaf module organized by category (JSON I/O, YAML Helpers, String/Path Utils, Journal Database, Constants). The module must never import from `paperforge.worker.*` or `paperforge.commands.*` — only from stdlib and `paperforge.config`. Functions are imported by workers as `from paperforge.worker._utils import read_json, ...`. Re-exports with `# Re-exported from _utils.py for backward compatibility` comments preserve backward compatibility in original modules.
**Consequences:**
- (+) Single source of truth for utility functions — changes in one place propagate to all workers.
- (+) No circular imports — leaf module constraint enforced by code review and consistency audit hook.
- (-) Re-export comments add minor clutter to original worker modules.
- (-) Module must be carefully audited at every change to maintain leaf status (import only stdlib + `paperforge.config`).
---
### ADR-013: Dual-Output Logging Strategy
**Status:** Accepted
**Phase:** 13 (Logging Foundation)
**Context:** The codebase used ad-hoc `print()` calls for all output — user-facing status, diagnostic traces, and error messages. This made it impossible to filter log levels, broke piped command output when diagnostic text appeared on stdout, and provided no structured error reporting for troubleshooting.
**Decision:** Implement a dual-output strategy:
1.`print()` preserved for user-facing formatted output on stdout only — existing piped commands and Agent scripts that parse stdout remain unmodified.
2.`logging.getLogger(__name__)` used for all diagnostic/trace/error output to stderr.
3. A `configure_logging(verbose: bool)` function in `paperforge/logging_config.py` configures the root logger.
4. Log level defaults to `INFO` (from `PAPERFORGE_LOG_LEVEL` env var) and switches to `DEBUG` when `--verbose`/`-v` is passed.
**Consequences:**
- (+) Structured log output with levels, module names, and timestamps — dramatically easier debugging.
- (+) Piped stdout remains clean — downstream consumers (Agent scripts, `grep`) see only user-facing output.
- (+) `--verbose` enables per-module debugging without modifying code.
- (-) Two output channels require documentation — users must know stdout (user-facing) vs stderr (diagnostic).
