---
phase: 59-state-machine-field-registry
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
  - paperforge/schema/__init__.py
  - paperforge/schema/field_registry.yaml
  - tests/test_field_registry.py
autonomous: true
requirements:
  - STAT-03

must_haves:
  truths:
    - "paperforge/schema/field_registry.yaml defines every field from formal note frontmatter, canonical index entries, and OCR meta.json"
    - "Each field entry has: name, type, required, public, owner, description"
    - "All 20+ frontmatter fields (zotero_key, domain, title, year, doi, pmid, collection_path, etc.) are defined in the registry"
    - "All 25+ canonical index fields (including derived: lifecycle, health, maturity, next_step) are defined"
    - "All OCR meta.json fields (ocr_provider, mode, ocr_job_id, etc.) are defined"
    - "Field types are specified as Python types (str, bool, int, list[str], dict, etc.)"
    - "A loader function exists that reads the YAML and validates its structure"
  artifacts:
    - path: "paperforge/schema/field_registry.yaml"
      provides: "Complete field definition registry"
      min_lines: 60
    - path: "paperforge/schema/__init__.py"
      provides: "Registry loader with validation"
      exports: ["load_field_registry", "get_field_definitions", "FieldDefinition"]
    - path: "tests/test_field_registry.py"
      provides: "Tests for registry loading and structure"
      min_lines: 50
  key_links:
    - from: "paperforge/schema/field_registry.yaml"
      to: "paperforge/core/state.py"
      via: "state field types reference Lifecycle/OcrStatus enums"
    - from: "paperforge/schema/__init__.py"
      to: "paperforge/schema/field_registry.yaml"
      via: "yaml.safe_load"
      pattern: "field_registry.yaml"
---

<objective>
Create the centralized field registry YAML that defines every field across all PaperForge data structures (formal note frontmatter, canonical index entries, and OCR meta.json).

Purpose: Create the single source of truth for field metadata — names, types, requirements, owners, descriptions — so that `paperforge doctor` can detect field drift (missing fields, unknown fields, type mismatches) in Phase 59-03.

Output: paperforge/schema/field_registry.yaml (field definitions), paperforge/schema/__init__.py (loader), unit tests.
</objective>

<execution_context>
@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md
@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/059-state-machine-field-registry/059-CONTEXT.md

<interfaces>
From paperforge/worker/sync.py (frontmatter template, lines 1012-1036):
```
title, year, journal, first_author, zotero_key, domain, doi, pmid, 
collection_path, impact_factor, abstract, has_pdf, do_ocr, analyze, 
ocr_status, deep_reading_status, pdf_path, fulltext_md_path, tags, supplementary
```

From paperforge/worker/asset_index.py (canonical index entry, lines 347-396):
```
zotero_key, domain, title, authors, first_author, abstract, journal, 
impact_factor, year, doi, pmid, collection_path, collections, collection_tags, 
collection_group, has_pdf, do_ocr, analyze, pdf_path, ocr_status, 
ocr_job_id, ocr_md_path, ocr_json_path, deep_reading_status, note_path, 
deep_reading_md_path, paper_root, main_note_path, fulltext_path, 
deep_reading_path, ai_path, lifecycle, health, maturity, next_step
```

From paperforge/worker/ocr.py ensure_ocr_meta (lines 60-80):
```
zotero_key, source_pdf, ocr_provider, mode, ocr_status, ocr_job_id, 
ocr_started_at, ocr_finished_at, page_count, markdown_path, json_path, 
assets_path, fulltext_md_path, error, retry_count, last_error, last_attempt_at
```
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Create field_registry.yaml with complete field definitions</name>
  <files>
    paperforge/schema/__init__.py
    paperforge/schema/field_registry.yaml
  </files>
  <action>
    Create the paperforge/schema/ directory with __init__.py and field_registry.yaml.

    **paperforge/schema/__init__.py**:
    A minimal package init that exports a typed dictionary loader:
    
    ```python
    from __future__ import annotations
    
    import os
    from pathlib import Path
    from typing import Any
    
    import yaml
    
    _REGISTRY_PATH = Path(__file__).parent / "field_registry.yaml"
    _registry_cache: dict[str, Any] | None = None
    
    def load_field_registry() -> dict[str, Any]:
        """Load and cache the field registry. Returns {owner: {field_name: definition}}."""
        global _registry_cache
        if _registry_cache is not None:
            return _registry_cache
        if not _REGISTRY_PATH.exists():
            raise FileNotFoundError(f"Field registry not found: {_REGISTRY_PATH}")
        with open(_REGISTRY_PATH, encoding="utf-8") as f:
            data = yaml.safe_load(f)
        if not isinstance(data, dict) or "owners" not in data:
            raise ValueError("field_registry.yaml must have top-level 'owners' key")
        _registry_cache = data
        return data
    
    def get_field_definitions(owner: str | None = None) -> dict[str, dict[str, Any]]:
        """Get all field definitions, optionally filtered by owner.
        
        Returns dict of {field_name: {name, type, required, public, owner, description, ...}}.
        """
        registry = load_field_registry()
        if owner:
            owners = registry.get("owners", {})
            return owners.get(owner, {})
        # Flatten all owners
        result = {}
        for owner_name, fields in registry.get("owners", {}).items():
            for field_name, definition in fields.items():
                result[field_name] = dict(definition, owner=owner_name)
        return result
    
    def get_all_field_names() -> set[str]:
        """Return the set of all known field names across all owners."""
        result = set()
        for owner_name, fields in registry.get("owners", {}).items():
            result.update(fields.keys())
        return result
    ```

    **paperforge/schema/field_registry.yaml** structure:
    
    Top-level keys:
    - `version`: schema version (string, e.g. "1.0")
    - `description`: short overview string
    - `owners`: dict of owner_name -> fields_dict
    
    Each owner groups fields by their data source. Three owners:
    1. `formal_note_frontmatter` — fields written to `<key> - <Title>.md` frontmatter
    2. `canonical_index` — fields in formal-library.json canonical index entries
    3. `ocr_meta` — fields in PaperForge/ocr/<key>/meta.json
    
    Each field entry has:
    - `type`: Python type name (str, bool, int, list[str], dict, etc.)
    - `required`: true/false — is this field mandatory for its owner?
    - `public`: true/false — exposed in dashboard/base views?
    - `description`: Human-readable explanation (1-2 sentences)
    - `example`: example value (optional)
    - `owners`: list of owners that include this field (added by get_field_definitions)
    
    Design the YAML file grouped by owner for readability. Fields that appear in multiple owners are repeated in each owner's section (the flattened get_field_definitions() handles dedup).

    The field definitions must include at minimum:
    
    **Formal note frontmatter** (18 fields):
    ```yaml
    formal_note_frontmatter:
      zotero_key:
        type: str
        required: true
        public: true
        description: "Zotero citation key (8-char alphanumeric)"
      domain:
        type: str
        required: true
        public: true
        description: "Classification domain matching Zotero collection top-level folder"
      title:
        type: str
        required: true
        public: true
        description: "Paper title from Zotero"
      year:
        type: int
        required: true
        public: true
        description: "Publication year"
      journal:
        type: str
        required: false
        public: true
        description: "Journal name"
      first_author:
        type: str
        required: false
        public: true
        description: "First author surname"
      doi:
        type: str
        required: false
        public: true
        description: "Digital Object Identifier"
      pmid:
        type: str
        required: false
        public: true
        description: "PubMed ID"
      collection_path:
        type: str
        required: false
        public: true
        description: "Zotero sub-collection path (pipe-separated)"
      impact_factor:
        type: str
        required: false
        public: false
        description: "Journal impact factor (from domain-level config)"
      abstract:
        type: str
        required: false
        public: false
        description: "Paper abstract from Zotero"
      has_pdf:
        type: bool
        required: true
        public: true
        description: "Whether paper has at least one PDF attachment"
      do_ocr:
        type: bool
        required: true
        public: true
        description: "User flag: run OCR on this paper?"
      analyze:
        type: bool
        required: true
        public: true
        description: "User flag: perform deep reading analysis?"
      ocr_status:
        type: str
        required: true
        public: true
        description: "OCR pipeline state (pending/processing/done/failed)"
      deep_reading_status:
        type: str
        required: true
        public: true
        description: "Deep reading state (pending/done)"
      pdf_path:
        type: str
        required: false
        public: true
        description: "Obsidian wikilink to main PDF attachment"
      fulltext_md_path:
        type: str
        required: false
        public: false
        description: "Obsidian wikilink to OCR fulltext markdown"
      tags:
        type: list[str]
        required: false
        public: false
        description: "YAML list tags (includes '文献阅读' and domain)"
      supplementary:
        type: list[str]
        required: false
        public: false
        description: "Obsidian wikilinks to supplementary PDF attachments"
    ```

    **Canonical index** (25+ fields — includes all frontmatter-derived fields plus index-specific):
    ```yaml
    canonical_index:
      zotero_key: {type: str, required: true, ...}
      domain: {type: str, required: true, ...}
      title: {type: str, required: true, ...}
      authors: {type: list[str], required: true, ...}
      first_author: {type: str, required: false, ...}
      abstract: {type: str, required: false, ...}
      journal: {type: str, required: false, ...}
      impact_factor: {type: str, required: false, ...}
      year: {type: int, required: true, ...}
      doi: {type: str, required: false, ...}
      pmid: {type: str, required: false, ...}
      collection_path: {type: str, required: false, ...}
      collections: {type: list[str], required: false, ...}
      collection_tags: {type: list[str], required: false, ...}
      collection_group: {type: list[str], required: false, ...}
      has_pdf: {type: bool, required: true, ...}
      do_ocr: {type: bool, required: true, ...}
      analyze: {type: bool, required: true, ...}
      pdf_path: {type: str, required: false, ...}
      ocr_status: {type: str, required: true, ...}
      ocr_job_id: {type: str, required: false, ...}
      ocr_md_path: {type: str, required: false, ...}
      ocr_json_path: {type: str, required: false, ...}
      deep_reading_status: {type: str, required: true, ...}
      note_path: {type: str, required: true, ...}
      deep_reading_md_path: {type: str, required: false, ...}
      paper_root: {type: str, required: false, ...}
      main_note_path: {type: str, required: false, ...}
      fulltext_path: {type: str, required: false, ...}
      deep_reading_path: {type: str, required: false, ...}
      ai_path: {type: str, required: false, ...}
      lifecycle: {type: str, required: false, ...}
      health: {type: dict, required: false, ...}
      maturity: {type: dict, required: false, ...}
      next_step: {type: str, required: false, ...}
      path_error: {type: str, required: false, ...}
    ```

    **OCR meta** (15 fields):
    ```yaml
    ocr_meta:
      zotero_key: {type: str, required: true, ...}
      source_pdf: {type: str, required: false, ...}
      ocr_provider: {type: str, required: true, ...}
      mode: {type: str, required: true, ...}
      ocr_status: {type: str, required: true, ...}
      ocr_job_id: {type: str, required: false, ...}
      ocr_started_at: {type: str, required: false, ...}
      ocr_finished_at: {type: str, required: false, ...}
      page_count: {type: int, required: false, ...}
      markdown_path: {type: str, required: false, ...}
      json_path: {type: str, required: false, ...}
      assets_path: {type: str, required: false, ...}
      fulltext_md_path: {type: str, required: false, ...}
      error: {type: str, required: false, ...}
      retry_count: {type: int, required: false, ...}
      last_error: {type: str, required: false, ...}
      last_attempt_at: {type: str, required: false, ...}
    ```

    Each field in each owner section must have the full metadata: `type`, `required`, `public`, `description`. The other fields from the CONTEXT.md spec (`name` is implicit from the key, `owner` is the section name) are included.

    Write the YAML cleanly with comments grouping fields by purpose (metadata, workflow flags, derived state, etc.).
  </action>
  <verify>
    <automated>python -c "
from paperforge.schema import load_field_registry, get_field_definitions, get_all_field_names
registry = load_field_registry()
assert 'owners' in registry, 'Missing owners key'
owners = registry['owners']
assert 'formal_note_frontmatter' in owners, 'Missing formal_note_frontmatter owner'
assert 'canonical_index' in owners, 'Missing canonical_index owner'
assert 'ocr_meta' in owners, 'Missing ocr_meta owner'

# Check frontmatter fields
fm = owners['formal_note_frontmatter']
for req_field in ['zotero_key', 'domain', 'title', 'year', 'has_pdf', 'do_ocr', 'analyze', 'ocr_status', 'deep_reading_status']:
    assert req_field in fm, f'Missing frontmatter field: {req_field}'
    assert 'type' in fm[req_field], f'{req_field} missing type'
    assert 'required' in fm[req_field], f'{req_field} missing required'
    assert 'description' in fm[req_field], f'{req_field} missing description'

# Check canonical_index fields
ci = owners['canonical_index']
for req_field in ['zotero_key', 'domain', 'title', 'authors', 'has_pdf', 'ocr_status', 'deep_reading_status', 'lifecycle', 'note_path']:
    assert req_field in ci, f'Missing index field: {req_field}'

# Check ocr_meta fields
om = owners['ocr_meta']
for req_field in ['zotero_key', 'ocr_status', 'ocr_provider', 'mode', 'page_count']:
    assert req_field in om, f'Missing ocr_meta field: {req_field}'

# Test get_all_field_names
all_names = get_all_field_names()
assert 'zotero_key' in all_names
assert len(all_names) > 30, f'Expected 30+ fields, got {len(all_names)}'
print(f'Field registry loaded: {len(owners)} owners, {len(all_names)} total fields')
print('FIELD REGISTRY VALID')
"</automated>
  </verify>
  <done>paperforge/schema/field_registry.yaml created with owner-grouped field definitions for formal_note_frontmatter, canonical_index, and ocr_meta. Loader validates structure.</done>
</task>

<task type="auto">
  <name>Task 2: Write unit tests for field registry loading and structure</name>
  <files>
    tests/test_field_registry.py
  </files>
  <action>
    Create tests/test_field_registry.py with pytest tests covering:

    1. TestFieldRegistryLoading:
       - load_field_registry returns dict with "version" and "owners" keys
       - Raises FileNotFoundError if YAML is missing (if file not found — test this by patching path)
       - Raises ValueError if YAML lacks "owners" key

    2. TestFieldRegistryStructure:
       - Exactly 3 owners: formal_note_frontmatter, canonical_index, ocr_meta
       - formal_note_frontmatter has at least 18 fields
       - canonical_index has at least 30 fields
       - ocr_meta has at least 15 fields
       - Every field entry has type, required, public, description keys
       - All required fields have type: bool or type: str (validate type values are valid Python types)

    3. TestGetFieldDefinitions:
       - get_field_definitions(None) returns all flattened fields
       - get_field_definitions("formal_note_frontmatter") returns only that owner's fields
       - get_field_definitions("nonexistent") returns empty dict

    4. TestGetAllFieldNames:
       - Returns set of all unique field names
       - de-duplicates fields that appear in multiple owners
       - zotero_key and ocr_status appear in all 3 owners

    5. TestFieldTypeValues:
       - Valid types: str, bool, int, list[str], dict
       - No unknown types in any field definition

    Use from paperforge.schema import load_field_registry, get_field_definitions, get_all_field_names.
  </action>
  <verify>
    <automated>python -m pytest tests/test_field_registry.py -v --tb=short</automated>
  </verify>
  <done>All field registry tests pass: loading, structure, field completeness, type validation.</done>
</task>

</tasks>

<verification>
- python -c "from paperforge.schema import load_field_registry, get_field_definitions; r = load_field_registry(); print(len(r['owners']), 'owners'); print(sum(len(f) for f in r['owners'].values()), 'fields')" shows 3 owners, 60+ total fields
- pytest tests/test_field_registry.py -v passes
</verification>

<success_criteria>
- [ ] paperforge/schema/field_registry.yaml exists with 3 owner sections (formal_note_frontmatter, canonical_index, ocr_meta)
- [ ] formal_note_frontmatter defines 18+ fields with types, required flags, descriptions
- [ ] canonical_index defines 30+ fields (frontmatter-derived + index-specific + derived fields)
- [ ] ocr_meta defines 15+ fields (OCR pipeline state + metadata)
- [ ] Every field entry has type, required, public, description
- [ ] Loader validates structure and caches result
- [ ] All field registry tests pass
</success_criteria>

<output>
After completion, create .planning/phases/059-state-machine-field-registry/059-02-SUMMARY.md
</output>
