---
phase: 59-state-machine-field-registry
plan: 03
type: execute
wave: 2
depends_on:
  - 59-state-machine-field-registry/01
  - 59-state-machine-field-registry/02
files_modified:
  - paperforge/schema/validator.py
  - paperforge/worker/status.py
  - tests/test_doctor_field_checks.py
autonomous: true
requirements:
  - STAT-04
  - STAT-05

must_haves:
  truths:
    - "paperforge doctor reports missing required fields with actionable warnings and migration suggestions"
    - "paperforge doctor reports unknown/invalid fields as 'drift detected' diagnostics"
    - "Fields present in data but absent from registry trigger 'drift detected' warning"
    - "Fields present in registry but absent from data trigger 'missing required field' with severity levels"
    - "All validation results are structured and include the field name, owner, and current value (or absence)"
    - "Validation can be run against individual formal notes, index entries, or OCR meta.json files"
  artifacts:
    - path: "paperforge/schema/validator.py"
      provides: "FieldRegistryValidator class with check_field_completeness()"
      exports: ["FieldRegistryValidator"]
    - path: "paperforge/worker/status.py"
      provides: "Integrated field check in doctor command"
    - path: "tests/test_doctor_field_checks.py"
      provides: "Tests for field completeness, drift detection, severity levels"
      min_lines: 80
  key_links:
    - from: "paperforge/schema/validator.py"
      to: "paperforge/schema/field_registry.yaml"
      via: "load_field_registry()"
      pattern: "from paperforge.schema import load_field_registry"
    - from: "paperforge/worker/status.py"
      to: "paperforge/schema/validator.py"
      via: "FieldRegistryValidator"
      pattern: "from paperforge.schema.validator import FieldRegistryValidator"
---

<objective>
Implement field completeness validation in `paperforge doctor` using the field registry from Plan 02 — detecting missing required fields, unknown/drift fields, and producing actionable diagnostics with severity levels.

Purpose: Make field drift detectable at `paperforge doctor` time. Ensure no required field silently disappears from any data structure. Provide migration suggestions for missing fields.

Output: paperforge/schema/validator.py (FieldRegistryValidator), integrated doctor checks in worker/status.py, unit tests.
</objective>

<execution_context>
@C:/Users/Lin/.opencode/get-shit-done/workflows/execute-plan.md
@C:/Users/Lin/.opencode/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/059-state-machine-field-registry/059-CONTEXT.md

<interfaces>
From paperforge/schema/__init__.py (Plan 059-02):
```python
def load_field_registry() -> dict[str, Any]:
    """Returns {owners: {owner_name: {field_name: definition}}}"""
def get_field_definitions(owner: str | None = None) -> dict[str, dict]:
def get_all_field_names() -> set[str]:
```

From paperforge/worker/status.py (doctor check pattern):
```python
def add_check(area: str, status: str, message: str, fix: str | None = None):
    """Add a check result: status in ('pass', 'warn', 'fail')"""
```

From paperforge/worker/asset_index.py (read_index):
```python
def read_index(vault: Path) -> list[dict]:
    """Reads canonical index (formal-library.json), returns list of entry dicts."""
```
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Create FieldRegistryValidator in paperforge/schema/validator.py</name>
  <files>
    paperforge/schema/validator.py
  </files>
  <action>
    Create paperforge/schema/validator.py with the FieldRegistryValidator class.

    ```python
    from __future__ import annotations
    
    from typing import Any
    
    from paperforge.schema import load_field_registry, get_field_definitions
    
    
    class FieldCheckResult:
        """Result of a single field check."""
        def __init__(self, field: str, owner: str, severity: str, message: str, suggestion: str | None = None):
            self.field = field
            self.owner = owner
            self.severity = severity  # "error", "warning", "info"
            self.message = message
            self.suggestion = suggestion
        
        def to_dict(self) -> dict:
            return {
                "field": self.field,
                "owner": self.owner,
                "severity": self.severity,
                "message": self.message,
                "suggestion": self.suggestion,
            }
    
        def __repr__(self) -> str:
            sev = {"error": "[!!]", "warning": "[!]", "info": "[i]"}.get(self.severity, "[?]")
            return f"{sev} {self.owner}.{self.field}: {self.message}"
    
    
    class FieldRegistryValidator:
        """Validate field completeness of PaperForge data structures against the field registry."""
    
        SEVERITY_MAP = {
            "missing_required": "error",
            "unknown_field": "warning",
            "deprecated_field": "warning",
            "type_mismatch": "warning",
            "info": "info",
        }
    
        def __init__(self):
            self._registry = load_field_registry()
            self._owners = self._registry.get("owners", {})
    
        def check_owner_fields(self, owner: str, actual_fields: dict[str, Any]) -> list[FieldCheckResult]:
            """Validate a single data dict against one owner's field definitions.
            
            Args:
                owner: Owner name (e.g. "formal_note_frontmatter", "canonical_index", "ocr_meta")
                actual_fields: The actual dict of field_name -> value
            
            Returns:
                List of FieldCheckResult (empty if perfectly compliant)
            """
            results: list[FieldCheckResult] = []
            owner_defs = self._owners.get(owner, {})
            if not owner_defs:
                results.append(FieldCheckResult(
                    field="*", owner=owner, severity="warning",
                    message=f"Unknown owner '{owner}' — not in field registry",
                ))
                return results
            
            actual_keys = set(actual_fields.keys())
            defined_keys = set(owner_defs.keys())
    
            # 1. Missing required fields (registry has them, data doesn't)
            for field_name, defn in owner_defs.items():
                if defn.get("required") and field_name not in actual_keys:
                    suggestion = f"Add '{field_name}: <value>' to {owner} data"
                    if "default" in defn:
                        suggestion += f" (default: {defn['default']})"
                    results.append(FieldCheckResult(
                        field=field_name, owner=owner,
                        severity="error",
                        message=f"Missing required field '{field_name}' ({defn.get('type', '?')})",
                        suggestion=suggestion,
                    ))
                elif field_name not in actual_keys and not defn.get("required"):
                    results.append(FieldCheckResult(
                        field=field_name, owner=owner,
                        severity="info",
                        message=f"Optional field '{field_name}' missing (type: {defn.get('type', '?')})",
                    ))
    
            # 2. Unknown fields (data has them, registry doesn't) — drift detection
            unknown = actual_keys - defined_keys
            for field_name in sorted(unknown):
                val = actual_fields.get(field_name)
                val_preview = str(val)[:80] if val is not None else "None"
                results.append(FieldCheckResult(
                    field=field_name, owner=owner,
                    severity="warning",
                    message=f"DRIFT: Unknown field '{field_name}' (value: {val_preview}) — not in field registry for '{owner}'",
                    suggestion=f"Add '{field_name}' to {owner} in paperforge/schema/field_registry.yaml, or remove it from data",
                ))
    
            # 3. Type mismatch (best-effort basic type checking)
            for field_name, defn in owner_defs.items():
                if field_name not in actual_keys:
                    continue
                expected_type = defn.get("type", "str")
                actual_val = actual_fields[field_name]
                if not self._type_matches(expected_type, actual_val):
                    actual_type_name = type(actual_val).__name__
                    results.append(FieldCheckResult(
                        field=field_name, owner=owner,
                        severity="warning",
                        message=f"Type mismatch for '{field_name}': expected {expected_type}, got {actual_type_name}",
                    ))
    
            return results
    
        @staticmethod
        def _type_matches(expected: str, actual: Any) -> bool:
            """Basic type checking — returns True if actual value is compatible with expected type."""
            type_map = {
                "str": str,
                "bool": bool,
                "int": (int, bool),  # bool is subclass of int
                "list[str]": list,
                "list": list,
                "dict": dict,
            }
            if actual is None:
                return True  # None is compatible with any type
            py_type = type_map.get(expected)
            if py_type is None:
                return True  # Unknown type -> skip
            if isinstance(py_type, tuple):
                return isinstance(actual, py_type)
            return isinstance(actual, py_type)
    
        def check_all_owners(self, data_map: dict[str, dict[str, Any]]) -> dict[str, list[FieldCheckResult]]:
            """Validate data against all registered owners.
            
            Args:
                data_map: dict of owner_name -> actual_fields_dict
            
            Returns:
                dict of owner_name -> list[FieldCheckResult]
            """
            return {
                owner: self.check_owner_fields(owner, fields)
                for owner, fields in data_map.items()
            }
    ```

    Module-level `__all__ = ["FieldRegistryValidator", "FieldCheckResult"]`.
  </action>
  <verify>
    <automated>python -c "
from paperforge.schema.validator import FieldRegistryValidator, FieldCheckResult

# Basic result creation
r = FieldCheckResult('zotero_key', 'formal_note_frontmatter', 'error', 'Missing field')
assert r.severity == 'error'
assert r.field == 'zotero_key'
assert 'zotero_key' in repr(r)

# Validate with fully compliant data
validator = FieldRegistryValidator()
results = validator.check_owner_fields('formal_note_frontmatter', {
    'zotero_key': 'ABC123',
    'domain': '骨科',
    'title': 'Test',
    'year': 2024,
    'has_pdf': True,
    'do_ocr': True,
    'analyze': False,
    'ocr_status': 'pending',
    'deep_reading_status': 'pending',
})
error_results = [r for r in results if r.severity == 'error']
if error_results:
    print(f'UNEXPECTED ERRORS: {[(r.field, r.message) for r in error_results]}')
assert len(error_results) == 0, f'Expected 0 errors for compliant data, got {len(error_results)}'

# Test drift detection
drift_results = validator.check_owner_fields('formal_note_frontmatter', {
    'zotero_key': 'ABC123',
    'domain': '骨科',
    'title': 'Test',
    'year': 2024,
    'has_pdf': True,
    'do_ocr': True,
    'analyze': False,
    'ocr_status': 'pending',
    'deep_reading_status': 'pending',
    'mystery_field_xyz': 'some_value',  # drift
})
drift = [r for r in drift_results if 'DRIFT' in r.message]
assert len(drift) >= 1, f'Expected drift detection, got {[(r.field, r.severity) for r in drift_results]}'
assert drift[0].severity == 'warning'

# Test unknown owner
unknown = validator.check_owner_fields('nonexistent', {'a': 1})
assert len(unknown) >= 1
assert 'Unknown owner' in unknown[0].message

print('VALIDATOR CORE TESTS PASS')
"</automated>
  </verify>
  <done>FieldRegistryValidator created with check_owner_fields (missing required, unknown/drift, type mismatch) and check_all_owners. All core checks working.</done>
</task>

<task type="auto">
  <name>Task 2: Integrate field completeness checks into paperforge doctor</name>
  <files>
    paperforge/worker/status.py
  </files>
  <action>
    Integrate FieldRegistryValidator checks into the doctor diagnostic workflow in paperforge/worker/status.py.

    **Add import** at top of file (after existing imports):
    ```python
    from paperforge.schema.validator import FieldRegistryValidator
    ```

    **Add a new helper function** `_check_field_registry(vault: Path, paths: dict, add_check: callable) -> None` near the end of status.py (before the main function, around line 1000+).

    The function should:
    1. Initialize FieldRegistryValidator
    2. Check canonical index entries (from formal-library.json):
       - Read the index using `asset_index.read_index(vault)` if available
       - For each entry, check against canonical_index owner definitions
       - Report missing required fields as "fail" via add_check
       - Report unknown fields as "warn" via add_check
    3. Check OCR meta.json files:
       - Scan `<system_dir>/PaperForge/ocr/*/meta.json`
       - For each meta file, check against ocr_meta owner definitions
       - Report patterns (e.g., "3 of 15 entries have missing required field 'ocr_job_id'")
    4. Check a sample of formal notes (up to 5):
       - Find literature notes via domain directories
       - Read frontmatter, check against formal_note_frontmatter definitions
       - Report summary statistics

    The checks should produce **aggregate** diagnostics (not per-entry noise). For example:
    ```
    [PASS] Field Registry: formal_note_frontmatter — all 18 fields compliant (sample: 5 notes)
    [WARN] Field Registry: canonical_index — 2 of 42 entries have unknown fields: ['extra_field_x']  
    [FAIL] Field Registry: ocr_meta — 1 of 15 entries missing required field 'ocr_provider'
    ```

    Use `add_check("Field Registry", status, message, fix_if_needed)` following the existing pattern.

    If no data files can be read (vault not set up), emit a single "skip" check with explanation.

    **Call the new function from the main doctor workflow**. Find where doctor runs vault-level checks (around vault structure checks, line 465+) and add a call:
    ```python
    # Field registry completeness check (Phase 59)
    _check_field_registry(vault, paths, add_check)
    ```

    The function should be placed after the existing vault structure checks (after the exports/BBT checks) but before the final summary.
  </action>
  <verify>
    <automated>python -c "
# Verify the function exists and imports correctly
from paperforge.worker.status import _check_field_registry
import inspect
assert callable(_check_field_registry), '_check_field_registry must be callable'
sig = inspect.signature(_check_field_registry)
params = list(sig.parameters.keys())
assert 'vault' in params, f'Expected vault param, got {params}'
assert 'paths' in params, f'Expected paths param, got {params}'
assert 'add_check' in params, f'Expected add_check param, got {params}'

# Verify the call is wired into the main doctor function
import re
with open('paperforge/worker/status.py', 'r') as f:
    source = f.read()
assert '_check_field_registry' in source, '_check_field_registry not found in status.py'
assert 'FieldRegistryValidator' in source, 'FieldRegistryValidator not found in status.py'
print('DOCTOR INTEGRATION VERIFIED: _check_field_registry found and importable')
"
  </verify>
  <done>_check_field_registry function created and wired into doctor workflow. Field checks for canonical_index, ocr_meta, and formal_note_frontmatter produce aggregate diagnostics.</done>
</task>

<task type="auto" tdd="true">
  <name>Task 3: Write comprehensive tests for doctor field completeness checks</name>
  <files>
    tests/test_doctor_field_checks.py
  </files>
  <behavior>
    - Test 1: FieldRegistryValidator.check_owner_fields returns empty list for fully compliant data
    - Test 2: Missing required field produces "error" severity with field name in message
    - Test 3: Unknown field produces "warning" severity with "DRIFT" in message
    - Test 4: Optional missing field produces "info" severity (not error)
    - Test 5: Multiple owners check_all_owners returns correct owner-&gt;results mapping
    - Test 6: check_all_owners handles empty data_map (returns empty dict)
    - Test 7: Type mismatch detected for bool field receiving str value
    - Test 8: None values are compatible with any type (type check passes)
    - Test 9: FieldCheckResult.to_dict() returns expected shape
  </behavior>
  <action>
    Write RED test file tests/test_doctor_field_checks.py (will fail until Task 1 is done):

    Test classes:
    1. TestCheckOwnerFields:
       - test_compliant_data_no_errors: Provide dict with all required frontmatter fields -> empty results
       - test_missing_required_field_error: Missing zotero_key -> error severity, field in message
       - test_missing_optional_field_info: Missing impact_factor -> info severity
       - test_unknown_field_drift_warning: Extra field not in registry -> warning with DRIFT
       - test_unknown_owner_warning: Unknown owner name -> warning with "Unknown owner"

    2. TestCheckAllOwners:
       - test_multiple_owners: Verify owner->results mapping
       - test_empty_data_map: Returns empty dict (no crash)

    3. TestTypeMismatch:
       - test_bool_field_receives_str: Type mismatch warning
       - test_none_passes_any_type: None = compatible, no warning
       - test_int_field_receives_str: Type mismatch warning

    4. TestFieldCheckResult:
       - test_to_dict: Returns {field, owner, severity, message, suggestion} shape
       - test_repr: Returns formatted string with severity indicator

    Write standard pytest assertions (no special fixtures needed — pure unit tests).
  </action>
  <verify>
    <automated>python -m pytest tests/test_doctor_field_checks.py -v --tb=short</automated>
  </verify>
  <done>All doctor field completeness tests pass: missing required fields, drift detection, type mismatches, severity levels.</done>
</task>

</tasks>

<verification>
- python -c "from paperforge.schema.validator import FieldRegistryValidator; v = FieldRegistryValidator(); r = v.check_owner_fields('formal_note_frontmatter', {'zotero_key': 'x', 'domain': 'y', 'title': 'z', 'year': 2024, 'has_pdf': True, 'do_ocr': True, 'analyze': False, 'ocr_status': 'pending', 'deep_reading_status': 'pending'}); assert len([x for x in r if x.severity == 'error']) == 0"
- pytest tests/test_doctor_field_checks.py -v passes
- python -c "from paperforge.worker.status import _check_field_registry; print('doctor integration: OK')" confirms function is importable
</verification>

<success_criteria>
- [ ] paperforge/schema/validator.py exists with FieldRegistryValidator class
- [ ] field_registry.yaml field definitions are used by validator for completeness checks
- [ ] Missing required fields detected with "error" severity and migration suggestion
- [ ] Unknown/drift fields detected with "warning" severity and "DRIFT" label
- [ ] Type mismatches detected with "warning" severity
- [ ] `_check_field_registry()` exists in paperforge/worker/status.py and is called from doctor workflow
- [ ] Doctor field checks produce aggregate diagnostics (not per-entry noise)
- [ ] All doctor field check tests pass
</success_criteria>

<output>
After completion, create .planning/phases/059-state-machine-field-registry/059-03-SUMMARY.md
</output>