obsidian-plugin/lllin000_PaperForge
1
0
Fork 0
mirror of https://github.com/lllin000/PaperForge.git synced 2026-07-22 06:50:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat(#37): auto-backup fulltext before rebuild

Browse source 
backup_render_before_rebuild() copies render/fulltext.md (and
render-map.json, heading-events.json) to versions/v{N}/ before
rebuild overwrites them. Creates/updates versions/manifest.json
with version metadata. Idempotent: skips when no render exists.

Hook placed between phase 3 and phase 4 in _rebuild_one_paper.

6 new tests.
This commit is contained in:
LLLin000 2026-07-10 00:34:44 +08:00
parent 06fec04651
commit 27e45f9419
4 changed files with 380 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

168
docs/design-v2-version-history-panel.md Normal file
View file

@ -0,0 +1,168 @@
# Design: Version History Panel (Phase 2a + 2b)
## Data Model
After #37 (backup), each paper's OCR directory looks like:
```
paper/ocr/{key}/
render/
fulltext.md ← current version
render-map.json
heading-events.json
versions/
manifest.json ← version index for this paper
v1/ ← pre-rebuild backup
fulltext.md
render-map.json
heading-events.json
v2/ ← second rebuild backup
...
```
`versions/manifest.json`:
```json
{
"versions": [
{
"label": "v1",
"created_at": "2026-07-09T14:30:00+08:00",
"source": "pre-rebuild",
"renderer_version": "2.0.0",
"structured_content_hash": "adbebf8c13e4e250",
"fulltext_size": 42156,
"pages": 3
}
],
"current": {
"label": "v2",
"created_at": "2026-07-10T10:00:00+08:00",
"source": "rebuild",
"renderer_version": "2.1.0",
"structured_content_hash": "7f8e3a2b1c4d6e5f",
"fulltext_size": 43892,
"pages": 3
}
}
```
## Entry Points
### 1. Maintenance Tab — Per Paper
Existing maintenance table adds a [版本历史] button for papers with backups.
### 2. Paper Mode — Dashboard
In PaperForge ItemView's paper mode (`_renderPaperMode`), add [版本历史] button in the status strip area.
### 3. Dedicated Panel — New ItemView Mode
A new "versions" mode, accessible from:
- Maintenance tab header button: "版本历史 (N)"
- Paper dashboard [版本历史] button
- Command palette: "PaperForge: Open Version History"
## Panel Layout (File Recovery-inspired)
```
+--------------------------------------------------+
| <- Back 版本历史 |
+----------------+---------------------------------+
| | Glueck_2005_Osteolysis |
| filter papers | ----------------------- |
| +----------+ | Versions |
| | search.. | | |
| +----------+ | [v3] (current) 2026-07-10 |
| | 43KB, renderer 2.1.0 |
| Glueck_2005 | |
| v1 v2 v3 | [v2] (rebuild) 2026-07-09 |
| | 42KB, renderer 2.0.0 |
| Gao_2020 | [restore] [compare] |
| v1 | |
| | [v1] (original) 2026-06-01 |
| | 38KB, OCR v1.5.15 |
| | [restore] [compare] |
| | |
| | +-- Compare (v2 vs current) --+ |
| | | 2/15 paragraphs changed | |
| | | Methods: restructured | |
| | | Results: table format fix | |
| | +------------------------------+ |
+----------------+---------------------------------+
| [restore selected] [clear old versions (free N MB)] |
+------------------------------------------------------+
```
### UI States
| State | Display |
|---|---|
| No backups | "No version history available" |
| Single version | No compare, just restore |
| Multiple versions | Timeline + compare + single-select restore |
| Loading | Skeleton |
| Error | "Cannot read version data" + retry |
## Compare View (Paragraph-level Diff)
Block-level comparison using `block_id` from structured blocks (if available) or paragraph-level text diff.
```
+---------------------------------------+
| Compare: Glueck_2005 |
| v2 (2026-07-09) vs current (2026-07-10) |
+---------------------------------------+
| Overview |
| -- Word count: 4250 -> 4310 (+60) |
| -- Paragraphs changed: 2/15 |
| -- Figures/tables: unchanged |
| |
| +-- Changes ------------------------+ |
| | Introduction | |
| | block_4: background paragraph | |
| | - The purpose of this study.. | |
| | + This study aimed to.. | |
| | | |
| | Methods | |
| | block_12: statistics paragraph | |
| | + We used SPSS version 26.. | |
| +------------------------------------+ |
| |
| [Restore this version] |
+---------------------------------------+
```
## Implementation Plan
### Phase 2a (#37): Backup (Python)
Modified files:
- `paperforge/worker/ocr_rebuild.py` — call `_backup_render_before_rebuild()` before phase 4
- New function in `paperforge/worker/ocr_versions.py`:
- Check if `versions/manifest.json` exists
- Copy render/ files to `versions/v{N}/`
- Write/update `versions/manifest.json`
- Returns version label (v1, v2, ...)
### Phase 2b (#38): Panel (TypeScript)
Modified files:
- `paperforge/plugin/src/views/dashboard.ts`:
- New `_renderVersionMode()` method
- Add [version history] button in `_renderPaperMode()`
- Add mode switch case for "versions"
- New file `paperforge/plugin/src/services/version-history.ts`:
- `scanVersionBackups(vaultPath, paperKey)` — read manifest.json
- `listPapersWithBackups(vaultPath)` — scan all OCR dirs
- `restoreVersion(vaultPath, paperKey, versionLabel)` — file copy
- `compareVersions(vaultPath, paperKey, vA, vB)` — paragraph diff
- `paperforge/plugin/styles.css` — panel layout + timeline + diff view
- `paperforge/plugin/src/i18n.ts` — new localization strings
### Data Contract
The `versions/manifest.json` format is the contract between Python (backup) and TypeScript (display). Must match exactly.
## Decisions
- Comparison granularity: **paragraph-level diff** (not line-level)
- Version badge: NOT in search results (user rejected)
- Entry points: maintenance tab + paper dashboard + dedicated panel

4
paperforge/worker/ocr_rebuild.py
View file

@ -576,6 +576,10 @@ def _rebuild_one_paper(vault: Path, key: str) -> dict:
table_inventory = phase3_result["table_inventory"]
reader_payload = phase3_result["reader_payload"]
# ── Backup current render before overwriting ──
from paperforge.worker.ocr_versions import backup_render_before_rebuild
backup_render_before_rebuild(paper_root)
rendered, health_overall = _phase4_render_health(
structured, resolved, figure_inventory, table_inventory,
reader_payload, doc_structure, ocr_meta, source_pdf_path,

76
paperforge/worker/ocr_versions.py
View file

@ -1,7 +1,11 @@
from __future__ import annotations
import datetime
import shutil
from pathlib import Path
from paperforge.core.io import read_json, write_json
# Version constants - single source of truth
EXPECTED_OCR_PROVIDER = "PaddleOCR"
EXPECTED_OCR_RAW_SCHEMA_VERSION = "1.0.0"
@ -139,3 +143,75 @@ def compute_structured_hash(vault: Path, key: str) -> str | None:
break
hasher.update(chunk)
return hasher.hexdigest()
def backup_render_before_rebuild(paper_root: Path) -> str | None:
"""Backup current render/ before rebuild overwrites it.
Copies render/fulltext.md (and related artifacts) to versions/v{N}/.
Creates or updates versions/manifest.json. Idempotent: skips when
render/fulltext.md doesn't exist (no prior render to preserve).
Returns version label (e.g. "v1") or None if nothing was backed up.
"""
render_dir = paper_root / "render"
ft_path = render_dir / "fulltext.md"
if not ft_path.exists():
return None
versions_root = paper_root / "versions"
manifest_path = versions_root / "manifest.json"
# Read existing manifest
manifest: dict = {"versions": [], "current": {}}
if manifest_path.exists():
try:
manifest = read_json(manifest_path)
except Exception:
manifest = {"versions": [], "current": {}}
# Determine next version label
existing = manifest.get("versions", [])
next_num = 1
if existing:
labels = [v.get("label", "") for v in existing]
nums = [int(l[1:]) for l in labels if l.startswith("v") and l[1:].isdigit()]
if nums:
next_num = max(nums) + 1
label = f"v{next_num}"
# Copy backup files
dest = versions_root / label
dest.mkdir(parents=True, exist_ok=True)
for fname in ["fulltext.md", "render-map.json", "heading-events.json"]:
src = render_dir / fname
if src.exists():
shutil.copy2(src, dest / fname)
# Build version entry
entry: dict = {
"label": label,
"created_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
"source": "pre-rebuild",
"fulltext_size": ft_path.stat().st_size,
}
# Carry forward structured_content_hash from meta.json if available
meta_path = paper_root / "meta.json"
if meta_path.exists():
try:
meta = read_json(meta_path)
h = meta.get("structured_content_hash")
if h:
entry["structured_content_hash"] = h
rv = meta.get("derived_version", {})
if rv:
entry["renderer_version"] = rv.get("renderer_version", "")
except Exception:
pass
existing.append(entry)
manifest["versions"] = existing
manifest["current"] = {"label": label}
write_json(manifest_path, manifest)
return label

132
tests/test_phase2a_backup.py Normal file
View file

@ -0,0 +1,132 @@
"""Tests for Phase 2a: rebuild backup mechanism."""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from paperforge.worker.ocr_versions import backup_render_before_rebuild
def test_backup_creates_version_manifest(tmp_path: Path):
"""Backup creates versions/v1/ with manifest.json and copied files."""
paper_root = tmp_path / "ocr" / "ME6BJZVS"
render = paper_root / "render"
render.mkdir(parents=True)
(render / "fulltext.md").write_text("# Original fulltext\n\nSome content.", encoding="utf-8")
(render / "render-map.json").write_text('{"meta": "v1"}', encoding="utf-8")
label = backup_render_before_rebuild(paper_root)
assert label == "v1"
manifest_path = paper_root / "versions" / "manifest.json"
assert manifest_path.exists()
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
assert len(manifest["versions"]) == 1
assert manifest["versions"][0]["label"] == "v1"
assert manifest["versions"][0]["fulltext_size"] > 0
assert (paper_root / "versions" / "v1" / "fulltext.md").exists()
assert (paper_root / "versions" / "v1" / "render-map.json").read_text(encoding="utf-8") == '{"meta": "v1"}'
def test_backup_increments_version_label(tmp_path: Path):
"""Second backup creates v2, third creates v3."""
paper_root = tmp_path / "ocr" / "ME6BJZVS"
render = paper_root / "render"
render.mkdir(parents=True)
# First backup
(render / "fulltext.md").write_text("# Version 1", encoding="utf-8")
v1 = backup_render_before_rebuild(paper_root)
assert v1 == "v1"
# Second backup (simulate rebuild's new content)
(render / "fulltext.md").write_text("# Version 2", encoding="utf-8")
v2 = backup_render_before_rebuild(paper_root)
assert v2 == "v2"
# Third backup
(render / "fulltext.md").write_text("# Version 3", encoding="utf-8")
v3 = backup_render_before_rebuild(paper_root)
assert v3 == "v3"
manifest = json.loads(
(paper_root / "versions" / "manifest.json").read_text(encoding="utf-8")
)
assert len(manifest["versions"]) == 3
assert manifest["versions"][0]["label"] == "v1"
assert manifest["versions"][1]["label"] == "v2"
assert manifest["versions"][2]["label"] == "v3"
def test_backup_no_render_returns_none(tmp_path: Path):
"""If no render/fulltext.md exists, returns None and creates no files."""
paper_root = tmp_path / "ocr" / "ME6BJZVS"
paper_root.mkdir(parents=True)
# No render/ at all
label = backup_render_before_rebuild(paper_root)
assert label is None
assert not (paper_root / "versions").exists()
def test_backup_carries_structured_content_hash(tmp_path: Path):
"""Backup carries forward structured_content_hash and renderer_version from meta.json."""
paper_root = tmp_path / "ocr" / "ME6BJZVS"
render = paper_root / "render"
render.mkdir(parents=True)
(render / "fulltext.md").write_text("# Test", encoding="utf-8")
# Write meta.json with hash and version info
meta = {
"structured_content_hash": "abc123",
"derived_version": {"renderer_version": "2.1.0"},
}
(paper_root / "meta.json").write_text(json.dumps(meta), encoding="utf-8")
label = backup_render_before_rebuild(paper_root)
assert label == "v1"
manifest = json.loads(
(paper_root / "versions" / "manifest.json").read_text(encoding="utf-8")
)
entry = manifest["versions"][0]
assert entry["structured_content_hash"] == "abc123"
assert entry["renderer_version"] == "2.1.0"
def test_backup_idempotent_corrupted_manifest(tmp_path: Path):
"""Backup handles corrupted manifest gracefully (starts fresh)."""
paper_root = tmp_path / "ocr" / "ME6BJZVS"
render = paper_root / "render"
render.mkdir(parents=True)
(render / "fulltext.md").write_text("# Text", encoding="utf-8")
# Write corrupted manifest
versions_dir = paper_root / "versions"
versions_dir.mkdir(parents=True)
(versions_dir / "manifest.json").write_text("{corrupted", encoding="utf-8")
label = backup_render_before_rebuild(paper_root)
assert label == "v1"
manifest = json.loads(
(versions_dir / "manifest.json").read_text(encoding="utf-8")
)
assert len(manifest["versions"]) == 1
def test_backup_non_default_files_not_required(tmp_path: Path):
"""Backup only requires fulltext.md; render-map and heading-events are optional."""
paper_root = tmp_path / "ocr" / "ME6BJZVS"
render = paper_root / "render"
render.mkdir(parents=True)
(render / "fulltext.md").write_text("# Only fulltext", encoding="utf-8")
# No render-map.json, no heading-events.json
label = backup_render_before_rebuild(paper_root)
assert label == "v1"
assert (paper_root / "versions" / "v1" / "fulltext.md").exists()
assert not (paper_root / "versions" / "v1" / "render-map.json").exists()
assert not (paper_root / "versions" / "v1" / "heading-events.json").exists()