anthonyfitzpatrick_manuscri.../ARCHITECTURE.md

471 lines
25 KiB
Markdown
Raw Permalink Normal View History

# Manuscript Compiler Architecture <img src="logo.svg" alt="Manuscript Compiler logo" width="48" align="right">
## Purpose and design principles
Manuscript Compiler is an offline, mobile-safe Obsidian plugin that converts an author-reviewed vault structure into six native manuscript formats. The architecture is designed around several non-negotiable principles:
1. **One interpretation of the manuscript.** Scanning, detection, correction, parsing, cleaning, and ordering produce one semantic `Book`.
2. **Prepared identity.** Preview and export share the same `Book` instance inside a `PreparedCompileSession`.
3. **Format independence.** Exporters receive a shared `SemanticDocument`; they do not inspect the vault or reinterpret structure.
4. **Validate before delivery.** Generated bytes pass their format validator before any browser download starts.
5. **No manuscript vault output.** Delivery is a Blob download controlled by the host, never a vault or Node filesystem write.
6. **Privacy by construction.** No network, telemetry, cloud, prose-bearing logs, absolute-path history, or external tool exists.
7. **Deterministic native output.** Exporters generate their formats directly and use only controlled local data.
8. **Obsidian ownership.** Commands, events, settings, DOM, focus, and lifecycle use documented Obsidian APIs.
These are architectural constraints, not current implementation accidents.
## High-level architecture
```text
┌──────────────────────── Obsidian host ────────────────────────┐
│ commands · File Explorer menu · settings · modal lifecycle │
└──────────────────────────────┬──────────────────────────────────┘
│ exact TFolder root
┌──────────────────── Preparation boundary ──────────────────────┐
│ VaultScanner → ContentPlan → Parser/Cleaner → semantic Book │
│ → statistics/warnings/fingerprints │
│ → PreparedCompileSession │
└──────────────────────────────┬──────────────────────────────────┘
│ same Book instance
┌─────────────┴─────────────┐
▼ ▼
preview/validation SemanticDocument
exhaustive exporter registry
│ bytes
exhaustive validator registry
│ valid bytes
BrowserDownloadService
│ dispatch result
privacy-safe history
```
## Compile pipeline
The authoritative production sequence is:
```text
TFolder manuscript root
→ BookRootResolver
→ CompilePreparationService
→ VaultScanner
→ inferred or author-edited ContentPlanItem[]
→ applyContentPlan
→ ManuscriptParser
→ ContentCleaningPipeline
→ ordering helpers
→ semantic Book
→ StatisticsEngine + WarningEngine
→ source fingerprint + input signature
→ PreparedCompileSession
```
No route may skip stages. Command-palette, current-book, selected-folder, guided, validation, and sample entry points all converge on `CompilePreparationService.prepareAuthoritative`.
## Physical discovery and folder detection
`VaultScanner` performs mechanical discovery inside one exact `TFolder`. It identifies Markdown files and broad physical Part/Chapter containers but does not decide what should be published. Its output, `ScannedBook`, contains Obsidian file/folder objects and is intentionally unsuitable for export.
`BookRootResolver` centralises root selection. A root explicitly chosen through the File Explorer context menu is authoritative and is returned exactly. Compatibility commands may resolve a root from settings or active-note ancestry. The selected root names the Book but never becomes a Part, Chapter, or transparent heading.
Folder/name policies live in `content-plan.ts`:
- manuscript containers such as Manuscript, Draft, Content, or Chapters can be transparent;
- project folders such as Archive, Development, Research, Dashboards, Templates, and Exports default to ignored;
- front/back matter is inferred from names and ancestry;
- structural numbering is extracted without inventing Part 0 or Chapter 0;
- explicit author corrections always outrank inference.
Detection is conservative. Ambiguous items remain reviewable instead of disappearing.
## Structure correction
`ContentPlanItem[]` is the bridge between physical discovery and semantic parsing. Each item records path, parent, inferred/current role, local inclusion, sibling order, reason/warning, and whether the author overrode detection.
The Contents screen has two presentation modes over the same plan:
- normal review shows the compact outline, ignored groups, and warnings;
- correction mode exposes inclusion, role, disclosure, and order controls.
`workspace/content-tree.ts` owns pure inclusion/order transformations. `ContentsTreeViewState` owns scroll, focus identity, folder expansion, review filter, and correction mode; it does not own manuscript semantics. `CompileWorkspaceController` applies mutations and invalidates prepared output.
Important correction invariants:
- disabling a folder does not erase descendant choices;
- re-enabling restores child roles/order;
- transparent containers reparent descendants to the nearest structural ancestor;
- matter inheritance updates untouched descendants but preserves explicit overrides;
- the root is never a plan row;
- manual sibling order is authoritative.
## Parsing and cleaning
`ManuscriptParser` reads only the content-plan-rewritten `ScannedBook`. It parses YAML metadata for structure/filter decisions, applies `MetadataFilterEngine`, cleans publishable Markdown through `ContentCleaningPipeline`, constructs `Book`, and orders Parts/Chapters/Scenes.
Cleaning removes authoring syntax and structured project metadata while preserving ordinary prose. Supported rules include YAML front matter, Obsidian comments, HTML comments, Dataview blocks, callout markers, internal-link syntax, known authoring sections, and structured metadata regions. Body-section aliases can extract manuscript prose from templated notes.
Parser invariants:
- metadata never leaks into prose;
- Synopsis and Revision Notes are not manuscript body;
- cancellation stops queued reads and never returns a partial Book;
- excluded or empty documents do not contribute to statistics/export;
- missing structural numbers remain missing;
- ordering is deterministic.
## Semantic model
`src/model.ts` defines the publication model:
```text
Book
├── frontMatter: MatterSection
├── parts: Part[]
│ ├── orphanScenes: Scene[]
│ └── chapters: Chapter[]
│ └── scenes: Scene[]
├── orphanScenes: Scene[]
└── backMatter: MatterSection
```
`Book` contains cleaned manuscript content and semantic hierarchy. It contains no dependency on exporters or the UI. Changing this model is a cross-format product change and requires migration/fixture/export review.
`ScannedBook` and `Book` must never be conflated: the former is physical discovery; the latter is publication truth.
## PreparedCompileSession
`PreparedCompileSession` is an immutable-in-practice snapshot owned by one command or workspace. It contains:
- the exact resolved request and content plan;
- resolved profile;
- physical scan for diagnostics only;
- semantic `Book`;
- statistics, warnings, exclusions, variables, and compile result;
- included source paths;
- source-content fingerprint;
- semantic input signature;
- route and purpose.
Preview, validation, and export retain this object. They must not reconstruct its Book.
### Preview reuse and invalidation
Changes to inclusion, role, order, root, cleaning inputs, title/author variables, title page, table of contents, scene separator, semantic headings, or formatting that changes bytes invalidate preparation. Selected format and filename do not affect manuscript interpretation and therefore do not rebuild the Book.
Before export, `ExportCoordinator` recalculates:
- the source fingerprint from included file contents;
- the input signature from semantic request fields and plan.
A mismatch blocks export and instructs the author to refresh preview. Source hashing detects equal-size edits even when file timestamps do not change.
## SemanticDocument export projection
`SemanticDocument` is a format-oriented projection of the prepared `Book`. It provides ordered sections and blocks:
- title/author;
- front/back matter headings;
- Part/Chapter number and title headings;
- optional body headings;
- first paragraphs;
- later body paragraphs;
- scene and page breaks;
- inline bold/italic/link semantics.
The projection centralises presentation-relevant classification so exporters do not duplicate structural interpretation. It does not scan, parse, clean, reorder, or replace `Book`.
Paragraph state is semantic and presentation-independent. The first paragraph after a structural/body heading or scene break is marked `first`; later ordinary prose is Body Text. The indentation toggle only changes style declarations.
## Export pipeline
`ExportCoordinator` owns the terminal transaction:
1. acquire the global export operation;
2. verify prepared source/input fingerprints;
3. apply format-independent blocking policy;
4. persist current safe defaults;
5. create one `SemanticDocument` from `session.book`;
6. generate bytes using `EXPORTERS[format]`;
7. validate using `EXPORT_VALIDATORS[format]`;
8. enter non-cancellable finalisation;
9. start one browser download;
10. record success only if dispatch started;
11. release operation state in `finally`.
Exporters do not own UI, settings persistence, validation policy, delivery, or history. Validators do not repair bytes. Download service does not know semantic content.
### Export contracts
`ExportFormat` is the exhaustive union:
```text
docx | odt | epub | html | markdown | xml
```
`ManuscriptExportContext` contains the prepared session, shared `SemanticDocument`, resolved formatting, and portable filename. `GeneratedExport` contains format, filename, MIME type, bytes, and non-prose warnings.
Adding a format requires updating both exhaustive registries and all UI/testing/documentation/package surfaces. A registry gap is a compile-time error.
## Export formats
### DOCX
`docx.ts` produces native WordprocessingML using named styles and `fflate`. Styles explicitly encode title/matter/Part/Chapter weight, First Paragraph versus Body Text indentation, scene separators, page size, chapter page starts, and optional TOC field. It never converts another format. `docx-validator.ts` verifies essential ZIP and OOXML structure; it does not claim complete OOXML conformance.
### ODT
`OdtExporter` produces a native OpenDocument Text ZIP. The `mimetype` entry is first and uncompressed. Generated content references named paragraph styles in `styles.xml`; heading styles explicitly declare bold and body styles explicitly declare indentation. Validation checks package entries, mimetype ordering/storage, manifest targets, XML shape, and required styles.
### EPUB
`EpubExporter` produces an EPUB 3 ZIP with container, package document, navigation, ordered XHTML sections, and embedded stylesheet. The mimetype entry is first/uncompressed. XHTML uses semantic manuscript classes; CSS explicitly styles structural headings and body indentation without scripts, remote assets, or inline styles.
### HTML
`HtmlExporter` produces one offline HTML5 document with embedded CSS and semantic manuscript classes. It contains no JavaScript or remote reference. Presentation uses manuscript-specific selectors rather than global heading-weight rules.
### Markdown
`MarkdownExporter` renders deterministic UTF-8 Markdown from `SemanticDocument`. It preserves heading syntax, emphasis, readable link text, Unicode, paragraph spacing, and exactly one final newline. It does not fake first-line indentation with spaces, tabs, HTML, entities, or CSS.
### XML
`XmlExporter` produces deterministic semantic XML in `https://manuscript-compiler.dev/schema` with `schemaVersion="1.0"`. It contains metadata, matter, Part/Chapter/Scene hierarchy, paragraph elements, and inline emphasis/link spans. It excludes source paths, settings, profile IDs, diagnostics, and presentation styling.
## Browser download
`BrowserDownloadService` is the only completed-export delivery path. It creates a Blob and object URL, assigns URL/filename as anchor properties, appends and clicks one temporary `<a download>`, removes it, and revokes the URL on success or failure.
This boundary exists because:
- Obsidian Community Plugins must remain mobile-safe;
- the host owns save/share behavior;
- Node/Electron filesystem paths are unavailable or non-compliant;
- the plugin must not write manuscript exports into the vault;
- final external persistence cannot be observed truthfully.
Dispatch success means the host accepted the click. It does not mean a known filesystem path was written.
## Validation
The registry in `export-validators.ts` mirrors `EXPORTERS`. Validators perform restrained structural checks appropriate to each format:
- DOCX: required package/XML parts;
- ODT: entries, mimetype, manifest, XML, required styles;
- EPUB: entries, mimetype, container/package/navigation/spine targets and XHTML;
- HTML: UTF-8 document structure and manuscript content;
- Markdown: UTF-8, semantic required content, forbidden content, deterministic equivalence, final newline;
- XML: namespace/schema, structure, escaping, and presentation/privacy exclusions.
Validation failure blocks download. Automated structural validation does not replace Word/Vellum/LibreOffice/reader/browser interoperability checks.
## Operation state and cancellation
`OperationStateController` permits one active global operation and defines these states:
```text
idle → preparing → ready
idle/ready → exporting → finalising → complete
preparing/exporting → cancelled
preparing/exporting/finalising → failed
```
`AbortController` supplies cancellation before finalisation. `CompilationCancelledError` lets callers distinguish cancellation from failure. Finalisation locks cancellation because the download click cannot be rolled back or observed reliably.
The workspace controller deduplicates preparation/export promises, cancels preparation when appropriate, and releases state in terminal paths. A partially prepared Book is never published.
## Settings and presets
`settings.ts` defines persisted schema/defaults. `simple-workflow.ts` resolves guided choices into a complete compatibility profile. Vellum and Standard Manuscript are deterministic formatting presets; Custom preserves manual values.
`main.ts` loads raw settings, then calls `repairSettings` before services or UI consume them. UI components receive repaired data only.
Historical fields may remain for backward-compatible loading. They are storage-only if their feature was removed. In particular, legacy vault-output or external-converter fields cannot activate production behavior.
## Migration and repair
`migrateSettings` translates known historical shapes. `repairSettings` validates and bounds untrusted persisted data, nested profiles, history, and logs. Both are idempotent and preserve explicit user choices, including false/zero values and custom formatting.
Migration rules:
- missing fields receive behavior-preserving defaults;
- legacy units convert once without drift;
- profile IDs and bounded nested structures are repaired;
- removed fields are never revived;
- a second migration/repair pass makes no changes.
## History and diagnostics
`CompileHistoryService` records bounded terminal facts only after truthful outcomes. Success follows validation and started download dispatch. Cancellation and failure are distinct. Records can include format, portable filename, counts, timings, validation status, and download-started flag.
`history-storage.ts` whitelists persisted fields and repairs malformed arrays. `DiagnosticsReportGenerator` produces a redacted support summary without reading notes. Neither history nor diagnostics may retain manuscript prose, metadata values, warning details, absolute paths, usernames, Blob URLs, or external destinations.
## Obsidian integration
`main.ts` is the composition root. It:
- loads and repairs settings;
- constructs history, operation, preparation/export, and command services;
- registers stable command IDs;
- registers the documented File Explorer `file-menu` event;
- registers settings UI;
- owns onboarding and unload cancellation.
The selected folder action accepts only `TFolder`. UI uses Obsidian `Modal`, `Setting`, `Notice`, suggestion, icon, and component APIs. No global `app` is used. CSS is scoped to plugin roots and retains theme variables, focus-visible behavior, reduced-motion, high-contrast, narrow-pane, and mobile behavior.
## Security, privacy, and dependency policy
Production guarantees:
- offline execution;
- no telemetry/accounts/cloud/network;
- no remote resources;
- no Electron or Node filesystem;
- no shell, `child_process`, or external executable;
- no other community-plugin dependency;
- no manuscript vault output;
- no unsafe HTML injection;
- deterministic escaping for XML/HTML/OOXML;
- controlled ZIP entry paths;
- `fflate` as the sole runtime dependency.
Dependency additions require product approval, security/licence/maintenance/bundle/mobile analysis, lockfile review, updated notices, clean tests, and clean audit. See [SECURITY.md](SECURITY.md).
## Performance model
Preparation reads notes with bounded concurrency, builds one Book, calculates shared statistics/warnings, and hashes included content. Exporters reuse one SemanticDocument and do not repeat preparation.
The benchmark constructs 500 Chapters, 2,000 Scenes, and 2 million words, then runs all six formats with indentation enabled and disabled from the same Book. Results are informational; there is no strict machine-specific threshold. Maintainers should investigate repeated scanning, quadratic transforms, or disproportionate memory growth.
## Testing strategy
| Suite | Primary contract |
| --- | --- |
| `npm test` | detection, parsing, cleaning, migration, workspace state, privacy, accessibility/compliance source guards |
| `test:docx` | native DOCX package, semantic Word styles, Vellum-oriented regression |
| format suites | complete generated package/markup and matching validator behavior |
| `test:exports` | exhaustive registries, all formats, delivery, forbidden content, deterministic output |
| `benchmark:large` | one preparation, all exporters, runaway regression guard |
| package validation | exact release allowlist and version identity |
Fixtures use synthetic content. Generated manual-inspection artifacts are ignored and excluded from packages. Live application checks remain unchecked in `MANUAL_TESTING.md` until actually performed.
## Directory layout
```text
src/ production TypeScript
src/workspace/ three-stage workspace state, views, and projections
tests/ regression, integration, fixture, export, benchmark code
tests/fixtures/ privacy-safe representative vault layouts
tests/golden/ deterministic Markdown expectations
samples/ author-facing example projects
scripts/package.mjs release allowlist creation/validation
.github/workflows/ci.yml continuous integration
styles.css scoped plugin presentation
manifest.json immutable plugin identity/version contract
USER_GUIDE.md author documentation and screenshot plan
DEVELOPER_GUIDE.md maintenance and extension procedures
SECURITY.md security guarantees and reporting
MANUAL_TESTING.md intentionally unchecked interoperability gates
RELEASE_READINESS.md automated/manual release gates
```
## Source map
### Composition and commands
- `main.ts` — Obsidian lifecycle and dependency composition.
- `commands.ts` — stable command IDs.
- `compile-command-service.ts` — command route orchestration.
- `folder-context-menu.ts` — exact-folder File Explorer entry.
- `book-root-resolver.ts` — root resolution policy.
### Preparation and semantic construction
- `compile-preparation.ts` — sole root-to-prepared-session boundary.
- `vault-scanner.ts`, `types.ts` — physical discovery.
- `content-plan.ts` — detection, correction authority, scan rewriting.
- `parser.ts` — semantic Book construction.
- `filters.ts`, `metadata-filter.ts` — cleaning/filtering.
- `ordering.ts`, `statistics.ts`, `warnings.ts` — deterministic derived semantics.
- `model.ts` — semantic Book and result contracts.
- `compiler.ts` — parser/statistics/warnings compatibility facade.
### Export
- `semantic-document.ts` — shared export projection.
- `export-types.ts` — format contracts and metadata.
- `native-exporters.ts`, `markdown-exporter.ts`, `docx.ts` — native byte generators.
- `export-validators.ts`, `docx-validator.ts` — pre-delivery validation.
- `export-coordinator.ts` — verified terminal transaction.
- `browser-download.ts` — Blob/object-URL delivery.
- `export-filename.ts`, `export-safety.ts`, `measurements.ts` — shared policies.
### Workspace and UI
- `compile-modal.ts` — modal shell and step composition.
- `workspace/compile-workspace-controller.ts` — state, invalidation, cancellation, dispatch.
- `workspace/*-step.ts` — DOM rendering only.
- `workspace/content-tree.ts` — pure plan mutations/projections.
- `workspace/contents-tree-view-state.ts` — focus/scroll/collapse state.
- `workspace/*view-model.ts`, `export-preview.ts` — pure display projections.
- `ui.ts`, `wizards.ts` — settings, onboarding, secondary/legacy modals.
### Persistence and support
- `settings.ts`, `profiles.ts`, `simple-workflow.ts` — schema, migration/repair, preset resolution.
- `compile-history.ts`, `history-storage.ts` — bounded privacy-safe records.
- `diagnostics.ts` — redacted support report.
- `operation-state.ts`, `cancellation.ts` — shared operation vocabulary.
- `validation.ts` — read-only session validation.
## Extension points
Permitted extension points preserve boundaries:
- add a detection rule in `content-plan.ts` with representative fixtures;
- add a pure structured cleaning rule with prose-preservation tests;
- add a shared SemanticDocument presentation mapping;
- add a validator invariant for a verified malformed-output case;
- add a pure workspace view-model projection;
- add privacy-safe fixtures and interoperability documentation.
New formats, semantic Book fields, workflow stages, dependencies, delivery routes, or persistent data are product changes requiring explicit architecture review.
## Things never to break
1. Preview and every exporter use the same prepared `Book`.
2. Exporters never read the vault or parse notes.
3. The selected root and transparent containers never become accidental headings.
4. Explicit author inclusion, roles, and order beat inference.
5. Part 0 and Chapter 0 are never invented.
6. First paragraphs after headings/scene breaks remain semantically distinct.
7. Validation occurs before delivery.
8. One successful action initiates exactly one download.
9. No manuscript export is written into the vault.
10. Cancellation and failure are recorded truthfully.
11. Histories/diagnostics exclude prose and private paths/metadata.
12. Output remains deterministic, escaped, offline, and dependency-restrained.
13. Command IDs, manifest identity, and mobile compatibility remain stable.
14. Release packages contain exactly the three Obsidian runtime assets; the editable logo remains repository-only branding source.
## Future roadmap
The product is feature complete. The roadmap is maintenance rather than expansion:
- keep pace with documented Obsidian APIs and official lint requirements;
- record manual interoperability across current Word, Vellum, LibreOffice, EPUB readers, browsers, Markdown renderers, and XML tools;
- add fixtures for verified real-world detection or cleaning defects;
- improve accessibility from measured/manual findings;
- strengthen validators only when real malformed output demonstrates a missing invariant;
- profile and optimise only verified large-manuscript regressions;
- review dependencies and security advisories regularly.
Any future feature proposal must first show how it preserves the semantic Book, prepared identity, exhaustive export/validation contract, offline privacy model, browser delivery, mobile safety, and dependency policy.