anthonyfitzpatrick_manuscript-compiler/ARCHITECTURE.md

Manuscript Compiler Architecture

Purpose and design principles

Manuscript Compiler is an offline, mobile-safe Obsidian plugin that converts an author-reviewed vault structure into six native manuscript formats. The architecture is designed around several non-negotiable principles:

1. One interpretation of the manuscript. Scanning, detection, correction, parsing, cleaning, and ordering produce one semantic Book.
2. Prepared identity. Preview and export share the same Book instance inside a PreparedCompileSession.
3. Format independence. Exporters receive a shared SemanticDocument; they do not inspect the vault or reinterpret structure.
4. Validate before delivery. Generated bytes pass their format validator before any browser download starts.
5. No manuscript vault output. Delivery is a Blob download controlled by the host, never a vault or Node filesystem write.
6. Privacy by construction. No network, telemetry, cloud, prose-bearing logs, absolute-path history, or external tool exists.
7. Deterministic native output. Exporters generate their formats directly and use only controlled local data.
8. Obsidian ownership. Commands, events, settings, DOM, focus, and lifecycle use documented Obsidian APIs.

These are architectural constraints, not current implementation accidents.

High-level architecture

┌──────────────────────── Obsidian host ────────────────────────┐
│ commands · File Explorer menu · settings · modal lifecycle    │
└──────────────────────────────┬──────────────────────────────────┘
                               │ exact TFolder root
                               ▼
┌──────────────────── Preparation boundary ──────────────────────┐
│ VaultScanner → ContentPlan → Parser/Cleaner → semantic Book    │
│            → statistics/warnings/fingerprints                  │
│            → PreparedCompileSession                            │
└──────────────────────────────┬──────────────────────────────────┘
                               │ same Book instance
                 ┌─────────────┴─────────────┐
                 ▼                           ▼
        preview/validation             SemanticDocument
                                              │
                                              ▼
                                  exhaustive exporter registry
                                              │ bytes
                                              ▼
                                  exhaustive validator registry
                                              │ valid bytes
                                              ▼
                                   BrowserDownloadService
                                              │ dispatch result
                                              ▼
                                      privacy-safe history

Compile pipeline

The authoritative production sequence is:

TFolder manuscript root
  → BookRootResolver
  → CompilePreparationService
  → VaultScanner
  → inferred or author-edited ContentPlanItem[]
  → applyContentPlan
  → ManuscriptParser
  → ContentCleaningPipeline
  → ordering helpers
  → semantic Book
  → StatisticsEngine + WarningEngine
  → source fingerprint + input signature
  → PreparedCompileSession

No route may skip stages. Command-palette, current-book, selected-folder, guided, validation, and sample entry points all converge on CompilePreparationService.prepareAuthoritative.

Physical discovery and folder detection

VaultScanner performs mechanical discovery inside one exact TFolder. It identifies Markdown files and broad physical Part/Chapter containers but does not decide what should be published. Its output, ScannedBook, contains Obsidian file/folder objects and is intentionally unsuitable for export.

BookRootResolver centralises root selection. A root explicitly chosen through the File Explorer context menu is authoritative and is returned exactly. Compatibility commands may resolve a root from settings or active-note ancestry. The selected root names the Book but never becomes a Part, Chapter, or transparent heading.

Folder/name policies live in content-plan.ts:

• manuscript containers such as Manuscript, Draft, Content, or Chapters can be transparent;
• project folders such as Archive, Development, Research, Dashboards, Templates, and Exports default to ignored;
• front/back matter is inferred from names and ancestry;
• structural numbering is extracted without inventing Part 0 or Chapter 0;
• explicit author corrections always outrank inference.

Detection is conservative. Ambiguous items remain reviewable instead of disappearing.

Structure correction

ContentPlanItem[] is the bridge between physical discovery and semantic parsing. Each item records path, parent, inferred/current role, local inclusion, sibling order, reason/warning, and whether the author overrode detection.

The Contents screen has two presentation modes over the same plan:

• normal review shows the compact outline, ignored groups, and warnings;
• correction mode exposes inclusion, role, disclosure, and order controls.

workspace/content-tree.ts owns pure inclusion/order transformations. ContentsTreeViewState owns scroll, focus identity, folder expansion, review filter, and correction mode; it does not own manuscript semantics. CompileWorkspaceController applies mutations and invalidates prepared output.

Important correction invariants:

• disabling a folder does not erase descendant choices;
• re-enabling restores child roles/order;
• transparent containers reparent descendants to the nearest structural ancestor;
• matter inheritance updates untouched descendants but preserves explicit overrides;
• the root is never a plan row;
• manual sibling order is authoritative.

Parsing and cleaning

ManuscriptParser reads only the content-plan-rewritten ScannedBook. It parses YAML metadata for structure/filter decisions, applies MetadataFilterEngine, cleans publishable Markdown through ContentCleaningPipeline, constructs Book, and orders Parts/Chapters/Scenes.

Cleaning removes authoring syntax and structured project metadata while preserving ordinary prose. Supported rules include YAML front matter, Obsidian comments, HTML comments, Dataview blocks, callout markers, internal-link syntax, known authoring sections, and structured metadata regions. Body-section aliases can extract manuscript prose from templated notes.

Parser invariants:

• metadata never leaks into prose;
• Synopsis and Revision Notes are not manuscript body;
• cancellation stops queued reads and never returns a partial Book;
• excluded or empty documents do not contribute to statistics/export;
• missing structural numbers remain missing;
• ordering is deterministic.

Semantic model

src/model.ts defines the publication model:

Book
├── frontMatter: MatterSection
├── parts: Part[]
│   ├── orphanScenes: Scene[]
│   └── chapters: Chapter[]
│       └── scenes: Scene[]
├── orphanScenes: Scene[]
└── backMatter: MatterSection

Book contains cleaned manuscript content and semantic hierarchy. It contains no dependency on exporters or the UI. Changing this model is a cross-format product change and requires migration/fixture/export review.

ScannedBook and Book must never be conflated: the former is physical discovery; the latter is publication truth.

PreparedCompileSession

PreparedCompileSession is an immutable-in-practice snapshot owned by one command or workspace. It contains:

• the exact resolved request and content plan;
• resolved profile;
• physical scan for diagnostics only;
• semantic Book;
• statistics, warnings, exclusions, variables, and compile result;
• included source paths;
• source-content fingerprint;
• semantic input signature;
• route and purpose.

Preview, validation, and export retain this object. They must not reconstruct its Book.

Preview reuse and invalidation

Changes to inclusion, role, order, root, cleaning inputs, title/author variables, title page, table of contents, scene separator, semantic headings, or formatting that changes bytes invalidate preparation. Selected format and filename do not affect manuscript interpretation and therefore do not rebuild the Book.

Before export, ExportCoordinator recalculates:

• the source fingerprint from included file contents;
• the input signature from semantic request fields and plan.

A mismatch blocks export and instructs the author to refresh preview. Source hashing detects equal-size edits even when file timestamps do not change.

SemanticDocument export projection

SemanticDocument is a format-oriented projection of the prepared Book. It provides ordered sections and blocks:

• title/author;
• front/back matter headings;
• Part/Chapter number and title headings;
• optional body headings;
• first paragraphs;
• later body paragraphs;
• scene and page breaks;
• inline bold/italic/link semantics.

The projection centralises presentation-relevant classification so exporters do not duplicate structural interpretation. It does not scan, parse, clean, reorder, or replace Book.

Paragraph state is semantic and presentation-independent. The first paragraph after a structural/body heading or scene break is marked first; later ordinary prose is Body Text. The indentation toggle only changes style declarations.

Export pipeline

ExportCoordinator owns the terminal transaction:

1. acquire the global export operation;
2. verify prepared source/input fingerprints;
3. apply format-independent blocking policy;
4. persist current safe defaults;
5. create one SemanticDocument from session.book;
6. generate bytes using EXPORTERS[format];
7. validate using EXPORT_VALIDATORS[format];
8. enter non-cancellable finalisation;
9. start one browser download;
10. record success only if dispatch started;
11. release operation state in finally.

Exporters do not own UI, settings persistence, validation policy, delivery, or history. Validators do not repair bytes. Download service does not know semantic content.

Export contracts

ExportFormat is the exhaustive union:

docx | odt | epub | html | markdown | xml

ManuscriptExportContext contains the prepared session, shared SemanticDocument, resolved formatting, and portable filename. GeneratedExport contains format, filename, MIME type, bytes, and non-prose warnings.

Adding a format requires updating both exhaustive registries and all UI/testing/documentation/package surfaces. A registry gap is a compile-time error.

Export formats

DOCX

docx.ts produces native WordprocessingML using named styles and fflate. Styles explicitly encode title/matter/Part/Chapter weight, First Paragraph versus Body Text indentation, scene separators, page size, chapter page starts, and optional TOC field. It never converts another format. docx-validator.ts verifies essential ZIP and OOXML structure; it does not claim complete OOXML conformance.

ODT

OdtExporter produces a native OpenDocument Text ZIP. The mimetype entry is first and uncompressed. Generated content references named paragraph styles in styles.xml; heading styles explicitly declare bold and body styles explicitly declare indentation. Validation checks package entries, mimetype ordering/storage, manifest targets, XML shape, and required styles.

EPUB

EpubExporter produces an EPUB 3 ZIP with container, package document, navigation, ordered XHTML sections, and embedded stylesheet. The mimetype entry is first/uncompressed. XHTML uses semantic manuscript classes; CSS explicitly styles structural headings and body indentation without scripts, remote assets, or inline styles.

HTML

HtmlExporter produces one offline HTML5 document with embedded CSS and semantic manuscript classes. It contains no JavaScript or remote reference. Presentation uses manuscript-specific selectors rather than global heading-weight rules.

Markdown

MarkdownExporter renders deterministic UTF-8 Markdown from SemanticDocument. It preserves heading syntax, emphasis, readable link text, Unicode, paragraph spacing, and exactly one final newline. It does not fake first-line indentation with spaces, tabs, HTML, entities, or CSS.

XML

XmlExporter produces deterministic semantic XML in https://manuscript-compiler.dev/schema with schemaVersion="1.0". It contains metadata, matter, Part/Chapter/Scene hierarchy, paragraph elements, and inline emphasis/link spans. It excludes source paths, settings, profile IDs, diagnostics, and presentation styling.

Browser download

BrowserDownloadService is the only completed-export delivery path. It creates a Blob and object URL, assigns URL/filename as anchor properties, appends and clicks one temporary <a download>, removes it, and revokes the URL on success or failure.

This boundary exists because:

• Obsidian Community Plugins must remain mobile-safe;
• the host owns save/share behavior;
• Node/Electron filesystem paths are unavailable or non-compliant;
• the plugin must not write manuscript exports into the vault;
• final external persistence cannot be observed truthfully.

Dispatch success means the host accepted the click. It does not mean a known filesystem path was written.

Validation

The registry in export-validators.ts mirrors EXPORTERS. Validators perform restrained structural checks appropriate to each format:

• DOCX: required package/XML parts;
• ODT: entries, mimetype, manifest, XML, required styles;
• EPUB: entries, mimetype, container/package/navigation/spine targets and XHTML;
• HTML: UTF-8 document structure and manuscript content;
• Markdown: UTF-8, semantic required content, forbidden content, deterministic equivalence, final newline;
• XML: namespace/schema, structure, escaping, and presentation/privacy exclusions.

Validation failure blocks download. Automated structural validation does not replace Word/Vellum/LibreOffice/reader/browser interoperability checks.

Operation state and cancellation

OperationStateController permits one active global operation and defines these states:

idle → preparing → ready
idle/ready → exporting → finalising → complete
preparing/exporting → cancelled
preparing/exporting/finalising → failed

AbortController supplies cancellation before finalisation. CompilationCancelledError lets callers distinguish cancellation from failure. Finalisation locks cancellation because the download click cannot be rolled back or observed reliably.

The workspace controller deduplicates preparation/export promises, cancels preparation when appropriate, and releases state in terminal paths. A partially prepared Book is never published.

Settings and presets

settings.ts defines persisted schema/defaults. simple-workflow.ts resolves guided choices into a complete compatibility profile. Vellum and Standard Manuscript are deterministic formatting presets; Custom preserves manual values.

main.ts loads raw settings, then calls repairSettings before services or UI consume them. UI components receive repaired data only.

Historical fields may remain for backward-compatible loading. They are storage-only if their feature was removed. In particular, legacy vault-output or external-converter fields cannot activate production behavior.

Migration and repair

migrateSettings translates known historical shapes. repairSettings validates and bounds untrusted persisted data, nested profiles, history, and logs. Both are idempotent and preserve explicit user choices, including false/zero values and custom formatting.

Migration rules:

• missing fields receive behavior-preserving defaults;
• legacy units convert once without drift;
• profile IDs and bounded nested structures are repaired;
• removed fields are never revived;
• a second migration/repair pass makes no changes.

History and diagnostics

CompileHistoryService records bounded terminal facts only after truthful outcomes. Success follows validation and started download dispatch. Cancellation and failure are distinct. Records can include format, portable filename, counts, timings, validation status, and download-started flag.

history-storage.ts whitelists persisted fields and repairs malformed arrays. DiagnosticsReportGenerator produces a redacted support summary without reading notes. Neither history nor diagnostics may retain manuscript prose, metadata values, warning details, absolute paths, usernames, Blob URLs, or external destinations.

Obsidian integration

main.ts is the composition root. It:

• loads and repairs settings;
• constructs history, operation, preparation/export, and command services;
• registers stable command IDs;
• registers the documented File Explorer file-menu event;
• registers settings UI;
• owns onboarding and unload cancellation.

The selected folder action accepts only TFolder. UI uses Obsidian Modal, Setting, Notice, suggestion, icon, and component APIs. No global app is used. CSS is scoped to plugin roots and retains theme variables, focus-visible behavior, reduced-motion, high-contrast, narrow-pane, and mobile behavior.

Security, privacy, and dependency policy

Production guarantees:

• offline execution;
• no telemetry/accounts/cloud/network;
• no remote resources;
• no Electron or Node filesystem;
• no shell, child_process, or external executable;
• no other community-plugin dependency;
• no manuscript vault output;
• no unsafe HTML injection;
• deterministic escaping for XML/HTML/OOXML;
• controlled ZIP entry paths;
• fflate as the sole runtime dependency.

Dependency additions require product approval, security/licence/maintenance/bundle/mobile analysis, lockfile review, updated notices, clean tests, and clean audit. See SECURITY.md.

Performance model

Preparation reads notes with bounded concurrency, builds one Book, calculates shared statistics/warnings, and hashes included content. Exporters reuse one SemanticDocument and do not repeat preparation.

The benchmark constructs 500 Chapters, 2,000 Scenes, and 2 million words, then runs all six formats with indentation enabled and disabled from the same Book. Results are informational; there is no strict machine-specific threshold. Maintainers should investigate repeated scanning, quadratic transforms, or disproportionate memory growth.

Testing strategy

Suite	Primary contract
npm test	detection, parsing, cleaning, migration, workspace state, privacy, accessibility/compliance source guards
test:docx	native DOCX package, semantic Word styles, Vellum-oriented regression
format suites	complete generated package/markup and matching validator behavior
test:exports	exhaustive registries, all formats, delivery, forbidden content, deterministic output
benchmark:large	one preparation, all exporters, runaway regression guard
package validation	exact release allowlist and version identity

Fixtures use synthetic content. Generated manual-inspection artifacts are ignored and excluded from packages. Live application checks remain unchecked in MANUAL_TESTING.md until actually performed.

Directory layout

src/                         production TypeScript
src/workspace/               three-stage workspace state, views, and projections
tests/                       regression, integration, fixture, export, benchmark code
tests/fixtures/              privacy-safe representative vault layouts
tests/golden/                deterministic Markdown expectations
samples/                     author-facing example projects
scripts/package.mjs          release allowlist creation/validation
.github/workflows/ci.yml     continuous integration
styles.css                   scoped plugin presentation
manifest.json                immutable plugin identity/version contract
USER_GUIDE.md                author documentation and screenshot plan
DEVELOPER_GUIDE.md           maintenance and extension procedures
SECURITY.md                  security guarantees and reporting
MANUAL_TESTING.md            intentionally unchecked interoperability gates
RELEASE_READINESS.md         automated/manual release gates

Source map

Composition and commands

• main.ts — Obsidian lifecycle and dependency composition.
• commands.ts — stable command IDs.
• compile-command-service.ts — command route orchestration.
• folder-context-menu.ts — exact-folder File Explorer entry.
• book-root-resolver.ts — root resolution policy.

Preparation and semantic construction

• compile-preparation.ts — sole root-to-prepared-session boundary.
• vault-scanner.ts, types.ts — physical discovery.
• content-plan.ts — detection, correction authority, scan rewriting.
• parser.ts — semantic Book construction.
• filters.ts, metadata-filter.ts — cleaning/filtering.
• ordering.ts, statistics.ts, warnings.ts — deterministic derived semantics.
• model.ts — semantic Book and result contracts.
• compiler.ts — parser/statistics/warnings compatibility facade.

Export

• semantic-document.ts — shared export projection.
• export-types.ts — format contracts and metadata.
• native-exporters.ts, markdown-exporter.ts, docx.ts — native byte generators.
• export-validators.ts, docx-validator.ts — pre-delivery validation.
• export-coordinator.ts — verified terminal transaction.
• browser-download.ts — Blob/object-URL delivery.
• export-filename.ts, export-safety.ts, measurements.ts — shared policies.

Workspace and UI

• compile-modal.ts — modal shell and step composition.
• workspace/compile-workspace-controller.ts — state, invalidation, cancellation, dispatch.
• workspace/*-step.ts — DOM rendering only.
• workspace/content-tree.ts — pure plan mutations/projections.
• workspace/contents-tree-view-state.ts — focus/scroll/collapse state.
• workspace/*view-model.ts, export-preview.ts — pure display projections.
• ui.ts, wizards.ts — settings, onboarding, secondary/legacy modals.

Persistence and support

• settings.ts, profiles.ts, simple-workflow.ts — schema, migration/repair, preset resolution.
• compile-history.ts, history-storage.ts — bounded privacy-safe records.
• diagnostics.ts — redacted support report.
• operation-state.ts, cancellation.ts — shared operation vocabulary.
• validation.ts — read-only session validation.

Extension points

Permitted extension points preserve boundaries:

• add a detection rule in content-plan.ts with representative fixtures;
• add a pure structured cleaning rule with prose-preservation tests;
• add a shared SemanticDocument presentation mapping;
• add a validator invariant for a verified malformed-output case;
• add a pure workspace view-model projection;
• add privacy-safe fixtures and interoperability documentation.

New formats, semantic Book fields, workflow stages, dependencies, delivery routes, or persistent data are product changes requiring explicit architecture review.

Things never to break

1. Preview and every exporter use the same prepared Book.
2. Exporters never read the vault or parse notes.
3. The selected root and transparent containers never become accidental headings.
4. Explicit author inclusion, roles, and order beat inference.
5. Part 0 and Chapter 0 are never invented.
6. First paragraphs after headings/scene breaks remain semantically distinct.
7. Validation occurs before delivery.
8. One successful action initiates exactly one download.
9. No manuscript export is written into the vault.
10. Cancellation and failure are recorded truthfully.
11. Histories/diagnostics exclude prose and private paths/metadata.
12. Output remains deterministic, escaped, offline, and dependency-restrained.
13. Command IDs, manifest identity, and mobile compatibility remain stable.
14. Release packages contain exactly the four allowlisted files: the three Obsidian runtime assets and the standalone logo.

Future roadmap

The product is feature complete. The roadmap is maintenance rather than expansion:

• keep pace with documented Obsidian APIs and official lint requirements;
• record manual interoperability across current Word, Vellum, LibreOffice, EPUB readers, browsers, Markdown renderers, and XML tools;
• add fixtures for verified real-world detection or cleaning defects;
• improve accessibility from measured/manual findings;
• strengthen validators only when real malformed output demonstrates a missing invariant;
• profile and optimise only verified large-manuscript regressions;
• review dependencies and security advisories regularly.

Any future feature proposal must first show how it preserves the semantic Book, prepared identity, exhaustive export/validation contract, offline privacy model, browser delivery, mobile safety, and dependency policy.