wiseguru_ReWrite-Voice-Notes/docs/FEATURES.md

ReWrite — Features To Add

Forward-looking backlog. Not committed to a release. Add items as they come up; move to a phase plan when picking one up.

Open

1. Plugin-managed local whisper.cpp server (desktop) — Phase B (auto-start lifecycle)

Phase A shipped (see Done). Remaining work:

• "Start automatically when Obsidian opens" toggle (default off).
• "Stop when idle for N minutes" toggle (default off; useful for the large-v3 user who doesn't want 1.5 GB resident all day).
• Process supervision hardening based on real-world Phase A usage (zombies, orphans, signal handling differences across platforms).

2. Mobile API management UX

Editing keys, base URLs, and model strings on mobile is rough. Two concrete issues plus one visual ask:

• The two profile sections in src/settings/tab.ts render identically, so it's easy to edit the desktop profile while sitting on the phone. Add a visual differentiator (color accent, badge, or border on the active-on-this-device profile) so the mobile vs desktop section is unmistakable.
• The settings tab re-renders the entire container on dropdown changes (provider, insertMode, activeProfileOverride per CLAUDE.md's Gotchas), which on mobile causes a visible scroll jump after committing a text field that sits next to a dropdown. Text fields already skip the re-render; audit whether any newer fields slipped into the redraw path and consider scrolling the focused row back into view after redraw.
• On mobile the on-screen keyboard frequently covers the input that just received focus. Likely needs a scrollIntoView({ block: 'center' }) on the active control's focus/blur events, or a keyboardWillShow-equivalent hook through Obsidian's API.

3. (Bug) Voxtral transcription model dropdown is empty

src/transcription/mistral-voxtral.ts does not implement listModels, so the Refresh button on the model field is a no-op for the Voxtral transcription provider. Mistral's /v1/models returns the full catalog including chat models; filter to audio/Voxtral capability (probably id.includes('voxtral') for a v1) when surfacing transcription model IDs. Cross-check: the LLM-side Mistral provider already uses the shared OpenAI adapter's listing, so this gap is transcription-only.

4. Real-time transcription mode (live STT, no cleanup)

Voxtral exposes a real-time STT model that doesn't accept whole-file uploads, and a few other providers have equivalents (Deepgram streaming, AssemblyAI realtime). Investigate adding an opt-in "Real-time" mode:

• Bound to a command/shortcut only (no main-modal entry, no template, no LLM cleanup pass).
• Inserts the live transcript at the cursor as it streams, on both mobile and desktop.
• Provider-side gating: only enable for transcription providers that expose a realtime endpoint; document which.
• Honest assessment: this is lower-value than the rest of the plugin (transcript with no cleanup is what Obsidian's existing voice plugins already do). Ship only if users ask.

5. (Bug) Web Speech repeats short phrases indefinitely

The Web Speech adapter in src/webspeech.ts currently loops short utterances: the same phrase is repeated over and over instead of producing one transcript line. Likely cause is the interim-result accumulator concatenating without deduping against final results, or the restart-on-end logic re-emitting buffered interim text. Needs investigation. Until fixed, document Web Speech as a "do not use" option in the README (or remove it from the provider dropdown).

6. Daily-note template: how far can the prompt drive a structured fill?

The current daily-note default template is a freeform cleanup. Investigate whether the system prompt can reliably lay the transcript into a real structured daily-note template with named sections (Mood / Highlights / Tasks / Tomorrow / etc.) rather than emitting prose. References:

• https://dannb.org/blog/2022/obsidian-daily-note-template/
• https://www.craft.do/templates/category/daily-notes

Open questions: do we ship this as an updated default template (prompt-only), or extend NoteTemplate to carry a target-template path the LLM should fill into? If the latter, that's a real schema change and needs design work first. Either way, validate on at least one of the linked templates before committing to a format.

---

Done

Mode selector on the Quick Record floater

The Quick Record floating mini-UI (src/ui/quick-record.ts) gained a third button between the timer and Stop that opens a popover-style list of plugin.templates. Selecting an entry updates QuickRecordController.template in place and the button label so the user can see which template will run. Switching does NOT update lastUsedTemplateId; that still only happens after a successful completion. The popover is a child of the floater div, dismisses on selection / Escape / outside-click (capture-phase listeners on document, registered and torn down per open), and is force-closed when setBusy runs so the user can't switch templates after Stop is pressed. CSS additions in styles.css: .rewrite-quick-template (the button), .rewrite-quick-popover (the floating list), .rewrite-quick-popover-item (rows, with an is-active modifier for the current selection). No destination-override UI on the floater per the v1 scope decision.

Per-invocation destination override on the main modal

The ReWrite modal (src/ui/modal.ts) renders a "Destination" row under the template picker exposing Insert mode (cursor / new file / append) and, conditionally, folder + filename-template inputs for newFile mode. The selection defaults to the active template's values and only becomes a non-null DestinationOverride once the user touches any field. A "Reset to template default" button clears it back to null. Switching the template selector resets the override (the new template's values become the new defaults). The override is plumbed through PipelineParams.destinationOverride and applied by applyDestinationOverride in src/pipeline.ts, which spreads the override onto a copy of the template before calling insertOutput — the template object passed in is never mutated, and the underlying Markdown file on disk is never touched. Applies to all three modal tabs (Record / Paste / From note) because they all converge on the modal's single execute() call site. Not exposed in Quick Record / runTextPipeline / runAudioFilePipeline to keep those paths frictionless; if a user asks for it later we can add a destination row to the template picker modal.

Move the assistant prompt into the vault; standardize branding to "Assistant"

The system-prompt preface that frames extracted ad-hoc directives moved from a settings textarea (GlobalSettings.adHocInstructionsPrompt) to a Markdown file in the vault (GlobalSettings.assistantPromptPath, default ReWrite/AssistantPrompt.md). src/assistant-prompt.ts exports loadAssistantPromptFromFile, populateDefaultAssistantPrompt, isPathAssistantPrompt, and DEFAULT_ASSISTANT_PROMPT. The loader strips any leading frontmatter (tolerated for future extensions but not consumed in v1) and returns the trimmed body. Cache lives at plugin.assistantPrompt: string | null, refreshed in src/main.ts on workspace.onLayoutReady, scoped vault create/modify/delete/rename events, the path-field change, and the Populate button. src/pipeline.ts reads params.host.assistantPrompt ?? DEFAULT_ASSISTANT_PROMPT inside cleanupTranscript, so a missing or empty file falls back to the previous hardcoded preface.

Branding standardized to "Assistant" everywhere the old "AI" / "Agent" labels appeared. GlobalSettings.aiName → assistantName; "AI name" label → "Assistant name"; "Agent control prompt" textarea → "Assistant prompt file" path field. GlobalSettings.adHocInstructionsPrompt is gone (the file is the source of truth). The wake-name term is preserved separately because it names the trigger, not the persona. Pre-release rule applies: no migration code, no compatibility reads — dev installs reset via deleting data.json. The settings tab's "Ad-hoc instructions" section heading stays (the feature is ad-hoc-instruction extraction; "Assistant" is the persona inside it).

Known nouns list in the vault

A new vault file at GlobalSettings.knownNounsPath (default ReWrite/KnownNouns.md) holds proper nouns the LLM should preserve verbatim, with optional misheard alternates. File format: YAML frontmatter for human-readable guidance (token-cost warning, format hint) plus a Markdown body with one noun per line, optional canonical: alt1, alt2 for misheard variants. Lines starting with # and blank lines are skipped. Frontmatter is parsed but NOT sent to the LLM in v1; the structure leaves a future opt-in possible. The default file body (created by the Populate button) includes both the guidance frontmatter and two example nouns the user can keep or delete.

src/known-nouns.ts exports loadKnownNounsFromFile, populateDefaultKnownNouns, isPathKnownNouns, and buildKnownNounsSystemPromptSection(nouns). Cache on plugin.knownNouns: KnownNoun[], refreshed on the same triggers as the assistant prompt. src/pipeline.ts appends the section returned by buildKnownNounsSystemPromptSection to the system prompt only when non-empty; order is template prompt → ad-hoc instructions → known nouns. The settings tab has a dedicated "Known nouns" section with the path field, a Populate button, an Open-file button, and a reminder that an unbounded list inflates token cost on every recording. The default file body carries the same warning so users see it the first time they open the file.

Side change: PipelineParams gained a host: PipelineHost field (narrow interface in src/types.ts: { assistantPrompt; knownNouns }). The plugin class implements PipelineHost; the pipeline never imports ReWritePlugin directly (would form a cycle through the UI layer). All four call sites (modal, Quick Record, text-source, audio-source) updated to pass host: plugin.

Reprocess a saved recording

Added src/ui/audio-source.ts (isAudioFile, collectAudioFiles, readAudioFileAsBlob, runAudioFilePipeline) and src/ui/audio-file-picker.ts (a FuzzySuggestModal<TFile> over the vault's audio files). Extended audio PipelineSource with an optional sourcePath: string; when set, src/pipeline.ts skips the persist-audio stage entirely and routes the existing path straight to the ![[<path>]]\n\n prepend, so reprocessed output keeps the audio link without writing a duplicate file. Three entry points: a rewrite-plugin:reprocess-audio command opens the fuzzy file picker, an editor-menu item appears when the cursor sits inside an ![[<audio>]] embed (resolved via app.metadataCache.getFirstLinkpathDest), and a file-menu item appears for audio files in the file explorer. All three open the existing TemplatePickerModal and call runAudioFilePipeline, which gates on the full voice profile (isProfileConfigured from src/ui/setup-card.ts) and falls back to opening the main modal if either side is unconfigured. Mime type for the reread blob is derived from the file extension via the new extensionToMime helper in src/audio-persist.ts (reverse of mimeToExtension; also exports an AUDIO_EXTENSIONS constant the picker and file-menu both filter against).

Custom prompt for ad-hoc instruction handling

Replaced the hardcoded preface line in the ## Ad-hoc instructions block at src/pipeline.ts with a user-editable string. Originally landed as a settings textarea (GlobalSettings.adHocInstructionsPrompt); later moved into a vault Markdown file (GlobalSettings.assistantPromptPath) and rebranded under "Assistant" — see "Move the assistant prompt into the vault" above. Both LLM adapters route systemPrompt into the API system slot, so neither change required provider edits.

Adopt orphaned whisper-server processes; surface external ones

Closing Obsidian doesn't reliably kill spawned whisper-server children on any platform (Linux reparents to init, Windows orphans, macOS SIGTERM may not land before exit). Previously the new plugin instance had no PID handle, so start() would hit "port in use", stop() was a no-op, and transcription would fail because src/transcription/whisper-local.ts artificially required host.status() === 'running' even though the HTTP API doesn't care who owns the process.

Reworked into a three-state model: 'spawned' (live child handle this session, log capture works), 'adopted' (port bound, PID sidecar matches a live PID — we started it last session, no log capture), 'external' (port bound but no sidecar match — someone else started it). Status enum gains 'external'. WhisperHost.start() writes a sidecar at <plugin folder>/whisper-host.pid.json once the server is ready; stop() and the child exit handler clear it. WhisperHost.probe(config) (called from onload and at the top of start()) does the discovery dance; never disturbs state when we already hold a live spawned child.

Transcription drops the status() check entirely and just asks host.baseUrl(), which returns the URL for both 'running' and 'external' so transcription Just Works regardless of ownership. Stop semantics: spawned uses the ChildProcess handle, adopted uses process.kill(pid, signal), external throws "not started by ReWrite — stop it via OS tools." (settings tab button is disabled with a tooltip, status-bar click shows a Notice, stop-whisper-host command is hidden). Status bar/settings labels read "Running on http://... (adopted from previous session, pid N).", "External whisper-server on http://... (not started by ReWrite).", etc. New is-external dot color (orange) in styles.css. On onload, a Notice fires when we adopt ("Adopted whisper-server from previous session (pid N)") or detect external ("Detected external whisper-server on URL. Transcription will use it; ReWrite won't stop it."). Preserves the "never kill a process we didn't start" invariant — the sidecar is proof we started it.

Start/stop local whisper.cpp server from outside settings

Surfaced a status-bar item (src/ui/whisper-status-bar.ts) showing a colored dot plus "Whisper: " label. Click toggles start/stop, hover tooltip matches the settings-tab status string (extracted to formatWhisperStatus in src/whisper-host.ts so both surfaces share one implementation). The item polls whisperHost.status() every 1 s via registerInterval and hides via the rewrite-hidden CSS class when on mobile or when the active profile's transcription provider is not whisper-local. Added two command-palette entries (start-whisper-host / stop-whisper-host) using checkCallback so they only appear when actionable on the current platform and host state. Same error-surfacing pattern as the settings tab (Notice with the error message). Did not add a ribbon variant or modal-header indicator, per the FEATURES.md "pick one" guidance.

Persist recorded audio to an attachments folder

Added src/audio-persist.ts with mimeToExtension, buildAudioFilename, and persistAudio. The pipeline (src/pipeline.ts) gained a new 'persist-audio' stage that runs BEFORE transcription on audio sources, so the user keeps the recording even if transcription fails. Filename is ReWrite-YYYY-MM-DD-HHmmss.<ext> with the extension derived from the blob's mime type (webm / m4a / ogg / wav / mp3, default webm). Path resolution: when the new GlobalSettings.attachmentsFolderPath setting is non-empty, the file lands under that folder with manual de-collision (-1, -2, ...); when empty, the path comes from app.fileManager.getAvailablePathForAttachment(filename), honoring Obsidian's own attachments setting. An ![[<path>]]\n\n embed is unconditionally prepended to the cleaned output when audio was persisted, regardless of insert mode. Persistence failure is non-fatal: a Notice fires and transcription proceeds. Cancel paths (modal, Quick Record) call recorder.cancel() before runPipeline(), so no orphan files are written on cancel. UI stage label "Saving audio..." added to both modal and Quick Record floater. Setting field lives under "Recording" in the settings tab.

Address the assistant directly mid-recording with a wake name (default "Scrivener")

Added src/wake-name.ts exposing extractAdHocInstructions(transcript, name). Regex pattern requires the vocative comma form (<name>,), captures up to a sentence terminator or the next name occurrence, drops filler tokens ("never mind", "scratch that", under 2 chars), and post-processes whitespace. Wired into cleanupTranscript in src/pipeline.ts: when GlobalSettings.adHocInstructionsEnabled is on and GlobalSettings.assistantName is non-empty, the transcript is scanned BEFORE the LLM call. Extracted instructions are stripped from the transcript and appended to the template prompt as a numbered ## Ad-hoc instructions block; both OpenAI and Anthropic adapters route this into the API system slot. A Notice ("Heard N ad-hoc instruction(s).") confirms detection to the user. Applies to all four PipelineSource variants (audio / paste / webspeech / text) since the scan runs after collectTranscript(). Off by default. Global setting (not per-template). v1 is exact-match regex; fuzzy/Levenshtein matching for whisper homophones deferred. Settings UI is a heading + Enabled toggle + (conditional) Assistant name field, re-rendered on toggle change. Originally landed under the label "AI name"; rebranded to "Assistant name" alongside the assistant-prompt-file change.

Templates as Markdown files in a vault folder

Removed the in-settings template editor (drag-to-reorder list, per-template editor, Add/Delete buttons, ConfirmModal for deletion). GlobalSettings.templates is gone; in its place is a templatesFolderPath string (default ReWrite/Templates). Each .md file in that folder is one template: YAML frontmatter holds id / name / insertMode / newFileFolder / newFileNameTemplate, the body is the LLM prompt. src/templates-folder.ts handles loading (sorted by basename so users can prefix 01-, 02- to order), and populateDefaultTemplates(app, folderPath) writes the five built-in defaults into the folder, skipping any whose id already exists on disk so the button is safe to re-run. plugin.templates is the cache, refreshed in src/main.ts on workspace.onLayoutReady and via vault create / modify / delete / rename events scoped to the templates folder. defaultTemplateId stayed as a settings field (now a dropdown of loaded templates) and references the frontmatter id, so renames don't break it. Used parseYaml / stringifyYaml from Obsidian rather than the metadata cache (which is async and doesn't populate immediately after app.vault.create).

Remove the "Advanced" disclosure around whisper-host extra args

Unwrapped the <details> that hid the extraArgs field in src/settings/tab.ts' renderLocalWhisperServer. The field now renders inline next to Binary path / Model path / Port. No setting-shape change, just one click less to reach a useful knob.

Plugin-managed local whisper.cpp server (Phase A: on-demand, button-driven)

Added src/whisper-host.ts WhisperHost class with start / stop / status / baseUrl / getLog. Lazy-requires child_process, net, fs inside a Platform.isDesktop guard (mirrors the src/secrets.ts safeStorage pattern). start() validates binary and model paths via fs.existsSync, probes the port via net.createServer().listen(port) to detect conflicts (does NOT kill unknown bound processes), spawns whisper-server, captures stdout/stderr to a 1 MB ring buffer, polls net.createConnection every 250ms for up to 5s before declaring 'running'. stop() sends SIGTERM with a 3s SIGKILL fallback. New settings section "Local whisper.cpp server (desktop)" with binary/model/port fields, an Advanced disclosure for extra args, status indicator with Start/Stop button, and a View log disclosure (last ~50k chars of the ring buffer). New transcription provider option 'whisper-local' filtered out of dropdowns on mobile; thin shim in src/transcription/whisper-local.ts POSTs to http://127.0.0.1:<port>/v1/audio/transcriptions (OpenAI-shaped). No API key field for this provider. src/main.ts instantiates the host in onload and fire-and-forget stops it in onunload. README has setup walkthrough + transparency disclosure. Phase B (auto-start lifecycle + idle timeout) deferred.

Model field as a dropdown of available models

Added an optional listModels(config, signal) method to both TranscriptionProvider and LLMProvider interfaces. Implementations: OpenAI / Groq / Mistral (shared via the OpenAI-shaped adapter), Anthropic, Gemini, Deepgram. Skipped (text-only fallback): openai-compatible (URL-specific, list-shape varies), AssemblyAI, Rev.ai, Web Speech. Settings tab renders a hybrid Setting per model: dropdown of cached models + Refresh button (when the provider supports listing) plus an always-canonical text input that wins for "Custom" model names not yet in the dropdown. Cache lives at GlobalSettings.modelCache.{transcription,llm}[providerId] = { ids, fetchedAt }. No auto-fetch on settings open; the user clicks Refresh once their key is set. Errors surface via Notice with the provider-attributed ProviderError message.

Act on existing text in a note

Added a { kind: 'text'; text: string } source variant to the pipeline (skips transcription, same path as paste). Three entry points: rewrite-plugin:process-text command, an editor-menu item "ReWrite with template...", and a new "From note" tab in the main modal. All three resolve text from the active editor (selection if non-empty, otherwise whole note body) and open a lightweight template quick-picker before running. Setup-card gating split into voice (full check) vs text (LLM-only check) via isProfileConfiguredForText and a purpose param on renderSetupCard; the modal's per-tab rendering picks the right gate. Helpers live in src/ui/text-source.ts (resolution + runTextPipeline) and src/ui/template-picker.ts (the quick-picker modal). Shipped without the per-invocation "Replace source" checkbox; can revisit if users ask.

Collapse API key settings: per-profile only, hide rarely-changed fields under "Advanced"

Removed the "Global API keys" section and the by-family apiKeys map. Each profile now owns its own transcription and LLM keys directly on transcriptionConfig.apiKey / llmConfig.apiKey. The per-profile fields are no longer framed as "overrides"; they're the only slot. "Transcription language" and "LLM max tokens" moved into a per-profile <details> "Advanced" disclosure. The resolver functions (resolveTranscriptionApiKey, resolveLLMApiKey) and family helpers (transcriptionProviderFamily, llmProviderFamily) are gone, along with the ProviderFamily type. secrets.json.nosync now only contains profile:{kind}:{side} IDs.