fix(perplexityService, directoryTemplateService): CORS bypass — Node.js https replaces activeWindow.fetch for all Perplexity streaming paths

Perplexity's API stopped including Access-Control-Allow-Origin headers for app://obsidian.md, so Electron's Chromium renderer began blocking every streaming response body — even on successful 200 OK requests. Every Perplexity query that used streaming was silently producing nothing. Both streaming call sites now use https.request from node:https instead of activeWindow.fetch. Node.js routes through the OS network stack and has no CORS constraint at all. The resulting IncomingMessage stream is wrapped in a Web ReadableStream<Uint8Array> and passed to the existing SSE parsing machinery unchanged — idle-timeout, chunk accumulation, citations, images, and AbortController signal handling all preserved. Non-streaming requests (Obsidian's request() function) were never affected and are untouched. perplexityService.ts: - Added import * as https from 'node:https' + import type { IncomingMessage } - Replaced activeWindow.fetch in queryPerplexity streaming branch with https.request → IncomingMessage → ReadableStream<Uint8Array> bridge directoryTemplateService.ts: - Same imports - Replaced activeWindow.fetch in streamPerplexityToFile with same bridge - AbortController.signal threaded into https.request signal option so ceiling-timer and user-cancel abort behavior is fully preserved Bumps to 0.3.1 — manifest.json, package.json, versions.json updated. Files changed: - src/services/perplexityService.ts - src/services/directoryTemplateService.ts - manifest.json - package.json - versions.json - changelog/2026-07-06_01.md - changelog/releases/0.3.1.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
feat(release): 0.3.0 — three analyst-grade templates for VC/PE/equities + four infrastructure fixes for deep-research reliability
2026-07-22 06:49:50 +00:00 · 2026-07-06 14:41:12 -05:00 · 2026-05-27 17:41:13 -05:00 · 2026-05-26 17:08:43 -05:00 · 2026-05-22 15:29:10 -05:00 · 2026-05-22 15:28:53 -05:00
29 changed files with 3284 additions and 132 deletions
--- a/README.md
+++ b/README.md
@ -3,6 +3,20 @@ # Perplexed: AI Content Generation for Obsidian
 **Perplexed** is an Obsidian plugin that enables AI-powered content generation with source citations using [Perplexity](https://www.perplexity.ai/), [Anthropic Claude](https://www.anthropic.com/), [Google Gemini](https://ai.google.dev/) (with Google Search grounding), and [Perplexica / Vane](https://github.com/ItzCrazyKns/Vane) (self-hosted). This plugin brings research-grade AI capabilities directly into your Obsidian workspace, allowing you to generate well-cited content for your notes.
 ## 💼 For Venture Capital, Private Equity, and Equities-Trading Workflows
 Perplexed ships a set of **analyst-grade directory templates** aimed at the research deliverables a VC analyst, PE associate, equity-research analyst, or trading-desk strategist produces daily. Drop an empty file into the matching folder, run *Apply directory template to current file*, and Perplexity Deep Research returns a 6-9K-word cited analyst draft you can curate into a memo for a partner, an IC, or a portfolio review.
 | Workflow | Template | What it produces |
 |---|---|---|
 | **Naming the players in a category** (incumbents vs challengers vs innovators by financial stage) | `market-category-profile.md` → `concepts/Market-Categories/` | Three-tier company landscape with explicit definitions: **Incumbents** (public / late-stage private / PE-owned), **Challengers** (Series C+ scale-ups, recently public), **Innovators** (Pre-Seed through Series B). Plus Why Now / What's Happening sections covering CAGR + category-creation momentum, and an Industry Coverage section sub-grouped into Market Reports (Gartner, IDC, Forrester, ABI) / Industry Articles / Financial News (Bloomberg, FT, Pitchbook). |
 | **Authoring a market map** (Known Category or Thesis-Driven) | `market-map-profile.md` → `lost-in-public/market-maps/` | Analyst-grade memo with 4-8 sub-segments, 20-40 named innovator cards (Offering / Funding / Why-they-matter / Coverage), Market Dynamics (Sizing / Adoption / Capital Flow), Frontier and Open Questions. Anti-incumbent editorial stance prevents big-tech over-representation. |
 | **Profiling an open spec or standard** an investment thesis depends on | `standards-and-specs-profile.md` → `Sources/Standards-and-Specs/` | Five-way authority typing (de-jure / consortium / vendor-led-open / community / de-facto), three-tier adoption framing with notable holdouts, named editors, stewardship-transition stories, named public critics with their arguments. |
 | **Catalogue an authoritative source** (book, person, channel, report, conference) | `source-profile.md` → `Sources/` | Type-aware emphasis (author / publisher / cadence / methodology), Google Books URL harvesting for books, signature-work catalog. |
 | **Encyclopedia entry on a concept, pattern, or mental model** the desk repeatedly invokes | `concept-profile.md` → `concepts/` | Definition, usage, history, examples, case studies. Mermaid + LaTeX rendering discipline baked in for diagrams and formulas. |
 Every analyst-grade template runs on `sonar-deep-research`, ships with idle-only timeout safety (`request-timeout-ms: 0`, per-chunk idle timer at 270s) and a 24,000-token output budget (`max-tokens: 24000`) — enough for the longest analyst drafts to land without silent mid-document truncation. See [Directory Templates](#directory-templates) below for the full set and the cft-block grammar for tuning your own.
 ## 🎯 Key Features
 ![Perplexed UI Modal interface](https://i.imgur.com/jaZ4UfS.png)
 - **Source-Cited AI Responses**: Get AI-generated content with proper citations and references
@ -50,6 +64,7 @@ ## Network use and accounts
 ## 📋 Table of Contents
 - [For Venture Capital, Private Equity, and Equities-Trading Workflows](#-for-venture-capital-private-equity-and-equities-trading-workflows)
 - [User Onboarding](#user-onboarding)
  - [Installation](#installation)
  - [Initial Setup](#initial-setup)
@ -435,9 +450,10 @@ ### Partials and preambles — shared guidance across templates
 ```
 Content-Dev/
-├── templates/         (your four profile templates)
+├── templates/         (your seven profile templates)
-├── partials/          (reusable snippets: mermaid-discipline, etc.)
+├── partials/          (reusable snippets included via {{include: name}})
-│   └── mermaid-discipline.md
+│   ├── mermaid-discipline.md      (paired BAD/GOOD examples + 6-item self-check)
 │   └── latex-discipline.md        (Obsidian MathJax delimiters + $ escaping)
 └── preambles/         (auto-attached to every request as system / user messages)
    ├── inline-citation.md
    ├── image-placement.md
@ -452,16 +468,19 @@ ### Partials and preambles — shared guidance across templates
 ### Shipped templates
-Four templates ship inlined into the plugin and seed into your vault on first plugin load:
+Seven templates ship inlined into the plugin and seed into your vault on first plugin load:
 | File | Targets | Use for |
 |---|---|---|
-| `concept-profile.md` | `concepts/**` | Encyclopedia-style entries on ideas, patterns, mental models. Anti-incumbent editorial stance baked in (tech giants treated as adopters/popularizers, not innovators, unless documented heyday-era origination supports otherwise). |
+| `concept-profile.md` | `concepts/**` | Encyclopedia-style entries on ideas, patterns, mental models. Anti-incumbent editorial stance baked in (tech giants treated as adopters/popularizers, not innovators, unless documented heyday-era origination supports otherwise). Includes both `mermaid-discipline` and `latex-discipline` partials. |
 | `vocabulary-profile.md` | `Vocabulary/**` | Term definitions with disambiguation through an innovation-consulting lens. |
 | `source-profile.md` | `Sources/**` | Profiles of books, people, channels, publications, journals, reports, events — type-aware, with Google Books URL harvesting for books. |
 | `toolkit-profile.md` | `Tooling/**` | Profiles of tools, products, platforms, frameworks. |
 | `market-map-profile.md` | `lost-in-public/market-maps/**`, `market-maps/**` | Analyst-grade market-map drafts — both Known Category (e.g., Humanoid Robots) and Thesis-Driven (e.g., Neural Network Hardware as Brains for Robotics). Runs on `sonar-deep-research` with idle-only timeout, `max-tokens: 24000`, and a 40-min absolute wall-clock ceiling. |
 | `standards-and-specs-profile.md` | `Sources/Standards-and-Specs/**`, `Standards-and-Specs/**` | Analyst-grade profiles of open specs and standards. Five-way authority typing (de-jure / consortium / vendor-led-open / community / de-facto). Three-tier structural adoption framing (incumbents / challengers / innovators) plus notable holdouts. Named editors, stewardship transitions, named critics. |
 | `market-category-profile.md` | `concepts/Market-Categories/**`, `Market-Categories/**` | Concept-folder reference card for a named market category. Three-tier company landscape with explicit FINANCIAL-STAGE definitions: Incumbents (public / late-stage private / PE-owned) → Challengers (Series C+ scale-ups, recently public) → Innovators (Pre-Seed through Series B). Separate Why Now / What's Happening sections covering CAGR + category-creation momentum. Industry Coverage sub-grouped into Market Reports / Industry Articles / Financial News. |
-All four use Perplexity's `sonar-pro` (deep-research is unreliable for image return).
+The first four templates use Perplexity's `sonar-pro`. The three deep-research templates (`market-map-profile`, `standards-and-specs-profile`, `market-category-profile`) use `sonar-deep-research` and declare per-template cft-block overrides for the wall-clock ceiling (`request-timeout-ms`), the per-chunk idle timer (`stream-idle-timeout-ms`), and the Perplexity output-token budget (`max-tokens: 24000`). See [`docs/directory-templates.md`](docs/directory-templates.md) for the full cft-block grammar and the diagnostic table for distinguishing wall-clock-timeout truncation from max_tokens truncation.
 ### Auto-seed behavior
--- a/changelog/2026-05-26_01.md
+++ b/changelog/2026-05-26_01.md
@ -0,0 +1,202 @@
 ---
 title: "Market-map analyst template + the per-template timeout override the long deep-research it needs surfaced as load-bearing"
 lede: "A fifth shipped directory template aimed at analyst-grade market maps (Known Category and Thesis-Driven flavors) — the kind of document a partner pays a $150K analyst three days to produce."
 date_work_started: 2026-05-26
 date_work_completed: 2026-05-26
 date_created: 2026-05-26
 date_modified: 2026-05-26
 publish: true
 category: Changelog
 tags:
  - Directory-Templates
  - Market-Maps
  - Deep-Research
  - Streaming-Timeouts
  - cft-Block
  - Perplexity-sonar-deep-research
  - Multi-Stage-Generation
 authors:
  - Michael Staton
 augmented_with:
  - Claude Code on Claude Opus 4.7 (1M context)
 files_changed:
  - src/docs/templates/market-map-profile.md
  - src/services/templateSeederService.ts
  - src/services/directoryTemplateService.ts
  - src/docs/templates/README.md
  - docs/directory-templates.md
  - main.ts
  - context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md
  - context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md
 ---
 # A template for analyst-grade market maps, and the timeout the work it produces actually needs
 ## Why care?
 If you write market maps — the kind of document where you enumerate every credible operator in a category, segment them, name the funders, surface the open questions, and hand it to a partner — Perplexed now ships a directory template that produces a respectable analyst draft in one pass. Drop a file into `lost-in-public/market-maps/` (or any top-level `market-maps/`), give it a title and a one-line lede, run *Apply directory template to current file*, and you get back a 6-8K-word draft on `sonar-deep-research` with named innovators, per-segment funding tables, media coverage, market dynamics, and a frontier-questions section. Then you spend your time on the curation pass — promoting `[Name](url)` references to `[[wikilink]]`s back into your vault, layering in your `:::tool-showcase` blocks, tightening the analyst voice — instead of on the enumeration grind. The framing is straightforward: an analyst-day at $150K-a-year rates is $600 of someone's time, the API run costs a few dollars, and the rest is your judgment doing what your judgment does best.
 If you author *any* directory template that runs on a deep-research model, the timeout refactor is the more load-bearing change. The plugin's wall-clock cap before today was 10 minutes — perfectly reasonable for `concept-profile` and `vocabulary-profile`, fatal for anything that legitimately needs 20-30 minutes to produce its long analytical body. Today's market-map run on *Humanoid Robots and their Input Industries* was the first observable instance: ~7,500 words of body produced, then a hard stop mid-sentence in the *Frontier and Open Questions* section as the `AbortController` fired at the ceiling. The fix: per-template `request-timeout-ms:` cft-block override, plugin-level default bumped to 30 min, and an open issue filed for the structural fix (port the idle-timeout discipline the legacy modal flow already uses).
 ## What's new?
 Three shipped behaviors:
 ```
 zz-cf-lib/
 ├── templates/
 │   ├── concept-profile.md
 │   ├── vocabulary-profile.md
 │   ├── source-profile.md
 │   ├── toolkit-profile.md
 │   └── market-map-profile.md       (new — runs on sonar-deep-research)
 ├── partials/         (existing — referenced via {{include: name}})
 └── preambles/        (existing — auto-attached to every request)
 ```
 - **`market-map-profile.md`** is auto-seeded into `Content-Dev/Templates/` on first plugin load and re-seedable from settings via the *Re-seed templates* button. It matches `lost-in-public/market-maps/**` and `market-maps/**` so both your `lost-in-public/`-organized vault and a top-level `market-maps/` folder work.
 - **`request-timeout-ms:` cft-block override** lets a template declare its own wall-clock budget that wins over the plugin-level setting for that template's runs. `market-map-profile.md` declares `2400000` (40 min); the other four don't declare it and inherit the new default.
 - **Plugin-level default bumped 600000 → 1800000** (10 min → 30 min). Settings-pane description rewritten to call out the override and the cost framing ("$10-$50 of analyst time per good output is worth waiting for").
 Two context-v artifacts captured at the same time, because the market-map work raised both as the next-natural moves:
 - **`context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md`** — opens the explorations folder for this submodule. Weighs three architectures for evolving the lean v1 template into a cooperative pipeline (zoned single-file appends, multi-doc folder per market map, per-section `cft-section` blocks); proposes an `include-sources:` YAML schema covering vault paths/globs and Chroma queries; translates the "Perplexity and Claude do not overwrite each other until editing" framing into three implementation strictness levels. Recommends Option A (zoned appends) for v1.
 - **`context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md`** — the Humanoid Robots truncation as motivating symptom, side-by-side diagnosis of the two timeout disciplines in this codebase (wall-clock here, per-chunk idle in the legacy modal), the partial fix shipped today, why it's only partial, and the structural-fix proposal.
 ## How the market-map template thinks
 A market map is the draft a well-paid analyst hands to a partner. It is not an encyclopedia entry, not a marketing post, not a listicle. The template's system prompt commits to this framing explicitly and asks the model to determine the *flavor* of the map from the title and frontmatter:
 - **Known Category** — Humanoid Robots, Quantum Computing, Agentic AI in Fintech. The taxonomy is roughly settled; the work is enumerating innovators within established sub-segments and explaining the frontier.
 - **Thesis-Driven** — Neural Network Hardware as Brains for Robotics, Blockchain and Web3 Institutional Invasions. A hypothesis traverses multiple known categories. The work is making the thesis legible, then enumerating innovators from each adjacent category the thesis pulls in.
 The skeleton works for both flavors. Sections (in order):
 1. **Market Snapshot** — italicized lede + headline-stat blockquote + orienting paragraph.
 2. **The Question this Map Answers** — what this map clarifies for an operator or investor reading it.
 3. **Why Now** — 3-5 cited unlocks (technical thresholds, regulatory shifts, capital patterns, customer-behavior shifts) that make the market mappable in the current quarter.
 4. **Map of the Market — Sub-Segments** — 4-8 sub-segments with 1-2 sentence definitions. Doubles as the TOC for the next section.
 5. **Lighthouse Examples** — 5-10 named operators per sub-segment in `[Name](url) — one-line` form, flat reference list.
 6. **Innovator Profiles** — 3-7 deeper cards per sub-segment in a strict Link / Offering / Funding / Why-they-matter / Coverage shape, followed by a per-sub-segment summary table.
 7. **Media, Voices, and Coverage** — publications, podcasts, YouTubers, analysts, operator-thinkers.
 8. **Market Dynamics** — Sizing & Growth, Adoption Patterns & Barriers, Capital Flow.
 9. **Frontier and Open Questions** — 4-7 specific questions phrased as questions.
 10. **Adjacent Concepts and Maps** — plain-text concept names (no wikilinks invented; the curator wikilinks during curation).
 The anti-incumbent editorial stance from the concept-profile family carries over: big tech is treated as adopters/popularizers, not innovators, unless a research-lab paper or heyday-era origination supports otherwise. Cap big tech at 1 of 5-10 in every sub-bucket. The market-map prompt phrases this in terms specific to category mapping — "name the named operators driving the curve, not the incumbents profiting from it."
 Image-return is off by design for this template. `sonar-deep-research` returns better citation density and analytical length than `sonar-pro` but its image metadata is unreliable; market-map imagery (banner, portrait, square) is generated separately via Ideogram and lives in frontmatter, so the per-section image markers the other templates use aren't load-bearing here.
 ## How the timeout override works
 The directory-template flow has always armed a single wall-clock `setTimeout` at fetch time, racing the entire stream against `settings.requestTimeoutMs`:
 ```ts
 const controller = new AbortController();
 const timer = activeWindow.setTimeout(() => controller.abort(), timeoutMs);
 ```
 This worked fine when the entire suite of shipped templates ran on `sonar-pro` and finished in 2-3 minutes. It doesn't work when a single template legitimately needs 25-30 minutes. The naïve fix would have been to bump `settings.requestTimeoutMs` to something generous for everyone, but that punishes the short-template case (a 90-second concept run that stalls silently shouldn't take 30 minutes to surface). So we put the dial where the cft already lives — inside the template itself:
 ```yaml
 provider: perplexity
 model: sonar-deep-research
 return-citations: true
 return-images: false
 request-timeout-ms: 2400000        # 40 minutes for this template specifically
 system: |
  ...
 ```
 Resolution sits in `directoryTemplateService.ts` just before the `streamPerplexityToFile` call:
 ```ts
 const cftTimeoutRaw = template.cftConfig['request-timeout-ms'];
 const cftTimeoutMs = typeof cftTimeoutRaw === 'number'
    ? cftTimeoutRaw
    : typeof cftTimeoutRaw === 'string'
        ? parseInt(cftTimeoutRaw, 10)
        : NaN;
 const effectiveTimeoutMs = Number.isFinite(cftTimeoutMs) && cftTimeoutMs > 0
    ? cftTimeoutMs
    : settings.requestTimeoutMs;
 ```
 Number or numeric-string both work. Non-positive / non-numeric values are silently ignored and the plugin-level default takes over — the template author can't accidentally lock themselves out of the run with a typo.
 Plugin-level default bumped because the old 10-minute ceiling was clearly tuned for the original four templates and not for any subsequent deep-research template anyone might write. 30 minutes is the conservative new default: comfortably above what any of the existing four templates need, generous enough that most new deep-research templates won't need to override, and visible-enough in the settings pane that a user who hits a longer-running case can find the dial.
 ## The bug that surfaced this — Humanoid Robots, line 400
 The first real-world run of `market-map-profile` was `lost-in-public/market-maps/Humanoid Robots and their Input Industries.md`. The model produced a draft that looked respectable through nine of the ten skeleton sections, then terminated mid-sentence inside the tenth:
 ```
 Will the evolution of robot-as-a-service (R
 ```
 That trailing `(R` is the last byte written to the file. The *Adjacent Concepts and Maps* heading never appeared. Frontmatter stamps (`cf_last_run`, `cf_last_run_model`) landed correctly — the run did make it to its `processFrontMatter` post-step — so what was lost was streamed body content that hadn't arrived when the `AbortController` fired. Classic wall-clock cut-off shape.
 The directory-template flow uses a single wall-clock timeout. The *legacy* modal flow (`perplexityService.ts:659-668`) has used a per-chunk idle-timeout for several iterations:
 ```ts
 const STREAM_IDLE_TIMEOUT_MS = isDeepResearch ? 270_000 : 90_000;
 const readWithIdleTimeout = (): Promise<...> => {
    let timer: number | undefined;
    const timeout = new Promise<never>((_, reject) => {
        timer = window.setTimeout(() => {
            reject(new Error(`stream went idle for ${...}s ...`));
        }, STREAM_IDLE_TIMEOUT_MS);
    });
    return Promise.race([reader.read(), timeout]).finally(() => {
        if (timer !== undefined) window.clearTimeout(timer);
    });
 };
 ```
 The idle-timeout pattern is structurally correct. Per `reader.read()`, a fresh timeout is armed; as long as bytes keep arriving, the timer keeps getting cleared and re-armed. The stream is only killed if it goes *quiet* for N seconds. Slow-but-healthy completes; stalled-but-silent fails fast.
 We didn't port the idle-timeout pattern into the directory-template flow today, because the structural refactor warrants its own design and its own commit, not a side-quest in a template-shipping changelog. Today's fix is the pressure-relief valve: a bigger ceiling, plus a per-template dial. The structural fix is filed at `context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md` with a 4-step concrete plan, a key-rename proposal (`stream-idle-timeout-ms:`), and a backstop-or-no-backstop open question.
 ## Under the hood — the things that earned their own design decisions
 **Both flavors with one template, not two.** The Known Category vs Thesis-Driven distinction is real, but it lives in the *content* of the map, not its *structure*. A Known Category map and a Thesis-Driven map both want sub-segments, both want innovator profiles, both want the same frontier-questions framing. The system prompt names the flavor difference once and asks the model to pick from `{{title}}` and `{{frontmatter}}` signals. Two templates would have produced more drift than value.
 **No invented wikilinks.** The model doesn't know what's in your vault. If it tries to emit `[[Tooling/AI-Toolkit/Agentic AI/Crew AI]]` from training data, it'll get the path wrong half the time and the basename wrong the rest of the time. So the prompt is explicit: use `[Name](url)` form for every entity, leave wikilink promotion to the curator. The future multi-stage pipeline (per the exploration doc) will inject canonical vault content via an `include-sources:` block at request time and authorize the model to emit `[[wikilink]]`s for the entries it's been shown — but that's v2.
 **`return-images: false` because deep-research's image metadata isn't reliable enough.** This is a documented limit on `sonar-deep-research` we've eaten before in the concept-profile changelog (2026-05-12). Market-map imagery doesn't lean on inline images anyway — `banner_image`, `portrait_image`, `square_image` in frontmatter is the canonical surface, generated separately via Ideogram.
 **The override key is named `request-timeout-ms:`, not `stream-idle-timeout-ms:`.** That'll likely change when the structural fix lands and the underlying primitive moves from wall-clock to idle-timeout. We chose the misnomer-tolerant present-day name on purpose: the cft schema can rename keys with a compatibility alias in one release, and naming the new key for what it actually does *today* keeps the surface honest until the underlying mechanism changes.
 **Length not trimmed.** The market-map template's system prompt is over 1,000 words of guidance. The skeleton with bullet instructions is another 1,000+. A first instinct was to trim — most prompts of that length are over-engineered — but every section of the template's prompt is doing real work: the editorial stance, the innovator-card shape, the dual-flavor detection, the wikilink discipline, the calibration on length ("lean long, not short"). We left it long. The cost is a longer first-time-readable system prompt for vault users; the benefit is fewer regenerations and a draft that's closer to a partner-ready document.
 ## Files touched
 ```
 plugin-modules/perplexed/
 ├── src/docs/templates/
 │   ├── market-map-profile.md                  (new)
 │   └── README.md                               (cft-block-keys list now mentions request-timeout-ms)
 ├── src/services/
 │   ├── directoryTemplateService.ts             (request-timeout-ms override resolution)
 │   └── templateSeederService.ts                (market-map-profile registered as 5th seeded template)
 ├── docs/
 │   └── directory-templates.md                  (shipped-templates table now has market-map row;
 │                                                new "Per-template timeout override" section)
 ├── context-v/
 │   ├── explorations/                           (new folder; opens with the multi-stage doc)
 │   │   └── Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md
 │   └── issues/
 │       └── Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md
 └── main.ts
    ├── directoryTemplatesRequestTimeoutMs default: 600000 → 1800000
    └── settings-pane description rewritten with the override + cost framing
 ```
 Pointer-bumped at the parent: `content-farm` commit `e31b982` folds in the perplexed pointer (`a44f23b`) and a 3-line drag-along addition to `context-v/plans/Create-a-Study-of-the-Best-Obsidian-Plugins.md` (`neural-composer` by oscampo flagged as a study-candidate plugin).
 ## What's next
 - **Run the rebuilt plugin against Humanoid Robots in append mode** to extend the truncated tail. With the 40-minute ceiling on the market-map template, the *Frontier and Open Questions* section should finish and *Adjacent Concepts and Maps* should land for the first time.
 - **The structural idle-timeout fix.** The wall-clock primitive in `directoryTemplateService.ts:511` has a known better replacement living in the same codebase (`perplexityService.ts:659-668`). The plan and the open design questions live at `context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md`. Worth doing before the multi-stage spec lands, because stage 3 (Claude editorial on a 7-8K-word draft) will hit the same pathology as stage 2 (Perplexity research).
 - **The multi-stage Claude+Perplexity+RAG spec.** The exploration doc commits to a tentative direction (Option A — zoned single-file appends with `include-sources:` and `editor:` keys added to the cft schema). The next move is a spec in `context-v/specs/` pinning down the schema additions, the zone-delimiter format, the resolution order for `include-sources`, and the migration path for the existing four templates. Then a chunked prompt in `context-v/plans/`. Then `market-map-profile` becomes the reference implementation.
 - **The anti-incumbent editorial stance is now duplicated across four templates.** `concept-profile`, `vocabulary-profile`, `source-profile`, and now `market-map-profile` all carry a variation. Per the partials/preambles work that landed 2026-05-19, this is the next obvious extraction — `partials/editorial-stance-anti-incumbent.md` referenced via `{{include: editorial-stance-anti-incumbent}}`. Not done today; flagged.
--- a/changelog/2026-05-26_02.md
+++ b/changelog/2026-05-26_02.md
@ -0,0 +1,120 @@
 ---
 title: "Long deep-research streams no longer get cut off at the ceiling — the timer now resets every time bytes arrive"
 lede: "The per-chunk idle-timeout discipline the legacy modal flow has used for two iterations now governs the directory-template flow too. A healthy slow stream completes naturally; a silently-stalled one fails in seconds instead of minutes."
 date_work_started: 2026-05-26
 date_work_completed: 2026-05-26
 date_created: 2026-05-26
 date_modified: 2026-05-26
 publish: true
 category: Changelog
 tags:
  - Directory-Templates
  - Streaming-Timeouts
  - Deep-Research
  - Idle-Timeout
  - cft-Block
  - Perplexity-sonar-deep-research
  - Structural-Fix
 authors:
  - Michael Staton
 augmented_with:
  - Claude Code on Claude Opus 4.7 (1M context)
 files_changed:
  - src/services/directoryTemplateService.ts
  - docs/directory-templates.md
  - src/docs/templates/README.md
  - src/docs/templates/market-map-profile.md
  - context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md
 ---
 # The structural fix the morning's market-map run earned
 ## Why care?
 If you run any directory template on Perplexity — especially the long deep-research templates like `market-map-profile` — your timeouts now behave the way you'd intuit they should. **A stream that is healthily producing bytes runs as long as it needs to run.** A stream that goes silent (Perplexity rate-limit, socket close, upstream stall) surfaces the failure inside seconds, not at the end of a 30-minute ceiling. The change is invisible when everything goes well and only noticeable when something would have gone wrong — which is when you most want it noticeable.
 Earlier today we shipped a market-map template, watched it produce a beautiful 7,500-word draft on Humanoid Robots, and then watched the last sentence cut off mid-word because the wall-clock timer fired before the stream finished. We patched the immediate pain by raising the ceiling. **This entry is the structural fix underneath it.**
 ## What's new?
 A single concept: the timer that decides "is this stream still alive?" is now armed *per chunk*, not once per request.
 - **Old behavior.** One `setTimeout(controller.abort, timeoutMs)` at fetch time. If the stream takes longer than `timeoutMs`, even by one byte, abort. If the stream goes silent for 27 of the 30 allotted minutes, wait the full 30 anyway before noticing.
 - **New behavior.** Each `reader.read()` is raced against a fresh timer. As long as bytes keep arriving, the timer is cleared and re-armed. The stream is only killed if it goes **quiet** for `idleMs` — by default 270 seconds for deep-research models, 90 seconds for everything else.
 Two cft-block keys now govern timeouts (both optional):
 ```cft
 provider: perplexity
 model: sonar-deep-research
 stream-idle-timeout-ms: 270000   # per-chunk idle (default already 270s for deep-research)
 request-timeout-ms: 2400000      # absolute wall-clock ceiling (40 min)
 system: |
  ...
 ```
 `stream-idle-timeout-ms:` is the new primary safety. `request-timeout-ms:` is the legacy key, now repurposed as an opt-in absolute ceiling — set to `0` to disable and rely on idle-only.
 ## How it works
 Two different streaming primitives have lived in this codebase since the directory-template flow forked from the legacy `PerplexityModal` flow. The legacy modal moved to per-chunk idle-timeout discipline two iterations ago after the same problem hit users there first. The directory-template flow never received the backport — until today.
 The pattern, ported from `perplexityService.ts:659-668` into `streamPerplexityToFile`:
 ```ts
 const readWithIdleTimeout = (): Promise<ReadableStreamReadResult<Uint8Array>> => {
    let timer: number | undefined;
    const timeout = new Promise<never>((_, reject) => {
        timer = activeWindow.setTimeout(() => {
            reject(new Error(`stream went idle for ${idleMs / 1000}s (likely API stall, rate limit, or socket close)`));
        }, idleMs);
    });
    return Promise.race([reader.read(), timeout]).finally(() => {
        if (timer !== undefined) activeWindow.clearTimeout(timer);
    });
 };
 // inside the read loop:
 ({ value, done } = await readWithIdleTimeout());
 ```
 `Promise.race([reader.read(), timeout])` resolves with whichever wins. If a byte arrives, the `.finally()` clears the timer before it can fire and the loop continues. If the timer fires first, it rejects, the surrounding catch sets `truncated = true`, the existing cleanup pipeline flushes whatever already arrived, and the run returns with a partial-but-cited draft on disk and a Notice telling the user to re-run.
 The `AbortController` stays — both as the cancel mechanism for the user-initiated *Cancel* command, and as the abort target for the optional wall-clock ceiling. The two timers are complementary: idle handles "is this stream alive right now?" and ceiling handles "regardless, do not let this run forever."
 ### The two pathologies the old design handled poorly
 **Shape 1 — slow but healthy stream.** Deep-research generations on long templates sustain a slow trickle of tokens for tens of minutes. The old wall-clock cap killed them at the ceiling regardless of whether they were still producing. The idle timer lets them complete as long as bytes keep arriving inside the idle window. (The ceiling is still there if you want a hard cap — it's just not what's making the moment-to-moment safety decision anymore.)
 **Shape 2 — silently stalled stream.** Conversely, a stream may go quiet at minute 3 (Perplexity rate-limit, socket close, upstream stall) and the old wall-clock cap wouldn't notice until minute 30. The user stared at an empty file for 27 unnecessary minutes. The idle timer surfaces the failure within `idleMs` seconds — fast feedback when something is genuinely wrong.
 Both shapes are real. The wall-clock pattern punished healthy-but-slow while tolerating stalled-but-silent. The idle-timeout pattern inverts both — slow-but-healthy completes; stalled-but-silent fails fast.
 ## Migration notes
 **Nothing breaks for existing templates.** The `request-timeout-ms:` key is still honored where declared; it just means "absolute wall-clock ceiling" now instead of "the only timer at all." Templates that didn't declare it still inherit `settings.requestTimeoutMs` as their ceiling. `market-map-profile.md` keeps its `2400000` (40 min) value — under the new semantics it's the ceiling on top of the 270s idle timer, which is exactly what an analyst-grade deep-research budget should look like.
 To get **truly unbounded healthy streams** (idle-only safety, no ceiling), set `request-timeout-ms: 0` in the cft block, or set the plugin-level *Request timeout (ms)* setting to `0`. The idle timer will catch silent stalls fast either way.
 To override **the idle timer itself**, declare `stream-idle-timeout-ms: <number>` in the cft block. The defaults (270s deep, 90s normal) match the legacy modal flow and have been load-bearing there for two iterations, so most templates won't need to touch this.
 ## What's deferred
 The [open issue](../context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md) listed four follow-ups. This entry closes items 1 and 2 (the port itself, and the dual-key naming decision). Two remain explicitly deferred:
 - **Cross-service audit.** The same wall-clock pathology may live in the Gemini service, the LM Studio service, and the Claude streaming flows. The idle-timeout discipline should be the house style across all of them; the audit is its own change.
 - **Settings-pane exposure of idle defaults.** Today the 270s/90s values are hardcoded in `streamPerplexityToFile` (matching `perplexityService.ts`). Exposing them as plugin settings is a follow-up if we ever need to tune them without a code change.
 The deep-research detection is by model-name regex (`/deep-research/i`) — the same shape `perplexityService.ts` uses. Robust enough for today's Perplexity model lineup; revisit if Perplexity ever ships a long-running model under a different naming convention.
 ## Why this matters for the multi-stage exploration
 The [Multi-Stage Cooperative Claude + Perplexity with RAG exploration](../context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md) explicitly listed this fix as upstream of the multi-stage spec. Reason: the eventual Claude editorial pass on a 7-8K-word draft is itself a long generation that would hit the same wall-clock cliff. With per-chunk idle-timeout discipline now load-bearing in the directory-template flow, the editorial pass can reuse the same `readWithIdleTimeout` primitive when it lands. The multi-stage spec is now unblocked on this dimension.
 ## Files touched
 - `src/services/directoryTemplateService.ts` — `streamPerplexityToFile` signature changed from `timeoutMs: number` to `timeouts: { idleMs: number; ceilingMs: number }`. `readWithIdleTimeout()` helper added. Callsite in `applyTemplate` resolves both keys from cft config with sensible fallbacks (idle defaults from deep-research detection, ceiling defaults from `settings.requestTimeoutMs`, explicit `0` disables ceiling).
 - `docs/directory-templates.md` — *Per-template timeout override* section rewritten as *Per-template timeout overrides* (plural), now documents both keys, the two pathologies the idle timer fixes, and the migration semantics.
 - `src/docs/templates/README.md` — cft-key list updated to mention both keys.
 - `src/docs/templates/market-map-profile.md` — inline comment on the `request-timeout-ms` declaration rewritten to explain it as the absolute ceiling on top of the 270s idle timer.
 - `context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md` — referenced as the canonical issue this closes.
--- a/changelog/2026-05-27_01.md
+++ b/changelog/2026-05-27_01.md
@ -0,0 +1,216 @@
 ---
 title: "Two analyst-grade templates land — spec profiles and market-category profiles — plus two quiet bugs the testing pass caught"
 lede: "Perplexed now ships templates for profiling open specs (with five-way authority typing and three-tier adoption) and named market categories (with explicit financial-stage tiers for incumbents, challengers, and innovators). The third shipped thing was a max_tokens override we built after diagnosing why the first real spec profile stopped halfway through with no warning at all. The fourth was a rendering-discipline upgrade — strengthened mermaid rules plus new LaTeX rules — after diagnosing that the vault's concept-profile template had drifted from the bundled source and was generating diagrams with no rendering rules in scope at all."
 date_work_started: 2026-05-27
 date_work_completed: 2026-05-27
 date_created: 2026-05-27
 date_modified: 2026-05-27
 at_semantic_version: 0.3.0
 publish: true
 category: Changelog
 tags:
  - Directory-Templates
  - Standards-and-Specs
  - Market-Categories
  - Three-Tier-Framing
  - Deep-Research
  - cft-Block
  - Perplexity-sonar-deep-research
  - Max-Tokens
  - Silent-Truncation
  - Mermaid-Rendering
  - LaTeX-Rendering
  - Partials
  - Vault-Drift
 authors:
  - Michael Staton
 augmented_with:
  - Claude Code on Claude Opus 4.7 (1M context)
 files_changed:
  - src/docs/templates/standards-and-specs-profile.md
  - src/docs/templates/market-category-profile.md
  - src/docs/templates/concept-profile.md
  - src/docs/templates/README.md
  - src/docs/templates/market-map-profile.md
  - src/docs/partials/mermaid-discipline.md
  - src/docs/partials/latex-discipline.md
  - src/services/templateSeederService.ts
  - src/services/directoryTemplateService.ts
  - docs/directory-templates.md
 ---
 # Two new templates, two quiet bug fixes, and the rendering-discipline upgrade we needed to catch them
 ## Why care?
 If you spend any time figuring out which open specs your team should care about — MCP versus A2A versus the latest agent-to-agent protocol someone is pitching at a conference — Perplexed now produces an analyst-grade profile of any one of them in a single run. Same workflow you already use for concepts and market maps: drop a file into `Sources/Standards-and-Specs/`, give it the spec's name, run *Apply directory template to current file*, and you get back a 6-9K-word draft with the spec's full authorship and stewardship history, three tiers of named implementations (incumbents, challengers, innovators), the publicly named critics, the political fault lines, and the open frontier.
 If your day involves naming the companies inside a market category — "who are the actual players in agentic workspaces, sorted by stage" — there's now a separate template for that too, in `concepts/Market-Categories/`. It uses the same three-tier framing the spec-profile template uses, but tuned for FINANCIAL stage rather than adoption tier: Incumbents (the legacy public-and-PE-owned crowd you can't ignore), Challengers (the well-funded scale-ups eating share), Innovators (Pre-Seed through Series B). And unlike the market-map template — which caps big-tech mentions to counteract training-data bias — the market-category template *wants* the big tech named in Incumbents, because for a category profile, knowing the incumbents is the point.
 If you author any directory template that runs on `sonar-deep-research`, the third shipped thing matters more than either template. We found and fixed a silent-truncation pathology where Perplexity's default `max_tokens` cap (~8192 tokens, ~6K words) was ending the stream **cleanly** mid-template — full sources footer, no Notice, no truncation flag — making it look like a healthy completion that just happened to be missing the back half. The new `max-tokens:` cft override lets any template that legitimately wants 6-9K words of body bump the budget; both shipped deep-research templates now declare it at 24,000.
 If you've been frustrated that Perplexity-authored Mermaid diagrams come back broken roughly 4-of-5 times — bare `&` characters inside labels, unquoted multi-word labels with parentheses, `\n` instead of `<br/>` for line breaks — the fourth shipped thing is for you. The visible fix was strengthening the `mermaid-discipline` partial with paired BAD/GOOD examples and adding a new `latex-discipline` partial for Obsidian's MathJax delimiters. The deeper fix was diagnostic: the vault's `concept-profile.md` had drifted from the bundled source and was missing the `{{include: mermaid-discipline}}` directive entirely — meaning every concept-profile run had been generating Mermaid with no rendering rules in scope at all. Strengthening the partial wouldn't have changed a thing until the include was wired up.
 ## What's new?
 Four shipped behaviors:
 ```
 zz-cf-lib/
 ├── templates/
 │   ├── concept-profile.md
 │   ├── vocabulary-profile.md
 │   ├── source-profile.md
 │   ├── toolkit-profile.md
 │   ├── market-map-profile.md
 │   ├── standards-and-specs-profile.md       (new — open specs profiler)
 │   └── market-category-profile.md           (new — financial-stage tiers)
 └── ...
 ```
 - **`standards-and-specs-profile.md`** — auto-seeded into `Content-Dev/Templates/` and re-seedable. Matches `Sources/Standards-and-Specs/**` and `Standards-and-Specs/**`. Runs on `sonar-deep-research`.
 - **`market-category-profile.md`** — auto-seeded too. Matches `concepts/Market-Categories/**` and `Market-Categories/**`. Runs on `sonar-deep-research`.
 - **`max-tokens:` cft-block override** — new per-template knob that overrides Perplexity's default output-token cap. Number or numeric string, non-positive ignored. Both the new templates ship with `max-tokens: 24000`; the existing `market-map-profile.md` was also bumped to 24000 (it was masked from this issue by the wall-clock cap firing first; with the recent `request-timeout-ms: 0` change, it would have started hitting the same silent-truncation cliff on the next thorough run).
 - **Rendering-discipline partials — strengthened and expanded.** The `mermaid-discipline.md` partial doubled in length (250 tokens → 440) with paired BAD/GOOD examples, the `&amp;` escape rule, the explicit `\n` vs `<br/>` ban with example, a six-item self-check before emit, and a simplify-rather-than-break ethos. A new `latex-discipline.md` partial (~165 tokens) covers the three Obsidian MathJax pitfalls (`$$...$$` not `\[...\]`, `$...$` not `\(...\)`, escape literal `\$` in prose). Both wired into `concept-profile.md` (the bundled source AND the vault copy, which had drifted) immediately after the "render a mermaid codefence here" instruction so the rules are in scope at the moment of generation.
 The `directory-templates.md` doc gained a *Per-template `max-tokens:` override* section with the diagnostic table operators need to tell wall-clock truncation apart from max-tokens truncation — a vocabulary that didn't exist before today and that we needed in the moment to debug our first real spec profile.
 ## How the standards-and-specs profile thinks
 A spec profile is the document an innovation consultant hands to a partner who has just asked "should we care about this spec?" It is not implementer documentation, not a tutorial, not marketing for the spec. The template's system prompt is explicit about this — conformance matrices, MUST/SHOULD/MAY discipline, and wire-format details are out of scope; stewardship, adoption tiers, political fault lines, and named critics are in scope.
 The template handles five authority types and tunes its treatment of every downstream section based on which one the spec is:
 | Type | Examples | What changes |
 |---|---|---|
 | **De-jure** | W3C, IETF/RFC, ISO, IEEE, ECMA, NIST | Editors lifted from spec cover; working-group archives cited |
 | **Industry consortium** | Khronos, Linux Foundation, OpenJS, OASIS, CNCF | Member companies named with their voting weight |
 | **Vendor-led-open** | MCP (Anthropic), OpenAPI's early Wordnik years | Both originating team AND cross-vendor contributors named |
 | **Community** | llms.txt (Jeremy Howard), AGENTS.md (post-OpenAI handoff) | Originator's identity is the political center |
 | **De-facto** | README convention, package.json shape, the curl interface | Dominant implementation defines the spec; stewardship shifts implicitly |
 The skeleton has twelve sections (Snapshot → The Question this Spec Answers → Identity & Status → Why It Matters → Position in the Ecosystem Stack → Lineage → Governance & Stewardship → Adoption by Tier → Critique & Open Disputes → Frontier & Open Questions → Media, Voices, and Coverage → Adjacent Specs and Standards). Two pieces of structural discipline are load-bearing:
 **One — created-by and maintained-by are first-class identity.** The Snapshot lede ends with an explicit one-line callout: `**Created by** {name(s)} ({year}) · **Maintained by** {name(s)} · **Type:** {authority-type}`. For specs where the creator and current steward differ, the Governance & Stewardship section gets a dedicated paragraph telling the transition story — when it happened, what triggered it, what changed in governance pace afterward. AGENTS.md (OpenAI → community via Sourcegraph), OpenAPI (Wordnik → SmartBear → Linux Foundation), HTTP (Tim Berners-Lee at CERN → IETF/W3C), JSON (Crockford → IETF + Ecma, two parallel stewards) — all named explicitly in the system prompt as the kind of transition the template wants surfaced with care.
 **Two — the three-tier adoption framing is STRUCTURAL.** The Adoption section partitions named implementations into Incumbents (the canonical/reference implementations and the dominant deployed OSS, plus the commercial implementations from market leaders — big tech belongs here, do NOT suppress), Challengers (production-grade alternatives from mid-sized companies and well-funded startups), and Innovators (early-stage / experimental / research implementations exploring the spec's edges). Each tier gets 4-8 named entries plus 2-3 deeper implementation cards for the most strategically significant. After the three tiers, Notable Holdouts surfaces the orgs that explicitly declined, forked, or are running incompatible alternatives — often as informative as the adopters about where the spec made enemies.
 ## How the market-category profile differs
 Same three-tier shape, completely different definitions. The market-category template is for a concept-folder entry (`concepts/Market-Categories/`), not a published memo. The reader is looking for a reference card on a named market — Humanoid Robots, Agentic Workspaces, Specialized Computing for Edge Robotics — and wants to know who plays in it.
 The three tiers are sorted by **financial stage**, not by adoption:
 - **Incumbents** — large public companies, tech giants, late-stage private (post-Series E or $1B+ valuation with 10+ years of operation), PE-owned behemoths. Legacy footprint with huge market presence. The companies an enterprise buyer already has a contract with, even if not in this category yet.
 - **Challengers** — well-funded scale-ups (Series C through pre-IPO, or recently public via SPAC/IPO with under 7 years of operation). Rapidly growing, hype-driven, capital position to credibly threaten incumbent share.
 - **Innovators** — Pre-Seed through Series B funded startups. Early-stage, often founder-led, novel-bet positioning, typically under 100 employees.
 The editorial stance flips relative to the market-map template. Market maps cap big tech at 1 of 5-10 to counteract training-data bias — for an analyst memo, the named operators driving the curve matter more than the incumbents profiting from it. Market-CATEGORY profiles invert this: the goal is the *full* financial landscape of a named market, and naming the incumbents IS the goal. Both editorial stances are right for their respective documents; the templates make the disagreement explicit so neither prompt has to apologize for it.
 The template also splits **Why Now** and **What's Happening** into separate sections because they answer different questions:
 - **Why Now** = what FORCES aligned to make this category coherent right now? (Enabling conditions: technological unlocks, regulatory shifts, capital-formation patterns, customer-behavior shifts.)
 - **What's Happening** = what is the current MOMENTUM? (CAGR figures with named reports, category-creation events like a defining IPO or acquisition, where capital is concentrating by tier.)
 The reader leaves Why Now understanding the conditions and What's Happening understanding the velocity. And the Industry Coverage and Market Data section is sub-grouped explicitly: Market Reports (Gartner, IDC, Forrester, ABI Research, McKinsey, Frost & Sullivan…) → Industry Articles (specialized trade press, operator-bloggers) → Financial News (Bloomberg, FT, WSJ, Pitchbook, Crunchbase News, dealroom.co).
 ## The max-tokens story — diagnosing a silent failure
 Our first real run of the new standards-and-specs template was a profile of ARM (the architecture, not the company; the model classified it correctly as VENDOR-LED-OPEN). The output looked beautifully clean: full think-output at the top, Snapshot through the start of the Adoption section, all 50 cited sources rendered correctly in the footer. No truncation Notice. No "stream went idle" warning. Just… missing six of the fourteen template sections (Challengers, Innovators, Notable Holdouts, Critique, Frontier, Media, Adjacent).
 This was hostile in a way the wall-clock issue from last week never was. The Humanoid Robots truncation that motivated yesterday's [`request-timeout-ms:` override and idle-timeout port](./2026-05-26_01.md) ended mid-sentence at "R" — obviously a stream abort. The ARM run ended at a clean H2 section break with the full sources footer rendered. Every diagnostic signal pointed to "healthy completion" — except eight of fourteen sections were missing.
 The cause was Perplexity's default `max_tokens` cap (~8,192 tokens, ~6,000 words for `sonar-deep-research`). The stream completed cleanly via `finish_reason: "length"` — the model self-rationed its output budget across the skeleton, ran out, and wrapped up with a clean tail. The plugin's existing `streamPerplexityToFile` cleanup pipeline saw a clean stream end, rendered the sources footer, and stamped frontmatter — exactly as designed for a healthy completion, because from the stream's perspective it WAS a healthy completion.
 The fix:
 ```ts
 // In buildPayload (directoryTemplateService.ts:451):
 const maxTokensRaw = cfg['max-tokens'];
 const maxTokens = typeof maxTokensRaw === 'number'
    ? maxTokensRaw
    : typeof maxTokensRaw === 'string'
        ? parseInt(maxTokensRaw, 10)
        : NaN;
 if (Number.isFinite(maxTokens) && maxTokens > 0) {
    payload.max_tokens = maxTokens;
 }
 ```
 Templates can now declare `max-tokens: 24000` (or any positive value) in their cft block to lift the cap. Both the new shipped templates declare 24,000 — generous enough that a thorough 6-9K-word body lands with headroom for the three-tier adoption section, deeper implementation cards, named critics, and the frontier. The market-map template was bumped to 24,000 too, because it was previously masked from this issue by the wall-clock cap firing first; with the recent `request-timeout-ms: 0` change disabling that, it would have hit this silent-truncation cliff on the next thorough run.
 The new doc section in `directory-templates.md` includes the diagnostic table operators need to distinguish the two pathologies:
 | Symptom | Cause |
 |---|---|
 | Mid-sentence cutoff (literally ends with a partial word or trailing punctuation); no sources footer; user sees "stream went idle" or no Notice at all | Wall-clock timer (`request-timeout-ms:`) or idle timer (`stream-idle-timeout-ms:`) fired during streaming |
 | Clean section-end cutoff (last byte is a paragraph break or full sentence); **sources footer renders correctly with all citations**; no Notice | Perplexity `max_tokens` cap; stream completed via `finish_reason: "length"` |
 The wall-clock and idle timers are streaming-time controls; `max-tokens` is an output-budget control. They're independent. A template can hit either, neither, or both. Set all three knobs generously for thorough deep-research templates and the safety mechanisms compose without interfering.
 ## The rendering-discipline fix — and the vault-drift bug it surfaced
 The other silent failure we caught today was different in kind from the max_tokens one but identical in flavor: a problem that LOOKS like a model output issue but is actually a configuration issue one layer down. Mermaid charts in concept-profile runs were coming back broken roughly 4 out of 5 times. Common failure modes: unquoted multi-word labels with parentheses (`A[Raw inputs (text, audio)]`), bare `&` characters inside labels, `\n` instead of `<br/>` for line breaks, subgraphs with spaces in titles but no quoted display title. Each broken diagram meant another round-trip to Claude to repair before the entry could ship.
 The instinctive fix was to strengthen the rendering rules in the prompt. The Claude Code shell drafted an excellent BAD/GOOD-paired set of rules — about 350 tokens, more concrete than the existing abstract-rule version, with a five-item self-check before emit and an explicit "simplify the labels rather than emit a broken diagram" ethos. Drafting models follow rules better when each rule comes with a BAD → GOOD pair than when stated abstractly; that part of the diagnosis was sound.
 But the deeper diagnosis was that the rules weren't the bottleneck at all:
 **The vault's `concept-profile.md` had drifted from the bundled source and was missing the `{{include: mermaid-discipline}}` directive entirely.** The model was generating mermaid with no rendering rules in scope at all. Strengthening the partial wouldn't have changed a thing until the include was wired up in the running prompt.
 This is the kind of bug a long-running plugin generates when the seeded templates are user-editable and re-seed only fills in MISSING files: as the bundled template improves over time, the vault copy stays frozen at whatever version was originally seeded. The improvements never reach the running prompt. We saw the same shape earlier in this conversation — the `request-timeout-ms` and `max-tokens` knobs both required vault-copy updates separate from the bundled-source updates, because the vault templates had user edits that the seeder won't overwrite.
 The combined fix:
 - **`mermaid-discipline.md` partial — strengthened.** Original ~250 tokens, now ~440. Kept the broader special-character list and shape rules from the existing version (which were more comprehensive than what the shell drafted — `=`, `+`, backtick, brackets, braces, angle brackets, pipe, hash, LaTeX), added the BAD/GOOD pair format throughout (which is what actually shifts model behavior), added the `&amp;` escape rule, the explicit `\n` vs `<br/>` ban with example, the six-item self-check before emit, and the simplify-rather-than-break guidance. Per the *no lazy outs* discipline: no escape hatch like "describe the diagram in prose if you're unsure" — the simplify-the-labels guidance forces a working diagram with shortened text rather than a broken one with the labels the model wanted.
 - **`latex-discipline.md` partial — new.** ~165 tokens. Covers the three Obsidian MathJax pitfalls: `$$...$$` for block math (not `\[...\]`), `$...$` for inline (not `\(...\)`), and escape literal `\$` in prose to avoid accidental inline-math spans where two unescaped dollar amounts in the same paragraph get parsed as a math delimiter pair.
 - **Wired both into `concept-profile.md`** — in both the bundled source AND the vault copy, immediately after the "render a mermaid codefence here" instruction so the rules are in scope at the moment of generation. The placement matters: rules need to be in scope BEFORE generation, not in a post-hoc "now check your work" section, because drafting models ignore later self-check instructions more often than they violate earlier in-scope rules.
 The vault-drift insight generalizes beyond this fix. Any partial or template improvement we ship needs both a bundled-source update AND a manual vault-copy update for users who've already seeded the folder. The re-seed button doesn't help — it only writes missing files, not updated ones. The long-term fix is probably a "diff and patch user files" mode for re-seed (or a per-file content-hash that flags when the user's copy diverges from the shipped version); the short-term fix is that every partial or template change ships with an explicit vault-copy step in the release notes.
 **One follow-up the diagnosis raises:** rendering-rule partials might belong as system-level **preambles** rather than user-prompt **partials**. Preambles get auto-attached to every request and live in the model's persistent-discipline scope rather than appearing inline next to template content. The current partial-include pattern requires every template author to remember to include the rules in every template that emits diagrams — and the vault-drift problem we just diagnosed is precisely the case where that discipline fails. Promoting to a preamble would make the rules apply to every run automatically, including the cases where `sonar-deep-research` decides to render a diagram spontaneously without the template asking for one. Deferred — we want a few cycles of evidence that the strengthened partial actually moves the 4/5 broken rate before adding the preamble plumbing.
 ## A different editorial discipline for each template
 The three deep-research templates now in the shipped set — market-map, standards-and-specs, market-category — share a common technical substrate (sonar-deep-research, idle-only timeout, max-tokens 24000, return-images false) and a common structural posture (named entities everywhere, inline citations, anti-padding discipline, deep-research analyst voice). But their editorial discipline diverges in ways that matter:
 - **Market-map** caps big tech at 1-of-5-10 in any sub-bucket to counteract training-data bias. The analyst's job is to name the named operators driving the curve.
 - **Standards-and-specs** names big tech freely in the Incumbents tier (large dominant implementations) and aggressively in the Innovators tier (where the next extensions come from). Suppression would distort the implementation landscape.
 - **Market-category** names big tech as the FIRST move in Incumbents because for a category profile, knowing the incumbents IS the work product. Suppression would be lying about what an enterprise buyer sees.
 Each template's system prompt states its editorial stance explicitly so a future template-modifier doesn't accidentally copy the wrong one across templates. This is the kind of thing a single comment in a single template can prevent days of "why did the market-map start naming Microsoft six times" debugging.
 ## Files touched
 - `src/docs/templates/standards-and-specs-profile.md` — new. Twelve-section skeleton, five-way authority typing, three-tier structural adoption framing, named editors / stewardship-transition discipline, named-critics framing. `max-tokens: 24000`, `request-timeout-ms: 0`.
 - `src/docs/templates/market-category-profile.md` — new. Ten-section skeleton with explicit financial-stage tier definitions, separate Why Now / What's Happening sections, sub-grouped Industry Coverage (Market Reports / Industry Articles / Financial News). Same plumbing.
 - `src/docs/templates/concept-profile.md` — wired in `{{include: latex-discipline}}` alongside the existing `{{include: mermaid-discipline}}`, both immediately after the "render a mermaid codefence here" instruction so the rendering rules land in scope at the moment of generation.
 - `src/docs/templates/market-map-profile.md` — added `max-tokens: 24000` (previously masked from this issue by the wall-clock cap; needed now that `request-timeout-ms: 0` disables that cap).
 - `src/docs/templates/README.md` — added both new templates to the shipped-templates table; updated the cft-key list to mention the new `max-tokens:` key.
 - `src/docs/partials/mermaid-discipline.md` — rewritten. ~250 tokens → ~440 tokens. Paired BAD/GOOD examples throughout, `&amp;` escape rule, explicit `\n` vs `<br/>` ban, six-item self-check before emit, simplify-rather-than-break ethos. Existing broader character list and shape rules preserved.
 - `src/docs/partials/latex-discipline.md` — new. ~165 tokens covering Obsidian MathJax pitfalls (block vs inline delimiters, escaping literal `$` in prose).
 - `src/services/templateSeederService.ts` — registered both new templates and the new latex partial so they auto-seed into vaults and are bundled into `main.js`.
 - `src/services/directoryTemplateService.ts` — added `max_tokens?: number` to the `PerplexityPayload` interface and `max-tokens` resolution + payload-setting in `buildPayload` (function at line 451; resolution roughly 15 lines).
 - `docs/directory-templates.md` — added a *Per-template `max-tokens:` override* section with the wall-clock-vs-max-tokens diagnostic table; updated the cft-key list at the top.
 Bundle size: `main.js` went from 325 KB to 351 KB across the day (the two new template files + the rewritten/new partials account for most of the increase). Both repo and vault copies deploy to identical timestamps via the existing dev wire.
 ## What's deferred
 Three follow-ups surfaced today, all worth their own future entries:
 - **The `cf_finish_reason` frontmatter stamp.** A small (10-line) addition that would write Perplexity's `finish_reason` value into the target file's frontmatter on every run. Would have made the ARM diagnostic instant ("oh, `cf_finish_reason: length`, that's a max-tokens issue, not a wall-clock issue") instead of an investigation. Deferred only because the user wanted to get the two new templates out first.
 - **A "Continue directory template on current file" command.** For runs that get cut off by `max_tokens` (or by user cancellation), a resume command would read the existing body, identify which template sections are present versus missing via skeleton diff, and re-prompt the model with only the missing sections plus a URL-keyed citation merge to keep `[N]` markers coherent across the two runs. Designed but not built; the design lives in this entry's conversation thread and will be promoted to a `context-v/specs/` doc if the demand materializes.
 - **README documentation correctness.** The shipped-templates README still says "first template wins" when multiple templates match a path. The actual behavior (per `main.ts:1239`) is that the plugin opens a picker modal with all matching templates and the user chooses. The README hasn't been updated; today's matching scenario (`concepts/Market-Categories/` files match both `concept-profile` and `market-category-profile`) made the discrepancy visible.
 - **Promote rendering-discipline partials to system preambles.** The current partial-include pattern relies on every template author remembering to wire in the rules — and the vault-drift bug we caught today is precisely the case where that discipline fails. Preambles auto-attach to every request and live in persistent-discipline scope, which would catch the case where `sonar-deep-research` decides to render a diagram spontaneously without the template asking for one. Deferred until we have a few cycles of evidence that the strengthened partial actually moves the 4/5 broken-mermaid rate.
 - **A "diff and patch user files" mode for re-seed.** Today's vault-drift diagnosis surfaced a long-running plugin pathology — re-seed only writes missing files, so improvements to bundled templates and partials never reach users who've already seeded the folder. A diff-and-patch mode (with explicit per-file confirmation) would close this loop. Filed as a follow-up because the short-term workaround — manually copy each updated partial/template to the vault — is fine for a single-user dev workflow but won't scale to community-plugin distribution.
 ## References
 - [Yesterday's market-map template ship + per-template timeout override](./2026-05-26_01.md) — the precedent this entry builds on for shipping a deep-research template + the cft-knob pattern.
 - [The structural idle-timeout fix](./2026-05-26_02.md) — the port that made `request-timeout-ms: 0` safe to set and made today's `max-tokens` diagnosis tractable (without idle-timeout discipline, we'd have been chasing two truncation pathologies simultaneously).
 - [`docs/directory-templates.md`](../docs/directory-templates.md) — the new *Per-template `max-tokens:` override* section with the diagnostic table.
 - [`src/docs/templates/standards-and-specs-profile.md`](../src/docs/templates/standards-and-specs-profile.md) and [`src/docs/templates/market-category-profile.md`](../src/docs/templates/market-category-profile.md) — the two new templates as shipped.
 - [`src/docs/partials/mermaid-discipline.md`](../src/docs/partials/mermaid-discipline.md) — the strengthened partial with paired BAD/GOOD examples and the six-item pre-emit self-check.
 - [`src/docs/partials/latex-discipline.md`](../src/docs/partials/latex-discipline.md) — the new Obsidian MathJax partial.
 - The Lossless Group's [`open-specs-and-standards` study](https://github.com/Lossless-Group/open-specs-and-standards) — the 17-spec curated reference collection that informed the standards-and-specs template's authority-typing taxonomy and stewardship-transition framing.
--- a/changelog/2026-07-06_01.md
+++ b/changelog/2026-07-06_01.md
@ -0,0 +1,48 @@
 ---
 date_created: 2026-07-06
 date_modified: 2026-07-06
 title: "CORS block fixed — Perplexity streaming now routes through Node.js and never touches Chromium's CORS enforcement"
 lede: "Perplexity streaming stopped working when Perplexity's API began enforcing CORS headers that block requests originating from app://obsidian.md. Both streaming call sites — the modal query flow and the directory-template batch runner — were using activeWindow.fetch, which runs in Electron's Chromium renderer and is fully CORS-constrained. Both are now replaced with Node.js https.request, which routes through the OS network stack and is never subject to CORS enforcement."
 publish: true
 authors:
  - Michael Staton
 augmented_with:
  - Claude Code on Claude Sonnet 4.6
 files_changed:
  - src/services/perplexityService.ts
  - src/services/directoryTemplateService.ts
 ---
 ## Why care?
 If you clicked **Ask Perplexity** or ran a directory template and got a CORS error — "Access to fetch at 'https://api.perplexity.ai/chat/completions' from origin 'app://obsidian.md' has been blocked" — this is the fix. Streaming was broken for every Perplexity query, regardless of model or template. Non-streaming requests were unaffected because they already used Obsidian's `request()` function, which bypasses CORS by routing through a different layer.
 The root cause: Perplexity's API stopped including the `Access-Control-Allow-Origin` header that Chromium's renderer requires before it will hand a response body to the calling code. The response was actually arriving and returning `200 OK` — Chromium was reading the headers, seeing no CORS allowance for `app://obsidian.md`, and then blocking the caller from reading the body. The underlying request was succeeding; the plugin just couldn't see any of it.
 ## What's new?
 Both streaming code paths now use `https.request` from Node.js's built-in `node:https` module instead of `activeWindow.fetch`. Node.js routes requests through the OS network stack — TLS negotiation, DNS resolution, TCP — entirely outside Chromium's process. CORS enforcement is a Chromium concept; it has no meaning at the OS socket layer.
 The esbuild config already marks all Node.js built-ins as external (`...builtins` in the external array, `platform: 'node'`), so `import * as https from 'node:https'` compiles to `require('https')` in the output bundle and is available at runtime in Electron's renderer.
 - **`src/services/perplexityService.ts`** — the modal query path. The `activeWindow.fetch` call was replaced with a `https.request` Promise that resolves to a Node.js `IncomingMessage` stream. That stream is wrapped in a Web `ReadableStream<Uint8Array>` and passed to the existing `handleStreamingResponse` method unchanged — no changes to the SSE parsing, idle-timeout logic, or citations handling.
 - **`src/services/directoryTemplateService.ts`** — the `streamPerplexityToFile` function used by the directory-template runner. Same technique: `https.request` → `IncomingMessage` → `ReadableStream<Uint8Array>`. The `AbortController` signal (used by the ceiling timer and the user-cancel path) is passed directly to `https.request` via its `signal` option, so abort behavior is preserved.
 ## How the fix works
 The key insight is that Obsidian's Electron renderer has full Node.js integration — `require('https')` is available exactly the same as in a plain Node.js process. The reason the plugin was using `activeWindow.fetch` in the first place was that Obsidian's `requestUrl()` API buffers the entire response before returning, which makes SSE streaming impossible. Node.js `https.request` gives you the raw `IncomingMessage` stream — a readable Node.js stream — which can be consumed chunk-by-chunk exactly like a Fetch API `ReadableStream`.
 The bridge between the two worlds is a one-way conversion: each `Buffer` chunk arriving on the Node.js stream is enqueued into a Web `ReadableStream<Uint8Array>`. The existing SSE parser, idle-timeout, and citations handling never see the difference.
 ```
 Before:  activeWindow.fetch → Chromium renderer → CORS check → BLOCKED
 After:   https.request → Node.js network stack → OS socket → no CORS
 ```
 Non-streaming requests (the toggle-off path in the Perplexity modal) continue to use Obsidian's `request()` function, which already routes through the main process and was never affected.
 ## Files touched
 - `src/services/perplexityService.ts` — added `import * as https from 'node:https'` and `import type { IncomingMessage } from 'node:http'`; replaced `activeWindow.fetch` in the streaming branch of `queryPerplexity` with a `https.request` → Web `ReadableStream` bridge.
 - `src/services/directoryTemplateService.ts` — same imports; replaced `activeWindow.fetch` in `streamPerplexityToFile` with the same bridge pattern; `AbortController.signal` threaded through to `https.request`'s `signal` option to preserve ceiling-timer and user-cancel behavior.
--- a/changelog/releases/0.2.1.md
+++ b/changelog/releases/0.2.1.md
@ -0,0 +1,110 @@
 ---
 title: "Perplexed 0.2.1 — deep research that actually keeps the research"
 lede: "Run a directory template with sonar-deep-research and you'd get a near-empty note — the streaming reader was taking the wrong field and discarding ~99% of every response before it reached the file. 0.2.1 fixes that, keeps the citations even when a long research stream drops mid-flight, adds a per-run model picker to the run dialog, and rebuilds the toolkit-profile template so the model searches the product's real name and triages SEO slop instead of citing it."
 date_authored_initial_draft: 2026-05-22
 date_authored_current_draft: 2026-05-22
 date_first_published: 2026-05-22
 date_last_updated: 2026-05-22
 at_semantic_version: 0.2.1
 status: Published
 category: Release
 tags:
  - Release-Narrative
  - Directory-Templates
  - Deep-Research
  - SSE-Streaming
  - Citation-Resilience
  - Source-Quality
  - Model-Picker
  - Obsidian-Community-Plugin
 authors:
  - Michael Staton
 augmented_with: Claude Code (Opus 4.7, 1M context)
 release_tag: "0.2.1"
 release_url: "https://github.com/lossless-group/perplexed-plugin/releases/tag/0.2.1"
 prior_release_tag: "0.2.0"
 release_commit: "3ae4012"
 ---
 ## Why care?
 If you generate research notes in Obsidian with Perplexed's directory templates — the "Apply directory template to current file" workflow that fills a `Tooling/` or concept file from a heading skeleton — 0.2.1 is the release where the deep-research path actually works.
 **The headline fix: `sonar-deep-research` runs now keep their output.** Perplexity's deep-research model does all its work server-side, then delivers the entire finished document — often 10,000+ characters — in the *first* event of its streaming response, inside a field the plugin wasn't reading. The streaming reader took `delta.content`, which is correct for the lighter `sonar` models that stream token-by-token, and on a deep-research response that field holds only a ~75-character tail fragment of a 10,000-character report. The other 99% arrived on the wire and was thrown away. If you ever ran a directory template on the deep-research model and got a near-empty note, that was this bug — and `toolkit-profile` shipped with `sonar-deep-research` as its *default* model, so it was the default that was broken. Fixed.
 **Citations survive a dropped connection.** Deep research can hold a stream open for minutes; a WiFi roam or an idle socket mid-stream used to discard everything — including the citations, which deep research delivers up front in that same first event. A cut-off run now keeps whatever content and sources already arrived, writes the citations footer anyway, and tells you it was truncated so you can re-run for the rest.
 **You pick the model when you run, not in a file you have to remember to edit.** "Apply directory template" now opens a run dialog with a model dropdown. And the loading toast finally names the model that's actually running, instead of the hardcoded "deep research" string it had been showing regardless.
 ## What's new?
 ### Deep-research streaming — read the snapshot, not the fragment
 Both streaming paths (the directory-template runner and the modal "Ask Perplexity" command) now treat Perplexity's cumulative `message.content` snapshot as the source of truth, diff it against what's already been written, and append only the new tail. This is correct for every model:
 | Model | How it streams | Old reader (`delta.content` only) | 0.2.1 reader (`message.content` snapshot) |
 |---|---|---|---|
 | `sonar`, `sonar-pro` | Token-by-token deltas | Worked | Works |
 | `sonar-deep-research` | Whole document in event 1, then tiny deltas | Captured ~75 of ~10,234 chars | Captures the full document |
 A `startsWith` guard means a prefix that ever changes is skipped rather than corrupting the note — Perplexity's stream is append-only in practice, but the reader doesn't assume it.
 ### Citations that survive a timeout
 `streamPerplexityToFile` no longer throws away a run when the socket drops. It catches the read error, returns whatever `streamed` / `sources` / `images` it has, and flags the run `truncated`. `applyTemplate` then writes the partial content *with* its sources footer and shows a notice — "stream cut off … saved partial content with N sources. Re-run to complete." The modal path got the same treatment: its catch block now salvages the citation metadata it already received before writing the error line.
 The deep-research idle timeout was also lengthened — 90s → 270s in the modal path, since deep research legitimately holds the socket silent for minutes while it researches server-side; 90s was firing on healthy runs.
 ### A model picker in the run dialog
 "Apply directory template to current file" used to be a bare fuzzy template picker. It now opens `DirectoryTemplateRunModal`: a template dropdown (when more than one template matches the file), a **model dropdown** — `sonar-pro`, `sonar-reasoning-pro`, `sonar`, `sonar-reasoning`, `sonar-deep-research` — and Run / Cancel. The model defaults to the template's `cft` `model:` and overrides it for that run only; the template file is never touched. The `cf_last_run_model` frontmatter stamp now records the model that *actually ran*, override included.
 ### Domain scoping — `cf_search_domains`
 New `search_domain_filter` support. A target file can carry a `cf_search_domains:` list in its frontmatter (entity-specific), and a template can carry a `search-domains:` key in its `cft` block (generic). Plain entries allowlist, a leading `-` denylists, capped at Perplexity's 10-domain limit. Job boards — Indeed, Glassdoor, ZipRecruiter, USAJobs — are denied on every run automatically, since a job listing is never a product source.
 Allowlisting is positioned as a *last resort*, not the workflow: hand-curating domains is the same labour as searching yourself. It's there for names so collision-prone that nothing else recovers them — `NATS` the messaging system versus `NATS` the UK air traffic authority being the canonical case.
 ### The `toolkit-profile` template, rebuilt
 The shipped `toolkit-profile` template got a full system-prompt overhaul, because the old one was quietly steering the model wrong:
 - **Anchored on the real name, not the scraped tagline.** The prompt used `{{title}}` as the entity identity — but in scraped Tooling files `title` is the page's `og:title`, a marketing tagline. Profiling `Kubernetes.md` told the model the entity was named "Production-Grade Container Orchestration," so it searched the tagline. The prompt now anchors on `{{basename}}` (the filename = the real name) and explicitly flags `{{title}}` as a tagline.
 - **Source-quality is a triage task, not an allow/deny list.** The old prompt hard-banned "consulting vendors." The new one tells the model it's the editor: search rankings are SEO-driven, so judge each page on its merits — favour editorial authority (the entity's own docs, standards bodies, established trade press) over rank, and treat a vendor's blog post as fine *if the page itself is substantive*. The only hard reject is a page about a different same-named entity.
 - **"Search, don't recall."** An explicit instruction that training knowledge is stale and uncitable, and that a profile assembled from memory rather than search results is a failed profile.
 - **Length discipline.** A `LENGTH AND ECONOMY` block turns the per-section sentence/paragraph counts already in the skeleton into hard ceilings, bans filler, and tells the model a thin section should be short — "length is not evidence of quality."
 - **Default model is now `sonar-pro`** — fast, clean, grounded — with `sonar-deep-research` a dialog click away when you want depth.
 ### Longer request timeout
 The directory-template request timeout default moved from 5 minutes to 10 (`DEFAULT_SETTINGS` in `main.ts`), so a `sonar-deep-research` run picked from the model dialog has room to finish.
 ## Under the hood
 0.2.1 is a single concentrated session of work — there are no intermediate commits between the 0.2.0 release and this one. The changeset, eight files:
 | File | Change |
 |---|---|
 | `src/services/perplexityService.ts` | `message.content`-snapshot streaming; metadata salvage in the catch block; deep-research idle timeout 90s → 270s |
 | `src/services/directoryTemplateService.ts` | Same streaming fix; `truncated` partial-result salvage; `search_domain_filter` support (`cf_search_domains` + `search-domains` + job-board denylist); per-run model override |
 | `src/modals/DirectoryTemplateRunModal.ts` *(new)* | The template + model run dialog |
 | `main.ts` | Run-modal wiring; 10-minute timeout default |
 | `src/docs/templates/toolkit-profile.md` | Rebuilt system prompt — entity anchoring, source triage, concision, `sonar-pro` default |
 | `manifest.json`, `package.json`, `versions.json` | 0.2.1; description aligned ("Perplexica (now Vane)") |
 The streaming bug was found the honest way — a raw `curl` against `sonar-deep-research` with `stream: true`, watching the SSE events scroll past. Event one carried a 10,169-character `message.content` and a 10-character `delta.content`. That was the whole diagnosis.
 ## What's next
 - **Validate the rebuilt template on clean runs.** The `toolkit-profile` overhaul ships in this release; the next pass confirms entity-anchoring and source-triage hold up across a spread of `Tooling/` entries.
 - **Carry the prompt rebuild to the other three templates.** `concept-profile`, `vocabulary-profile`, and `source-profile` still use the old prompt shape and likely mis-anchor the same way. The entity-name fix and the triage instruction generalize directly.
 - **A `max-tokens` ceiling.** If prompt-level concision isn't enough to rein in deep research, a `max-tokens:` `cft` key would give a hard cap — held back here because a hard cap truncates mid-sentence and prompt discipline is the cleaner lever.
 - **Diff-aware regeneration.** Still true from 0.2.0's list: a re-run on a file that already has `cf_last_run` stamped should offer to *replace* rather than silently append. Right now stacked re-runs pile up.
 ## References
 - **Release on GitHub:** https://github.com/lossless-group/perplexed-plugin/releases/tag/0.2.1
 - **Install from Obsidian:** Settings → Community Plugins → Browse → "Perplexed"
 - **Prior release:** [`0.2.0`](./0.2.0.md) — Google Gemini joins the provider lineup
 - **Perplexity models:** [docs.perplexity.ai](https://docs.perplexity.ai/) — `sonar`, `sonar-pro`, `sonar-reasoning-pro`, `sonar-deep-research`
 - **Sibling Lossless plugins:** [`image-gin`](https://github.com/lossless-group/image-gin), [`cite-wide`](https://github.com/lossless-group/cite-wide), [`metafetch`](https://community.obsidian.md/plugins/metafetch)
--- a/changelog/releases/0.3.0.md
+++ b/changelog/releases/0.3.0.md
@ -0,0 +1,176 @@
 ---
 title: "Perplexed 0.3.0 — the analyst-grade release: three deep-research templates for VC / PE / equities workflows, and the four infrastructure fixes they needed to ship"
 lede: "0.3.0 turns Perplexed into a serious analyst tooling layer. Three new deep-research templates (market-map, standards-and-specs, market-category) produce 6-9K-word cited analyst drafts in a single run. Underneath them, four infrastructure fixes — a per-template wall-clock override, an idle-timeout structural port, a max_tokens output-budget knob, and a strengthened mermaid + new LaTeX rendering discipline — turn deep-research from a brittle 'might work' workflow into something you can put real analyst time behind."
 date_authored_initial_draft: 2026-05-27
 date_authored_current_draft: 2026-05-27
 date_first_published: 2026-05-27
 date_last_updated: 2026-05-27
 at_semantic_version: 0.3.0
 status: Published
 category: Release
 tags:
  - Release-Narrative
  - Directory-Templates
  - Deep-Research
  - Market-Maps
  - Standards-and-Specs
  - Market-Categories
  - Three-Tier-Framing
  - Streaming-Timeouts
  - Idle-Timeout
  - Max-Tokens
  - Silent-Truncation
  - Mermaid-Rendering
  - LaTeX-Rendering
  - Partials
  - Vault-Drift
  - Obsidian-Community-Plugin
  - Venture-Capital
  - Private-Equity
  - Equities-Trading
 authors:
  - Michael Staton
 augmented_with: Claude Code (Opus 4.7, 1M context)
 release_tag: "0.3.0"
 prior_release_tag: "0.2.1"
 ---
 ## Why care?
 If your work involves naming the actual companies in a market category, mapping who's playing where in an emerging market, or profiling the open specs and protocols an investment thesis depends on — **0.3.0 is the release where Perplexed becomes a serious analyst tooling layer**, not just a notes-with-citations plugin.
 Three new directory templates do the bulk of the work. Each produces a 6-9K-word cited analyst draft in a single Perplexity Deep Research run. Each was designed for the kind of document a VC analyst, PE associate, or equities-trading desk strategist hands to a partner or IC:
 - **Market-category profiles** name the companies in a category, sorted into three explicit financial-stage tiers — Incumbents (public / late-stage private / PE-owned), Challengers (Series C+ scale-ups), Innovators (Pre-Seed through Series B) — with separate Why Now / What's Happening sections covering CAGR and category-creation momentum, and a fully-sourced Industry Coverage section sub-grouped into Market Reports (Gartner, IDC, Forrester) / Industry Articles / Financial News (Bloomberg, FT, Pitchbook).
 - **Market-map drafts** produce the analyst memo itself — 4-8 sub-segments, 20-40 named innovator cards with funding stage and lead investor, Market Dynamics with cited sizing data and capital concentration, and a Frontier section with open questions framed as questions.
 - **Standards-and-specs profiles** turn an open protocol or convention into a strategic-context document — five-way authority typing (de-jure / consortium / vendor-led-open / community / de-facto), three-tier structural adoption framing (incumbents / challengers / innovators) plus notable holdouts, named editors, stewardship-transition stories, named public critics with their arguments. The framing surfaces exactly what an analyst needs to know: is this protocol surviving its originating vendor, who has it captured, who's working to break the capture.
 If you author any directory template that runs on `sonar-deep-research`, the four infrastructure fixes that made these templates ship reliably matter more than any single template. Each fix solved a class of "deep-research silently produces broken output" failure mode that would otherwise have kept the analyst-grade workflow brittle.
 If you ever ran a long template and got a draft cut off mid-sentence, the streaming-timeout refactor fixes that — the timer that decides "is this stream still alive?" is now armed per-chunk, not once per request, so a healthy slow stream completes naturally and a silently-stalled one fails in seconds. If you ever ran a long template and got a draft that looked complete but was missing the back half of the section skeleton, the new `max-tokens:` override fixes that — Perplexity's default output cap was silently truncating thorough drafts by ending the stream cleanly mid-template. If you've been frustrated that Perplexity-authored Mermaid diagrams come back broken 4-of-5 times, the strengthened mermaid partial (plus a new LaTeX partial for Obsidian MathJax) addresses that — provided your vault's `concept-profile.md` is wired to include them, which the release notes here will help you verify.
 ## What's new?
 ```
 zz-cf-lib/
 ├── templates/
 │   ├── concept-profile.md
 │   ├── vocabulary-profile.md
 │   ├── source-profile.md
 │   ├── toolkit-profile.md
 │   ├── market-map-profile.md              (new in 0.3.0)
 │   ├── standards-and-specs-profile.md     (new in 0.3.0)
 │   └── market-category-profile.md         (new in 0.3.0)
 ├── partials/
 │   ├── mermaid-discipline.md              (strengthened in 0.3.0 — ~250 → ~440 tokens)
 │   └── latex-discipline.md                (new in 0.3.0)
 └── preambles/
    ├── inline-citation.md
    ├── image-placement.md
    └── research-framing.md
 ```
 Three new templates, three new cft-block knobs, one new partial, one strengthened partial:
 - **`market-map-profile.md`** — analyst-grade market-map drafts in Known Category (Humanoid Robots, Quantum Computing) or Thesis-Driven (Neural Network Hardware as Brains for Robotics) shape. Matches `lost-in-public/market-maps/**` or `market-maps/**`. Ships with `sonar-deep-research`, `request-timeout-ms: 2400000` (40-min ceiling), `max-tokens: 24000`.
 - **`standards-and-specs-profile.md`** — open-spec/standard profiles with five-way authority typing and three-tier adoption framing. Matches `Sources/Standards-and-Specs/**`. Ships with `sonar-deep-research`, `request-timeout-ms: 0` (idle-only safety), `max-tokens: 24000`.
 - **`market-category-profile.md`** — concept-folder reference card for a named market category with three financial-stage tiers. Matches `concepts/Market-Categories/**`. Ships with `sonar-deep-research`, `request-timeout-ms: 0`, `max-tokens: 24000`.
 - **`request-timeout-ms:` cft override** — per-template absolute wall-clock ceiling that wins over the plugin-level setting. Set to `0` to disable and rely on idle-only safety.
 - **`stream-idle-timeout-ms:` cft override** — per-template per-chunk idle timer (default 270s for deep-research, 90s otherwise). The primary safety mechanism after the structural port from the legacy modal flow.
 - **`max-tokens:` cft override** — per-template Perplexity output-token budget override. Lifts the silent ~8K-token default ceiling that was clean-truncating thorough drafts. Both new deep-research templates and the market-map template now declare 24,000.
 - **Strengthened `mermaid-discipline.md` partial** — paired BAD/GOOD examples throughout, the `&amp;` escape rule, explicit `\n` vs `<br/>` ban, six-item self-check before emit, simplify-rather-than-break ethos. Existing broader special-character list and shape rules preserved.
 - **New `latex-discipline.md` partial** — Obsidian MathJax delimiters (`$$...$$` for block, `$...$` for inline, NOT `\[...\]` / `\(...\)`), plus `\$` escaping for literal dollar amounts to avoid accidental inline-math spans.
 The plugin-level default request timeout was also bumped from 600,000 ms (10 min) to 1,800,000 ms (30 min). Settings-pane description rewrote to call out the cost framing — "$10-$50 of analyst time per good output is worth waiting for."
 ## The streaming-timeout story — wall-clock to per-chunk idle
 The first thing 0.3.0 fixes is also the most invisible. Until 0.2.x, the directory-template runtime armed a single `setTimeout(controller.abort, timeoutMs)` at fetch time. If the stream took longer than the ceiling, even by one byte, abort. If the stream went silent for 27 of 30 allotted minutes, wait the full 30 anyway before noticing.
 We hit both pathologies in the same week. The first real run of `market-map-profile` on a Humanoid Robots map produced ~7,500 words then cut off mid-sentence at "R" in the Frontier section because the deep-research run legitimately needed longer than the ceiling. Two days later, the first real run of `standards-and-specs-profile` on the ARM architecture stopped at a clean section break with all 50 citations rendered correctly — and was missing six of fourteen template sections. Same template, two different failure modes, neither one obvious from the output until you knew what to look for.
 The fixes ran in two waves:
 **Wave 1 — pressure-relief valves.** Per-template `request-timeout-ms:` cft override, plugin-level default bumped 10 → 30 min. The market-map template declares 40 min. Bought headroom; didn't fix the structural problem.
 **Wave 2 — port the idle-timeout discipline from the legacy modal flow.** The `PerplexityModal` flow had moved to per-chunk idle-timeout discipline two iterations ago — each `reader.read()` raced against a fresh timer; as long as bytes keep arriving, the timer is cleared and re-armed; only a stream that goes *quiet* for `idleMs` is killed. The directory-template flow was forked from an earlier iteration and never received the backport. 0.3.0 ports it. The wall-clock timer becomes an optional belt-and-suspenders ceiling; per-chunk idle (270s for deep-research, 90s otherwise) is the primary safety.
 The result: healthy slow streams complete naturally regardless of how long they take. Stalled streams fail in seconds, not at the ceiling. Setting `request-timeout-ms: 0` in a cft block disables the ceiling entirely and relies on idle-only safety — the recommended setting for the two new analyst-grade templates.
 ## The max_tokens story — when "healthy completion" lies
 The second silent failure was different in flavor. The ARM spec-profile run completed cleanly — full sources footer, all 50 citations, no Notice, no truncation flag, clean section-end cutoff. Every diagnostic signal pointed to "healthy completion." Eight of fourteen sections were missing.
 The cause: Perplexity's default `max_tokens` cap (~8,192 tokens, ~6,000 words for `sonar-deep-research`). The stream completed via `finish_reason: "length"` — the model self-rationed across the skeleton, ran out of output-token budget, and wrapped up with a clean tail. The plugin's cleanup pipeline saw a clean stream end, rendered the sources footer, stamped frontmatter — exactly as designed for a healthy completion, because from the stream's perspective it WAS a healthy completion.
 The fix is a new `max-tokens:` cft override that passes through to Perplexity's `max_tokens` parameter. Both new deep-research templates and the market-map template declare `max-tokens: 24000` (~18K-word budget). The new section of `docs/directory-templates.md` includes the diagnostic table operators need to distinguish the two pathologies:
 | Symptom | Cause |
 |---|---|
 | Mid-sentence cutoff (literally ends with a partial word); no sources footer; user sees "stream went idle" or no Notice at all | Wall-clock or idle timer fired during streaming |
 | Clean section-end cutoff; **sources footer renders correctly with all citations**; no Notice | Perplexity `max_tokens` cap; stream completed via `finish_reason: "length"` |
 The wall-clock, idle, and max-tokens knobs are independent. A template can hit any of them, all of them, or none. The three new templates set all three generously, and the cleanup pipeline behaves correctly for whichever (if any) fires.
 ## The rendering-discipline story — and the vault-drift bug it surfaced
 The third silent failure was a model output issue layered on top of a configuration issue. Concept-profile runs were producing broken Mermaid diagrams roughly 4 of 5 times — unquoted multi-word labels with parentheses, bare `&` characters, `\n` instead of `<br/>`, subgraphs with spaces in titles but no quoted display title. Each broken diagram meant another round-trip to Claude to repair before the entry could ship.
 The instinctive fix was to strengthen the rendering rules. The bundled `mermaid-discipline.md` partial already existed and was already pulled into `concept-profile` via `{{include: mermaid-discipline}}` — but the existing rules were stated abstractly and didn't include paired BAD/GOOD examples (which is what actually shifts model behavior).
 The deeper diagnosis showed the rules weren't the bottleneck at all:
 **The vault's `concept-profile.md` had drifted from the bundled source and was missing the `{{include: mermaid-discipline}}` directive entirely.** The model was generating Mermaid with no rendering rules in scope at all. Strengthening the partial wouldn't have changed a thing until the include was wired up in the running prompt.
 This is the kind of bug a long-running plugin generates when seeded templates are user-editable and re-seed only fills in MISSING files: as the bundled template improves over time, the vault copy stays frozen at whatever version was originally seeded. The improvements never reach the running prompt.
 The 0.3.0 fix is twofold:
 - **`mermaid-discipline.md`** rewritten with paired BAD/GOOD examples throughout, the `&amp;` escape rule, the explicit `\n` vs `<br/>` ban, a six-item self-check before emit, and a simplify-rather-than-break ethos (no escape hatch like "describe in prose if unsure" — the discipline forces a working diagram with shortened text rather than a broken one with the labels the model wanted).
 - **`latex-discipline.md`** added as a new partial covering Obsidian MathJax delimiters and `$` escaping in prose.
 Both wired into `concept-profile.md` in both the bundled source AND the vault copy. **If you've been seeding the templates folder before today, run a quick check after upgrading**: open `Content-Dev/Templates/concept-profile.md` and verify it contains both `{{include: mermaid-discipline}}` and `{{include: latex-discipline}}` directives in the Defining and Describing section. If they're missing (as they were in our vault), add them manually — the re-seed button won't update files that already exist.
 The vault-drift insight generalizes beyond this fix and is filed as a 0.4.x follow-up: any partial or template improvement we ship needs both a bundled-source update AND a manual vault-copy update for users who've already seeded the folder. A "diff and patch user files" mode for re-seed would close the loop properly.
 ## Editorial discipline by template — three different stances on big tech
 The three new deep-research templates share a common technical substrate (sonar-deep-research, idle-only timeout, max-tokens 24000, return-images false) and a common structural posture (named entities everywhere, inline citations, anti-padding discipline). Their editorial discipline diverges in ways that matter for analyst output quality:
 - **Market-map** caps big tech at 1-of-5-10 in any sub-bucket to counteract training-data bias. The analyst's job is to name the named operators driving the curve, not the incumbents profiting from it.
 - **Standards-and-specs** names big tech freely in the Incumbents tier (large dominant implementations) and aggressively in the Innovators tier (where the next extensions come from). Suppression would distort the implementation landscape.
 - **Market-category** names big tech as the FIRST move in Incumbents because for a category profile, naming the incumbents IS the work product. Suppression would be lying about what an enterprise buyer sees.
 Each template's system prompt states its editorial stance explicitly so a future template-modifier doesn't accidentally copy the wrong one across templates.
 ## Upgrade notes
 **Vault template drift — the one you have to check by hand.** The biggest gotcha in 0.3.0 is the rendering-discipline fix described above. If your `Content-Dev/Templates/concept-profile.md` was seeded before today, it may be missing the `{{include: mermaid-discipline}}` and `{{include: latex-discipline}}` directives — meaning even after upgrading the plugin and getting the strengthened partials in the vault, the running prompt won't include them until you wire the includes into the template manually. The README's Directory Templates section has the placement; insert both directives immediately after the "render a `mermaid` codefence here" instruction in the *Defining and Describing* section. Five seconds of editing; saves you from continued 4-of-5 broken Mermaid.
 **Re-seed pulls in the new templates and the new latex partial.** The seven shipped templates plus the two partials are all bundled into `main.js`. After upgrading to 0.3.0, click *Settings → Directory templates → Re-seed templates* to write the new files into your vault (the button only adds files whose filenames don't already exist; nothing existing gets overwritten). You'll see `market-map-profile.md`, `standards-and-specs-profile.md`, `market-category-profile.md` appear in your templates folder, and `latex-discipline.md` appear in your partials folder.
 **Backward compatibility.** All cft-block additions (`request-timeout-ms`, `stream-idle-timeout-ms`, `max-tokens`) are opt-in. Templates that don't declare them inherit sensible defaults. The four pre-0.3.0 templates (concept, vocabulary, source, toolkit) run unchanged.
 **Plugin-level default request timeout bumped 10 min → 30 min.** Visible in *Settings → Directory templates → Request timeout (ms)*. If you had previously set a custom value, it's preserved.
 ## What's deferred
 Three follow-ups surfaced during the 0.3.0 work cycle and are worth tracking for 0.4.x:
 - **A `cf_finish_reason` frontmatter stamp** on every template run. Would have made the ARM max-tokens diagnosis instant ("oh, `cf_finish_reason: length`, that's a max-tokens issue, not a wall-clock issue") instead of an investigation.
 - **A "Continue directory template on current file" command.** For runs that get cut off by `max_tokens` or by user cancellation, a resume command would read the existing body, identify which template sections are present vs missing via skeleton diff, and re-prompt the model with only the missing sections plus a URL-keyed citation merge to keep `[N]` markers coherent across the two runs.
 - **A "diff and patch user files" mode for re-seed.** The vault-drift bug we caught today is structural: re-seed only writes missing files, so improvements to bundled templates and partials never reach users who've already seeded the folder. A diff-and-patch mode (with explicit per-file confirmation) would close the loop.
 ## Engineering changelog references
 The full work that landed in 0.3.0 is captured across three engineering-changelog entries:
 - [`changelog/2026-05-26_01.md`](../2026-05-26_01.md) — the market-map template + the per-template `request-timeout-ms:` override + the plugin-level default bump.
 - [`changelog/2026-05-26_02.md`](../2026-05-26_02.md) — the structural idle-timeout port from `perplexityService.ts:659-668` into `streamPerplexityToFile`. Closed the "Wall-clock timeout cuts off long deep-research streams" issue.
 - [`changelog/2026-05-27_01.md`](../2026-05-27_01.md) — the two new templates (standards-and-specs, market-category), the `max-tokens` cft override, the strengthened mermaid + new LaTeX partials.
 The 0.2.1 → 0.3.0 commit range will appear in the GitHub release page when this version is tagged.
 ## Compatibility
 `minAppVersion: 1.8.10` — unchanged from 0.2.1. Desktop-only (`isDesktopOnly: true`) — unchanged. No new external dependencies; the three new templates and two partials are bundled into `main.js` at build time.
--- a/changelog/releases/0.3.1.md
+++ b/changelog/releases/0.3.1.md
@ -0,0 +1,64 @@
 ---
 title: "Perplexed 0.3.1 — Perplexity streaming restored after CORS enforcement change"
 lede: "Perplexity streaming broke when Perplexity's API stopped including the CORS headers that Electron's Chromium renderer requires. Every streaming query — modal and directory templates alike — was silently failing with a 200 OK response that Obsidian couldn't read. 0.3.1 routes all streaming through Node.js https instead of fetch, bypassing CORS enforcement entirely. Non-streaming requests were never affected."
 date_authored_initial_draft: 2026-07-06
 date_authored_current_draft: 2026-07-06
 date_first_published: 2026-07-06
 date_last_updated: 2026-07-06
 at_semantic_version: 0.3.1
 status: Published
 category: Release
 tags:
  - Release-Narrative
  - CORS
  - Streaming
  - Node-https
  - Perplexity-API
  - Bug-Fix
  - Electron
 authors:
  - Michael Staton
 augmented_with:
  - Claude Code on Claude Sonnet 4.6
 release_tag: "0.3.1"
 prior_release_tag: "0.3.0"
 ---
 ## Why care?
 If you upgraded to Obsidian recently or noticed that **Ask Perplexity** and your directory templates silently stopped producing output — no streamed text, no error message in the note, just a CORS error in the developer console — this release fixes it.
 Perplexity's API changed its CORS headers. Electron's Chromium renderer enforces CORS strictly: if a server's response doesn't include `Access-Control-Allow-Origin` for the calling origin, Chromium reads the response headers and then refuses to hand the body to the calling code. The request was completing successfully on Perplexity's end (`200 OK`), but the plugin received nothing because Chromium blocked access to the body.
 Every streaming query was affected: the quick **Ask Perplexity** modal, the deep-research modal, and the directory-template batch runner (`streamPerplexityToFile`). Non-streaming requests (the toggle-off path) were never affected — they already used Obsidian's `request()` function, which routes through the main process and has no CORS constraint.
 0.3.1 routes all streaming through Node.js's built-in `https` module. Node.js operates below the Chromium layer entirely — it's a direct OS socket connection. CORS is a browser concept; it doesn't exist at the socket level.
 ## What changed?
 Two files, minimal surface area:
 **`src/services/perplexityService.ts`** — the modal query path. `activeWindow.fetch` replaced with `https.request` from `node:https`. The resulting Node.js `IncomingMessage` stream is wrapped in a Web `ReadableStream<Uint8Array>` and handed to the existing `handleStreamingResponse` method. The SSE parsing, idle-timeout, chunk accumulation, citations, and images handling are untouched.
 **`src/services/directoryTemplateService.ts`** — the `streamPerplexityToFile` function. Same approach. The existing `AbortController` (used by the ceiling timer and user-cancel path) is threaded directly into `https.request`'s `signal` option — abort behavior is fully preserved.
 ```
 Before:  activeWindow.fetch → Chromium renderer → CORS enforcement → body blocked
 After:   https.request → Node.js network stack → OS socket → no CORS
 ```
 No changes to the Perplexity API payload, model list, streaming protocol, or any vault-side behavior. This is a pure transport-layer swap.
 ## Upgrade notes
 No action required. Reload the plugin after upgrading (Settings → Community Plugins → Perplexed → toggle off, then on) and streaming will work immediately. There are no vault-template changes, no new cft-block keys, no settings-pane changes.
 If you are running Perplexed via a development symlink (`ln -s` from the repo into your vault's `.obsidian/plugins/`), run `pnpm build` from the `perplexed/` directory and then reload the plugin.
 ## Compatibility
 `minAppVersion: 1.8.10` — unchanged. Desktop-only — unchanged. No new external dependencies; the fix uses `node:https` from Node.js's standard library, which is already available in Obsidian's Electron runtime. All 0.3.0 features (three analyst-grade templates, per-template timeout and max-tokens overrides, rendering-discipline partials) are fully intact.
 ## Engineering changelog
 - [`changelog/2026-07-06_01.md`](../2026-07-06_01.md) — the full diagnosis and implementation details.
--- a/context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md
+++ b/context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md
@ -0,0 +1,346 @@
 ---
 title: "Multi-Stage Cooperative Claude + Perplexity with RAG"
 lede: "What if a market-map draft were the output of three agents — a RAG pre-flight that names what to include, a Perplexity research run that finds the rest, and a Claude editorial pass that never steps on either — instead of one single-shot prompt?"
 date_created: 2026-05-26
 date_modified: 2026-05-26
 authors:
  - Michael Staton
 augmented_with:
  - Claude Opus 4.7 (1M context)
 semantic_version: 0.0.0.1
 tags:
  - Exploration
  - Perplexed
  - Multi-Stage-Generation
  - Agentic-Orchestration
  - RAG
  - Chroma
  - Market-Maps
  - Claude
  - Perplexity-Deep-Research
 status: Open
 related:
  - "[[market-map-profile]]"
  - "[[Using-Files-as-Prompt-Outlines]]"
  - "[[Partials-And-Preambles-For-Perplexed-Templates]]"
  - "[[Getting-Claude-to-Respond-With-Research]]"
  - "[[Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams]]"
 ---
 # Multi-Stage Cooperative Claude + Perplexity with RAG
 A market map is the kind of document where one-shot generation hits its limit. The lean v1 `market-map-profile` template ([`src/docs/templates/market-map-profile.md`](../../src/docs/templates/market-map-profile.md)) produces a respectable analyst draft in a single Perplexity deep-research run, but every analyst who has read one of our existing market maps notices the same gaps: the wikilinks back to our own `Tooling/` and `concepts/` aren't there, the segmentation occasionally misses a sub-bucket we cover heavily in the vault, and the editorial voice drifts from "well-paid analyst" toward "encyclopedia summary" without a second pass. The fix is multi-stage, but multi-stage has more design surface than it looks: who runs first, how they hand off, where the RAG context lives, and — the question that motivated this doc — **how do we keep Claude and Perplexity from overwriting each other's work?**
 ## The question
 We want to evolve `market-map-profile` (and, by precedent, any other research-heavy template) from a one-shot Perplexity run into a cooperative pipeline that includes:
 1. A **RAG pre-flight** that pulls canonical Lossless source material (vault tools, concepts, prior market maps; Chroma-indexed corpus chunks) and bakes it into the prompt context as named, citeable inputs.
 2. A **Perplexity research stage** that does what `sonar-deep-research` does today, but primed with the RAG context so it cites our own canonical entries by name rather than generating shadow versions of them.
 3. A **Claude editorial stage** that enforces analyst voice, prunes over-enumeration, restructures sub-segments where the data demands, emits `[[wikilink]]`s for entries that exist in the vault, and surfaces "the question this map answers" framing.
 The specific design questions:
 - **How do RAG-provided sources get expressed in a template?** The user articulated this as an `include_sources:` argument — a list of sources to include, distinct from `search-domains:` (which constrains where Perplexity searches the open web). What's the surface shape — a list of file paths, a Chroma query spec, both?
 - **How do the three stages share a target file without clobbering each other?** Perplexity streams text into the file body. Claude's editorial pass needs to *edit* that text. If Claude runs while Perplexity is still streaming, or if Claude edits the same region Perplexity wrote, we get a race or a confused output.
 - **Single file or multi-document folder?** A market map could remain one large markdown file with sectioned output, or it could become a folder where each sub-segment is its own document with a manifest at the root. The latter is more agentic-friendly (each section is independently re-runnable) but breaks the single-file mental model that every other directory template uses.
 - **Where does the RAG index live, and how does it stay fresh?** The Chroma collections in [`lossless-monorepo/CLAUDE.md`](../../../../CLAUDE.md) (`context-vigilance-corpus`, `lossless-changelog`, `claude-code-sessions`, `claude-code-tool-traces`) are reachable via the `chroma` MCP server inside Claude Code, but the perplexed plugin runs inside Obsidian — it can't reach an MCP server. The plugin would need its own client to the same Chroma instance, OR the RAG pre-flight runs *outside* the plugin and writes its output into the target file before the plugin's Perplexity stage triggers.
 ## Why we don't already know
 Three reasons.
 **One — the existing `cft` template surface is intentionally minimal.** The `ParsedTemplate` interface ([`src/services/directoryTemplateService.ts:31`](../../src/services/directoryTemplateService.ts)) gives a template four things: a config block, a system prompt, a user skeleton, and the four shipped templates ride that surface comfortably because each is *one* provider call. The moment we add a second provider — Claude after Perplexity, or a RAG client before either — we need to decide whether the template is still one file (with multiple stages declared inside) or whether it's a *pipeline* with a per-stage manifest. We have no precedent in this codebase for the latter.
 **Two — `include_sources` isn't a Perplexity API field, and we haven't decided what it means.** `search_domain_filter` constrains where Perplexity searches; it doesn't inject our content into the prompt. What the user is gesturing at — "a list of sources to include" — is closer to **retrieval-augmented generation**: pull canonical text from named documents (or a Chroma vector query), splice it into the prompt as named, citeable inputs, then let the model lean on it. We already have the splice primitive (`{{include: name}}` partials, [`directoryTemplateService.ts:83`](../../src/services/directoryTemplateService.ts)) — but partials are pure snippets, content-agnostic. RAG sources need different handling: they want to be named, attributed, and bounded so the model knows what it's quoting.
 **Three — we don't yet know whether the editing stage should run as an *edit* or as an *additive pass*.** The user's framing — *"Perplexity and Claude do not overwrite each other until editing"* — implies a model where Perplexity writes its draft in one zone of the file, Claude writes in another, and only at the explicit edit-phase do they consolidate. That's a real design choice with consequences for the file shape, the cft block schema, and how the user knows where they are in the pipeline.
 ## Options
 Three architectures to weigh. They are not mutually exclusive — Option C is closer to a generalization of A and B than a distinct third path.
 ### Option A — Sequential single-file appends with named zones
 The template stays one markdown file in `zz-cf-lib/templates/`. The cft block grows new keys but the file shape stays familiar.
 Pipeline:
 1. **RAG pre-flight (in-plugin).** Plugin reads `include_sources:` from the cft block, resolves it (vault paths and/or Chroma queries), splices the retrieved text into the system prompt as a named context block. No file writes happen at this stage — RAG context flows straight into the prompt.
 2. **Perplexity stage.** Runs as today: streams body content into the target file, stamps `cf_last_run` / `cf_last_run_model`, appends `# Sources` footer. Output lands in a zone delimited by `<!-- perplexity:start -->` and `<!-- perplexity:end -->` HTML comments.
 3. **Claude editorial stage.** Reads the entire file, runs Claude with a specific edit prompt (`edit-system:` from the cft block, or a separate `claude` block), writes the edited version into a NEW zone delimited by `<!-- claude:start -->` and `<!-- claude:end -->`. The Perplexity zone is preserved untouched as provenance; the Claude zone is what the published doc draws from.
 cft block grows:
 ```yaml
 provider: perplexity
 model: sonar-deep-research
 include-sources:
  - vault: "Tooling/AI-Toolkit/Agentic AI/**"
  - vault: "concepts/Explainers for AI/Agentic Workspaces.md"
  - chroma:
      collection: context-vigilance-corpus
      query: "{{title}} market map prior art"
      n_results: 5
 system: |
  ...
 editor:
  provider: anthropic
  model: claude-opus-4-7
  pass: editorial-restructure
  system: |
    You are editing the Perplexity-drafted market map below into final form...
 ```
 **Pros:**
 - One file per market map; consistent with every other directory template.
 - The zoned-append discipline (Perplexity zone, Claude zone) makes provenance auditable — you can read exactly what each agent produced.
 - Both stages re-runnable independently. Re-run Perplexity to refresh the source data; re-run Claude to refresh the edit without re-burning a deep-research credit.
 - `include-sources` becomes a structured field that handles both vault-path and Chroma-query forms, with the resolution logic centralized in the plugin.
 **Cons:**
 - One file gets long fast. Market maps are already 4-8K words; doubling the body to keep both zones means the on-disk file is hefty. Obsidian handles it, but readability degrades.
 - The Chroma client has to run inside the Obsidian plugin. That means embedding a Chroma HTTP client in `main.js`, configuring the Chroma URL in plugin settings, and handling the failure mode where Chroma is unreachable.
 - The HTML-comment zone markers are fragile — a user editing the file in another pane could break them, and at that point the re-run logic doesn't know where to write.
 - Claude's editorial output sitting in a sibling zone means downstream rendering (the public site) needs to know to draw from the Claude zone, not the Perplexity zone. That's a coupling that didn't exist before.
 ### Option B — Multi-document folder, section per file, with a manifest
 Each market map becomes a folder, not a file. `lost-in-public/market-maps/Quantum Computing is Confusing/` rather than `Quantum Computing is Confusing.md`. Inside the folder:
 ```
 Quantum Computing is Confusing/
 ├── _manifest.md           # frontmatter only — title, lede, banner_image, etc.
 ├── 00-snapshot.md         # Market Snapshot section
 ├── 01-question.md         # The Question this Map Answers
 ├── 02-why-now.md          # Why Now
 ├── 03-map.md              # Map of the Market — Sub-Segments
 ├── 04-lighthouse.md       # Lighthouse Examples
 ├── 05-profiles-superconducting.md   # one file per sub-segment
 ├── 05-profiles-photonic.md
 ├── 05-profiles-trapped-ion.md
 ├── 06-media.md
 ├── 07-dynamics.md
 ├── 08-frontier.md
 ├── 09-adjacent.md
 └── _sources.md            # consolidated sources footer
 ```
 Each `NN-*.md` file is independently re-runnable. The template becomes a *pipeline manifest* that names which agent runs on which file and in what order. The cft format extends to declare per-file stages.
 Pipeline:
 1. **Scaffold.** A new command — `Initialize market map folder` — creates the folder structure from a template manifest, populating each section file's frontmatter with its position in the pipeline.
 2. **Per-section RAG + Perplexity.** Each section file declares its own `include-sources:` and runs its own targeted Perplexity query. Sub-segment files (`05-profiles-*.md`) get the most aggressive RAG injection (vault tools matching the sub-segment).
 3. **Section-level Claude pass.** Claude edits each section file independently against a section-specific edit prompt.
 4. **Roll-up.** A final command stitches the section files into one rendered markdown file for publication, or the public site renders the folder directly via a folder-aware renderer.
 **Pros:**
 - Each section is independently re-runnable. The biggest pain in a one-shot 6K-word generation is "the trapped-ion sub-segment is thin, I'd like to re-run just that one." Folder form makes that natural.
 - RAG context can be tightly scoped per section — the Lighthouse Examples section gets a different vault query than the Media section. With one mega-prompt, RAG context is one-size-fits-all.
 - Per-section editorial passes are cheaper. Editing a 600-word section is faster and tighter than editing a 6K-word document.
 - The folder maps cleanly to how an actual analyst works — they don't write the whole memo in one sitting; they research the sub-segments first, then sequence the framing prose.
 - Agentic orchestration becomes natural: a future orchestrator-agent can iterate over the section files, decide which need refresh, and dispatch jobs in parallel.
 **Cons:**
 - Breaks every existing convention in the perplexed plugin and the vault. The four shipped templates target one file each; making market maps target a folder is a tier shift.
 - The existing `applies-to-paths` glob ([`directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts)) matches a file against a template; we'd need parallel `applies-to-folders` matching, or a new command entirely.
 - Cross-section consistency is harder. The Innovator Profiles section needs to reference the same sub-segment names declared in the Map of the Market section. With separate files, drift across sections becomes likely.
 - The published-on-the-web rendering needs a folder-aware renderer. Astro Knots can do this (it already renders nested content), but every existing market map is a single file and migrating them is its own project.
 - Users edit market maps as one document in their head. A folder breaks that mental model.
 ### Option C — Per-section refresh blocks inside a single file
 Take the multi-`cft` idea floated in the directory-templates doc's *Known limits and open items* and run with it. The template file stays one markdown file. The skeleton declares per-section refresh prompts via dedicated `cft` blocks inline under each H1.
 ```markdown
 # Market Snapshot
 ```cft-section
 provider: perplexity
 model: sonar-deep-research
 include-sources: [...]
 prompt: |
  Write the Market Snapshot section per the instructions below.
 ```
 - Bullet instructions for this section...
 # Innovator Profiles
 ```cft-section
 provider: perplexity
 model: sonar-deep-research
 include-sources:
  - vault: "Tooling/AI-Toolkit/Agentic AI/**"
 prompt: |
  Write the Innovator Profiles section. Use RAG content to name actual
  vault tools where possible.
 ```
 - Bullet instructions...
 ```
 A header-level `cft` block stays at the top of the template declaring the editorial stage (Claude pass) that runs across the whole file at the end.
 Pipeline:
 1. **Section-by-section Perplexity runs.** The plugin walks each `cft-section` block and runs its own targeted query. Section content lands under that H1.
 2. **File-level Claude editorial pass.** After all sections are filled, Claude reads the assembled file and writes the edited version per the file-level `editor:` block.
 **Pros:**
 - One file per market map (keeps Option A's familiarity).
 - Per-section RAG and Perplexity targeting (gets Option B's tight scoping).
 - Re-runnable per section — invoke `Refresh section` on the active section's H1 to re-run just that `cft-section`.
 - Sub-segment-aware RAG: each section's `include-sources` can be tuned to what that section needs.
 **Cons:**
 - The `cft-section` schema is novel; we'd have to design the multi-block parser, the per-section command surface, and the interaction with the existing one-`cft` flow.
 - The file is longer at-rest because it carries every section's prompt inline. This is a feature (visible provenance) and a bug (clutter) at the same time.
 - "Append vs. fill" semantics become per-section. The mode logic in `applyTemplate` gets more states.
 ## The `include_sources` sub-exploration
 This is the design call most likely to set the trajectory of everything else. The user named it as a key argument, and it sits at the intersection of three primitives we already have:
 | Primitive | What it does today | What's missing for `include_sources` |
 |---|---|---|
 | `search-domains:` ([`directoryTemplateService.ts:486`](../../src/services/directoryTemplateService.ts)) | Constrains Perplexity's open-web search to a domain allowlist + a job-board denylist | Doesn't inject content into the prompt; only filters where the model searches |
 | `{{include: name}}` partials ([`directoryTemplateService.ts:83`](../../src/services/directoryTemplateService.ts)) | Splices a named partial's body into the prompt, content-agnostic | No naming/attribution wrapper for the spliced content; no Chroma-query form |
 | Chroma collections (`context-vigilance-corpus` et al, [`lossless-monorepo/CLAUDE.md`](../../../../CLAUDE.md)) | Section-chunked vault content indexed and queryable via the `chroma` MCP server | Only reachable from inside Claude Code today, not from inside Obsidian plugin runtime |
 Proposed shape for `include-sources:` in the cft block:
 ```yaml
 include-sources:
  # Vault path — read the file, splice its body into the prompt as a named,
  # attributed context block. Glob form is allowed; matched files are
  # concatenated with their basenames as the attribution label.
  - vault: "concepts/Explainers for AI/Agentic Workspaces.md"
  - vault: "Tooling/AI-Toolkit/Agentic AI/**"
    max_files: 10            # cap for glob expansion
    body_only: true          # strip each file's frontmatter (default true)
  # Chroma query — run a semantic search against a named collection,
  # splice the top N results into the prompt with their source_path
  # and source_repo_slug as attribution.
  - chroma:
      collection: context-vigilance-corpus
      query: "{{title}} market segmentation"
      n_results: 5
      where:                 # optional metadata filter
        source_repo_slug: "lossless"
  # External URL — fetch and inline. Lowest priority because we can't
  # cache, but useful for one-off canonical references the user wants
  # baked in. Requires explicit opt-in in settings.
  - url: "https://www.lossless.group/projects/gallery/agentic-workspaces"
    max_chars: 8000
 ```
 How the spliced content lands in the prompt:
 ```
 ## Canonical sources (use these as primary attributions where applicable)
 ### Source: Tooling/AI-Toolkit/Agentic AI/Crew AI.md
 <body of that vault file>
 ### Source: concepts/Explainers for AI/Agentic Workspaces.md
 <body of that vault file>
 ### Source: Chroma (context-vigilance-corpus) — n=3
 <top chunks, each labeled with source_path>
 ## Search the open web for additional sources beyond the canonical set above.
 ```
 This shape solves the "named, attributed" problem (the model knows what it's quoting and can cite by vault path), keeps glob expansion bounded (`max_files`), and degrades gracefully when Chroma is unreachable (skip the chroma entries with a warning notice; vault entries still resolve).
 Open sub-questions:
 - **Does `include-sources` belong at the template level or the section level?** Template-level is simpler. Section-level is more powerful but only matters if we go Option B or C.
 - **How big can the canonical-sources block grow before we OOM the prompt?** `sonar-deep-research` has generous context but not infinite. A vault glob matching 100 files is too much. The `max_files` cap matters; defaults should be conservative (5-10).
 - **Is the Chroma client an in-plugin dependency or an out-of-plugin pre-flight?** If we ship a Chroma client in `main.js`, plugin size grows and we need to surface Chroma URL configuration in settings. If we keep RAG pre-flight outside the plugin (a CLI that writes a `_rag-context.md` partial the plugin then includes), the plugin stays lean but the workflow gains an extra step.
 ## "Perplexity and Claude do not overwrite each other until editing"
 This is the design heuristic that distinguishes Option A's zoned-append from a naive sequential pipeline. Three implementations of the heuristic, in increasing strictness:
 1. **Soft separation (zones).** Perplexity writes into one zone, Claude writes into another. Both zones live in the same file. Provenance is preserved; the published doc draws from the Claude zone but the Perplexity zone is readable.
 2. **Hard separation (files).** Perplexity writes its draft to `<map>.perplexity.md`. Claude reads that file and writes its edit to `<map>.md` (or vice versa, depending on which is canonical). The original is preserved untouched.
 3. **Strict additive (no edits, only appendages).** Both agents append; neither edits. The final document is a stitched sequence of their outputs with explicit attribution. Useful if we want to preserve "the analyst's draft" and "the editor's reorganization" as separate readable artifacts.
 The strict additive form (3) has appeal — it makes the multi-agent collaboration audit-trail-perfect — but it's hostile to the actual editorial goal, which is *to merge the two voices into one coherent document*. Soft separation (1) hits the right tradeoff for a v1: provenance is preserved, the final published form is one voice, and re-running either stage doesn't destroy the other's work.
 The "until editing" clause is where the user's framing gets sharp. **The editing stage is the only point at which the two agents' outputs are consolidated.** Before that, they're parallel artifacts. This is why Option A's zoned append is the most faithful interpretation: Perplexity and Claude do their work in parallel zones; the editing stage (which Claude itself performs) is when the consolidation happens, producing the canonical Claude zone.
 ## Live example surfacing the timeout issue
 The first real-world run of the v1 `market-map-profile` template — `lost-in-public/market-maps/Humanoid Robots and their Input Industries.md`, executed 2026-05-26 — produced a ~7,500-word draft that terminated mid-sentence inside *Frontier and Open Questions* because the directory-template runtime's wall-clock `AbortController` fired before the stream completed. That truncation is the motivating use case for the **per-template `request-timeout-ms:` cft-block override** we shipped on the same day as this exploration (default bumped to 30 min, market-map-profile overridden to 40 min). The pressure-relief valve is in place; the structural fix — porting the idle-timeout discipline from `perplexityService.ts:659-668` into `streamPerplexityToFile` — is captured as its own issue at [[Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams]] and feeds back into this exploration's *Open items* below.
 This is the shape of validation we want for the multi-stage pipeline overall: ship the lean v1, run it on a real document, surface the structural gaps as their own context-v issues (where they get the attention they deserve), and let the issues feed into the eventual spec.
 ## Findings so far
 From this exploration session, the [partials-and-preambles issue](../issues/Partials-And-Preambles-For-Perplexed-Templates.md) the architecture builds on, and the [wall-clock-timeout issue](../issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md) surfaced by the first real market-map run:
 - **The `{{include: name}}` partial primitive is already structurally close to what `include-sources` needs.** What's missing is (a) the per-source attribution wrapper, (b) the Chroma-query form, and (c) the multi-source named-block formatting in the prompt. The partial loader's depth and cycle guards transfer directly.
 - **The cft config is already a structured YAML surface.** Adding `include-sources:` (a list) and `editor:` (an object) doesn't break the existing parser; the four shipped templates remain valid because they don't declare either key.
 - **`search-domains:` and `include-sources:` are complementary, not redundant.** Domains constrain where Perplexity searches the open web; include-sources injects canonical content into the prompt directly. A future market-map run would likely use both: include-sources to seed our canonical entries, search-domains to keep open-web search restricted to credible analyst sources (Bloomberg, FT, sector trade press, founder blogs).
 - **Claude can already stream to a file** ([`claudeService.ts:48`](../../src/services/claudeService.ts) `queryClaude`). The editorial stage doesn't need a new transport — it needs a different prompt construction path and a different write-zone strategy.
 - **The `editor:` block can be opt-in.** Templates that don't declare it stay single-stage and behave exactly as they do today. The four shipped templates would stay single-stage; only `market-map-profile` and any future research-heavy templates would declare an `editor:`.
 - **Chroma reachability from inside Obsidian is the long pole.** The MCP server only exists in Claude Code; the Obsidian plugin runs in Electron and needs its own HTTP client to a Chroma instance, OR we keep RAG pre-flight as an out-of-band step that writes a context partial the plugin then includes.
 ## Tentative direction
 Lean toward **Option A (zoned single-file appends) with the `include-sources:` and `editor:` keys added to the cft schema**, and keep RAG pre-flight as an *in-plugin* feature with Chroma queries optional (vault-path includes always work; Chroma includes degrade to a Notice when unreachable).
 Reasoning:
 - Option A is the smallest delta from the existing template surface. The four shipped templates remain unchanged; market-map-profile gains two new keys.
 - Vault-path includes get us most of the RAG win without the Chroma-reachability problem. The dominant value of "include our canonical tools and concepts" is satisfied by globbed vault includes; Chroma adds discoverability over the wider corpus but isn't load-bearing for the first iteration.
 - Zoned appends preserve the "Perplexity and Claude do not overwrite each other until editing" heuristic literally: each agent writes its zone, the editing stage consolidates. If we want hard separation later, we can promote the zones into separate files without redesigning the schema.
 - Option B (multi-doc folder) is the right end-state for sufficiently complex maps but is a tier shift that should be earned, not pre-emptively designed.
 - Option C (per-section cft blocks) is the right *generalization* but is also the right way to over-design. Defer until we've shipped Option A and felt the actual pain.
 Defer for a follow-up spec:
 - The Chroma in-plugin client work. Ship vault-path includes first; revisit Chroma after we know whether the in-plugin client size cost is worth the discoverability gain.
 - The multi-doc folder form (Option B). Track as a possible v3 once we have 10+ market maps and can feel the cross-section consistency pain.
 - The per-section `cft-section` block (Option C). Track as a possible v2 alternative if Option A's zoned appends turn out to be insufficient for sub-segment scoping.
 ## Open items before we promote to a spec
 - [ ] Confirm zone delimiter shape. HTML comments are tolerant in Obsidian and Astro Knots renderers; named-anchor headings (`<!-- region: perplexity -->`) work too. Pick one and document.
 - [ ] Decide whether `editor:` is a block or a list. A list (`editor: [{ provider: anthropic, ... }, { provider: anthropic, pass: copyedit, ... }]`) would let us chain multiple Claude passes; a block keeps the single-pass v1 simpler.
 - [ ] Spec the `include-sources` resolution order and de-duplication policy. If a vault glob and a Chroma query both surface the same source file, do we splice it twice?
 - [ ] Decide attribution-block format. Does each included source get a top-level H2 in the prompt context, or a fenced block, or YAML-front-matter-wrapped? Trade off model-comprehension against prompt-token economy.
 - [ ] Decide what `cf_last_run` stamps look like for a multi-stage run. One timestamp per stage (`cf_last_run_perplexity`, `cf_last_run_editor`)? A composite? This matters for the "is this map stale?" query later.
 - [ ] Verify Chroma collections are queryable from a non-MCP HTTP client. The [chroma-local skill](../../../../context-v/skills/chroma-local/SKILL.md) covers `ChromaClient`/`HttpClient` setup; should be a 30-minute spike.
 - [ ] Resolve [[Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams]] before the multi-stage spec lands — the current wall-clock primitive cuts off long stage-2 (Perplexity) runs and will cut off long stage-3 (Claude editorial) runs too, since the editor's per-section reasoning on a 7-8K-word draft is itself a long generation. The structural fix (idle-timeout discipline) belongs upstream of multi-stage, not as part of it.
 ## Outcome
 Open. When this resolves, the expected artifacts are:
 1. A spec in [`context-v/specs/`](../specs/) — `Multi-Stage-Templates-with-Include-Sources-and-Editor.md` (working title) — that pins down the cft schema additions, the zone-delimiter format, the resolution order for `include-sources`, the editorial-stage contract, and the migration path for existing templates.
 2. A prompt in [`context-v/plans/`](../plans/) — chunked implementation steps for the spec.
 3. The `market-map-profile` template updated to use the new keys, treated as the reference implementation.
 4. A v2 follow-up exploration in this folder once we've felt how Option A actually behaves with one or two real market maps generated through it.
 ## Related
 - [[market-map-profile]] — the lean v1 template this exploration extends
 - [[Using-Files-as-Prompt-Outlines]] — the original spec that motivated directory templates
 - [[Partials-And-Preambles-For-Perplexed-Templates]] — the issue that established the splice primitive `include-sources` builds on
 - [[Getting-Claude-to-Respond-With-Research]] — prior work on Claude as a research-aware agent
 - [[search-lossless-corpus]] — the Claude-Code-side skill that already encodes the four-collection RAG discipline; the in-plugin RAG flow is the Obsidian analog
 - [`lossless-monorepo/CLAUDE.md`](../../../../CLAUDE.md) — the canonical description of the four Chroma collections this exploration's RAG stage would draw on
--- a/context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md
+++ b/context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md
@ -0,0 +1,129 @@
 ---
 title: Wall-clock timeout cuts off long deep-research streams
 lede: "The directory-template runtime caps every stream by total wall-clock duration, but the legacy modal flow already moved to per-chunk idle-timeout discipline two iterations ago — and the discrepancy is now actively truncating analyst-grade market-map drafts mid-sentence."
 date_created: 2026-05-26
 date_modified: 2026-05-26
 date_resolved: 2026-05-26
 authors:
  - Michael Staton
 augmented_with:
  - Claude Opus 4.7 (1M context)
 semantic_version: 0.0.0.2
 type: issue
 status: resolved
 target_repo: perplexed
 tags:
  - Issue-Resolution
  - Perplexed
  - Streaming-Timeouts
  - Deep-Research
  - Directory-Templates
 related:
  - "[[market-map-profile]]"
  - "[[Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG]]"
  - "[[Partials-And-Preambles-For-Perplexed-Templates]]"
 ---
 # Wall-clock timeout cuts off long deep-research streams
 ## Symptom
 Running `market-map-profile` on `lost-in-public/market-maps/Humanoid Robots and their Input Industries.md` produced a ~7,500-word draft that terminated mid-sentence inside the *Frontier and Open Questions* section:
 ```
 Will the evolution of robot-as-a-service (R
 ```
 The trailing parenthesis is the last byte written. The *Adjacent Concepts and Maps* section — the final heading in the template skeleton — never appeared. The frontmatter stamps (`cf_last_run`, `cf_last_run_model`) landed correctly, so the run did complete its `processFrontMatter` post-step; what was lost was the streamed body content that hadn't yet arrived when the `AbortController` fired.
 The Humanoid Robots run is the first observable instance of this specific cut-off shape, but the pattern is structural — it will reproduce on any sufficiently long deep-research generation run through the directory-template flow.
 ## Diagnosis
 Two different streaming primitives live in this codebase, with two different timeout disciplines:
 **Directory-template flow** ([`src/services/directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts) `streamPerplexityToFile`, the function `market-map-profile` and the four other shipped templates run through):
 ```ts
 const controller = new AbortController();
 const timer = activeWindow.setTimeout(() => controller.abort(), timeoutMs);
 ```
 A **single wall-clock `setTimeout`** is armed at the moment of fetch. After `timeoutMs` elapses — regardless of whether the stream is actively producing bytes — `controller.abort()` fires and the in-flight `reader.read()` throws. The catch sets `truncated = true` and falls through to a final flush of whatever streamed so far. The plugin-level default was `600_000` ms (10 min) before today's fix; analyst-grade deep-research runs routinely run 15-25 min, so the cut-off was inevitable on long templates.
 **Legacy modal flow** ([`src/services/perplexityService.ts:659`](../../src/services/perplexityService.ts), `PerplexityModal`):
 ```ts
 const STREAM_IDLE_TIMEOUT_MS = isDeepResearch ? 270_000 : 90_000;
 const readWithIdleTimeout = (): Promise<...> => {
    let timer: number | undefined;
    const timeout = new Promise<never>((_, reject) => {
        timer = window.setTimeout(() => {
            reject(new Error(`stream went idle for ${...}s ...`));
        }, STREAM_IDLE_TIMEOUT_MS);
    });
    return Promise.race([reader.read(), timeout]).finally(() => {
        if (timer !== undefined) window.clearTimeout(timer);
    });
 };
 ```
 **Per-chunk idle timeout.** A fresh `setTimeout` is armed and racing each `reader.read()` call. As long as bytes keep arriving, the timer keeps getting cleared and re-armed. The stream is only killed if it goes *quiet* for 270 s (deep-research) or 90 s (normal). Total wall-clock duration is unbounded.
 The legacy modal moved to this pattern because the same problem hit users there first — but the directory-template flow was forked from an earlier iteration of the streaming code and never received the idle-timeout backport. The legacy `PerplexityModal` and `streamPerplexityToFile` now disagree on how to time-bound a Perplexity stream, and the directory-template flow has the strictly weaker discipline.
 ## What we shipped today (partial fix)
 Not a structural fix — a pressure-relief valve. Two changes:
 1. **Bumped the plugin-level default** ([`main.ts:333`](../../main.ts)) from `600_000` ms (10 min) to `1_800_000` ms (30 min). The settings-pane description was updated to call out the override and the cost framing (`$10-$50 of analyst time per good output is worth waiting for`).
 2. **Added a per-template override** — `request-timeout-ms:` in the cft block. Resolution code lives in [`directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts) just before the `streamPerplexityToFile` call; accepts number or numeric-string, silently falls back on non-positive / non-numeric values.
 3. **`market-map-profile.md`** declares `request-timeout-ms: 2400000` (40 min) with an inline comment explaining the budget; the other four shipped templates inherit the plugin-level 30-min default.
 4. **Docs** — [`docs/directory-templates.md`](../../docs/directory-templates.md) gained a *Per-template timeout override* section with override semantics and a "when to bump" checklist; [`src/docs/templates/README.md`](../../src/docs/templates/README.md) calls out the key in the cft-block list.
 This buys headroom. It does not fix the structural problem: any sufficiently long deep-research run will still hit the wall eventually. The 40-min cap is a *guess* about how long the longest reasonable market-map should take, not a property derived from the stream's actual behavior.
 ## Why this is only a partial fix
 The wall-clock timeout fails in two distinct shapes that the idle-timeout pattern handles correctly:
 **Shape 1 — slow but healthy stream.** Deep-research generations on long templates may sustain a slow trickle of tokens for 30-45 minutes. The wall-clock cap kills them at the ceiling regardless of whether they're still producing. The idle-timeout pattern lets them complete naturally as long as some byte arrives every N seconds.
 **Shape 2 — silently stalled stream.** Conversely, a stream may go quiet at minute 3 (Perplexity rate-limit, socket close, upstream stall) and the wall-clock cap won't notice until minute 30. The user stares at an empty file, watching the spinner spin, for 27 unnecessary minutes. The idle-timeout pattern surfaces the failure within `STREAM_IDLE_TIMEOUT_MS` seconds — fast feedback when something is genuinely wrong.
 Both shapes are real. The wall-clock pattern punishes the healthy-but-slow case while tolerating the stalled-but-silent case. The idle-timeout pattern inverts both — slow-but-healthy completes; stalled-but-silent fails fast.
 ## Proposed structural fix
 Port the idle-timeout pattern from [`perplexityService.ts:659-668`](../../src/services/perplexityService.ts) into `streamPerplexityToFile`. Concretely:
 1. Replace the single wall-clock `setTimeout(controller.abort, timeoutMs)` with a per-chunk `readWithIdleTimeout()` that wraps each `reader.read()` in a `Promise.race` against a fresh timeout.
 2. Choose idle-timeout values consistent with the existing modal flow: `270_000` ms (4.5 min) for deep-research models, `90_000` ms (1.5 min) for normal models. Detect deep-research from the resolved model name string (`/deep-research/i`), the same way `perplexityService.ts` does.
 3. Retain the cft-block override key, but rename it to `stream-idle-timeout-ms:` for accuracy. Templates that have been declaring `request-timeout-ms:` need a compatibility alias for one release; document the rename in the changelog entry.
 4. Optionally retain a generous absolute wall-clock ceiling (60 min or 120 min) as a sanity backstop — but that's belt-and-suspenders, not load-bearing. The idle timeout is doing the actual safety work.
 The diff is moderate — `streamPerplexityToFile` is ~140 lines today; the refactor touches roughly the first 40 of those (the timer setup and the `reader.read()` call inside the loop). The post-stream cleanup pipeline (`wrapThinkBlocks`, `processContentWithImages`, `buildSourcesFooter`) is unaffected.
 ## Resolution (2026-05-26)
 The structural fix shipped the same day as the issue was filed. See [`changelog/2026-05-26_02.md`](../../changelog/2026-05-26_02.md) for the full ship note.
 What landed:
 - [x] **The port itself.** `streamPerplexityToFile` now wraps each `reader.read()` in a `readWithIdleTimeout()` race, ported from `perplexityService.ts:659-668`. Signature changed from `timeoutMs: number` to `timeouts: { idleMs: number; ceilingMs: number }`. AbortController retained for cancel + ceiling.
 - [x] **Dual-key naming decision.** Kept `request-timeout-ms:` as the legacy key (now semantically the *absolute wall-clock ceiling*) and added `stream-idle-timeout-ms:` as the new key (the per-chunk idle timer, primary safety). No rename, no compatibility alias needed — existing template values for `request-timeout-ms:` still work and now mean exactly what their name suggests.
 - [x] **Ceiling-vs-idle defaults.** Idle defaults match the legacy modal flow: 270s for deep-research models (`/deep-research/i` on resolved model name), 90s otherwise. Ceiling defaults to `settings.requestTimeoutMs` (30 min); explicit `0` in either settings or cft disables the ceiling entirely.
 Deferred (now their own follow-ups, not blocking):
 - [ ] **Cross-service audit.** The same wall-clock pathology may live in the Gemini service, the LM Studio service, and the Claude streaming flows. The idle-timeout discipline should be the house style across all of them. Track as its own issue.
 - [ ] **Settings-pane exposure of idle defaults.** Today the 270s/90s values are hardcoded in `streamPerplexityToFile` (matching `perplexityService.ts`). Exposing them as plugin settings is a follow-up if we ever need to tune them without a code change.
 - [ ] **Robustness of deep-research detection.** `/deep-research/i` against the resolved model name is the current heuristic — same as `perplexityService.ts`. Revisit if Perplexity ever ships a long-running model under a different naming convention.
 ## Related
 - [[market-map-profile]] — the template that surfaced the bug
 - [[Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG]] — the broader exploration this issue feeds findings back into; the idle-timeout refactor is listed there as an open item
 - [[Partials-And-Preambles-For-Perplexed-Templates]] — the architecture-review structure of that issue is the precedent this one follows
 - [`src/services/perplexityService.ts`](../../src/services/perplexityService.ts) lines 659-668 — the idle-timeout implementation in the legacy modal flow that we're proposing to port
 - [`src/services/directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts) `streamPerplexityToFile` lines 499-635 — the wall-clock implementation in the directory-template flow that the port replaces
--- a/docs/directory-templates.md
+++ b/docs/directory-templates.md
@ -79,7 +79,7 @@ # User Notes
 ### The three zones
 1. **Frontmatter** (between `---` lines at the top) — carries `title`, `applies-to-paths` (array of globs), and an optional `description`. The runtime uses `applies-to-paths` to match a template to a target file.
-2. **`cft` block** (a code fence with language `cft`) — YAML carrying `provider`, `model`, optional `search-recency`, `return-citations`, `return-images`, and a multi-line `system:` prompt. Everything **above** the `cft` block is dropped from the request.
+2. **`cft` block** (a code fence with language `cft`) — YAML carrying `provider`, `model`, optional `search-recency`, `return-citations`, `return-images`, optional `stream-idle-timeout-ms:` and `request-timeout-ms:` (per-template timeout overrides — see *Per-template timeout overrides* below), optional `max-tokens:` (per-template output-token budget — see *Per-template max-tokens override* below), and a multi-line `system:` prompt. Everything **above** the `cft` block is dropped from the request.
 3. **Heading skeleton** — the markdown structure between the `cft` block's closing fence and the first `***` divider. This becomes the user prompt. Bullets under each heading are *instructions to the model*, not literal output. Everything **below** the first `***` is excluded from the request.
 ### Interpolation tokens
@ -140,7 +140,7 @@ ### Frontmatter stamps
 ## Shipped templates
-Four templates ship inlined into `main.js` (via esbuild's `.md` text loader) and are seeded into the user's vault on first plugin load.
+Five templates ship inlined into `main.js` (via esbuild's `.md` text loader) and are seeded into the user's vault on first plugin load.
 | File | Targets | Model | Notes |
 |---|---|---|---|
@ -148,6 +148,7 @@ ## Shipped templates
 | `vocabulary-profile.md` | `Vocabulary/**` | `sonar-pro` | Term definitions with disambiguation through an innovation-consulting lens. |
 | `source-profile.md` | `Sources/**` | `sonar-pro` | Profiles of trusted sources — books, people, channels, publications, journals, reports, events. Type-aware: the system prompt enumerates seven canonical types and the model picks one from frontmatter signals (`youtube_channel_url` → channel, `aliases` → likely book, etc.). Each section has per-type bullet shapes. |
 | `toolkit-profile.md` | `Tooling/**` | `sonar-pro` | Profiles of tools, products, platforms, frameworks. |
 | `market-map-profile.md` | `lost-in-public/market-maps/**`, `market-maps/**` | `sonar-deep-research` | Analyst-grade market-map drafts. Dual flavor: Known Category (Quantum Computing, Humanoid Robots) or Thesis-Driven (e.g., Neural Network Hardware as Brains for Robotics) traversing adjacent categories. Lean v1, single-stage. Skips image return by design — deep-research's image metadata is unreliable and market-map imagery is generated separately (Ideogram → frontmatter `banner_image` / `portrait_image` / `square_image`). Multi-stage v2 (RAG pre-flight to inject canonical Lossless tools/concepts as context + Claude editorial pass to emit `[[wikilink]]`s) is deferred — see the User Notes zone of the template for the roadmap. |
 `source-profile` is the trickiest because `Sources/` is genuinely heterogeneous. The solution is one template, type-conditional content. Books also trigger Google Books URL handling: frontmatter `google_books_url` is used if present, otherwise the model finds it; either way the URL is harvested into frontmatter post-generation via regex, so subsequent runs skip the search.
@ -199,6 +200,85 @@ ## Commands
 ---
 ## Per-template timeout overrides
 The runtime applies **two complementary timers** to every Perplexity stream (see [`directoryTemplateService.ts`](../src/services/directoryTemplateService.ts) `streamPerplexityToFile`):
 | Timer | What it does | Default |
 |---|---|---|
 | **Stream-idle timeout** (per-chunk) | Each `reader.read()` is raced against a fresh timer. As long as bytes keep arriving, the timer is cleared and re-armed — the stream only dies if it goes **quiet** for the idle duration. | `270_000` ms (4.5 min) for deep-research models (matched by `/deep-research/i` against the resolved model name); `90_000` ms (1.5 min) otherwise. Matches the legacy modal flow in `perplexityService.ts:659`. |
 | **Wall-clock ceiling** (absolute) | An optional `AbortController` deadline armed once at fetch time. Belt-and-suspenders backstop. | Plugin-level `settings.requestTimeoutMs` (default **30 min**). Set to **0** in settings to disable; the idle timer alone is responsible for safety. |
 The idle timer is the primary safety mechanism. It catches the two pathologies that a single wall-clock timer handles poorly:
 - **Silently stalled stream** (Perplexity rate-limit, socket close, upstream stall): the idle timer surfaces the failure within `idleMs` seconds rather than making the user wait the full ceiling.
 - **Slow-but-healthy stream**: as long as some byte arrives every `idleMs` seconds, the stream completes naturally — it does not get cut off mid-sentence at the ceiling regardless of whether it's still producing.
 Templates can override either or both via their `cft` block:
 ```cft
 provider: perplexity
 model: sonar-deep-research
 stream-idle-timeout-ms: 270000   # per-chunk idle (default already 270s for deep-research)
 request-timeout-ms: 2400000      # absolute ceiling (40 min)
 system: |
  ...
 ```
 Override semantics:
 - A numeric value or a numeric string both work.
 - For `stream-idle-timeout-ms:`: non-positive / non-numeric values fall through to the model-class default (270s deep, 90s normal).
 - For `request-timeout-ms:`: an explicit `0` disables the ceiling entirely (idle-only). Non-numeric or missing falls through to `settings.requestTimeoutMs`.
 - Overrides apply only to that template's runs.
 - The runtime tolerates mid-stream truncation gracefully: a stream killed by either timer leaves a partial draft on disk with `truncated: true` flagged in the result and a Notice telling the user to re-run.
 When to override the defaults:
 - `sonar-deep-research` on a multi-section analyst template (market maps, in-depth research reports, sector overviews) routinely emits 6–8K-word bodies. The 270s idle default lets the slow trickle complete naturally — no override needed for the *idle* key in most cases.
 - Bump `request-timeout-ms:` (or set to `0`) when you want truly unbounded healthy runs and trust the idle timer to catch stalls.
 - Lower `stream-idle-timeout-ms:` for short non-deep-research templates that should fail fast (the 90s default already does this for them).
 - Templates that declare a large `include-sources:` block (per the [multi-stage exploration](../context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md)) when that feature ships — RAG context inflation adds first-byte latency; consider bumping `stream-idle-timeout-ms:` if first-byte routinely exceeds the default.
 Market-map-profile ships with `request-timeout-ms: 2400000` (40 min ceiling) as a safety cap on absolutely-runaway runs; the per-chunk idle timer (270s, inherited) is what actually keeps the stream healthy. Concept, vocabulary, source, and toolkit profiles declare neither key and inherit `settings.requestTimeoutMs` for the ceiling and the 90s idle default — they all finish comfortably under both.
 ---
 ## Per-template `max-tokens:` override
 Perplexity's default output-token cap silently truncates long analyst-grade templates by ending the stream **cleanly** with `finish_reason: "length"` mid-skeleton. The symptom is hostile: the run looks like a healthy completion (sources footer renders, no Notice, no truncation warning), but the back half of the section skeleton never appears. Default is approximately 8,192 tokens for `sonar-deep-research` (~6,000 English words).
 Templates can override the default by declaring `max-tokens:` inside their `cft` block:
 ```cft
 provider: perplexity
 model: sonar-deep-research
 max-tokens: 24000   # ~18K-word budget
 system: |
  ...
 ```
 Override semantics:
 - A numeric value or a numeric string both work; non-positive / non-numeric values are silently ignored and Perplexity's default applies.
 - The override is per-template. Other templates continue to use Perplexity's default.
 - Perplexity may itself cap the request below your declared `max-tokens:` based on the model's context window minus the prompt length. If you request more than the model can deliver, Perplexity returns the model's actual cap; your override is the ceiling you're willing to accept.
 When to bump above the default:
 - **Any analyst-grade template that asks for >6K words of body.** `market-map-profile` and `standards-and-specs-profile` both ship with `max-tokens: 24000` for this reason.
 - **Templates with deeply nested skeletons** (3+ heading levels) — the model rations its token budget across the skeleton, and a deep skeleton means more sections competing for the same budget.
 - **Templates that demand multiple cards per section** (e.g., the three-tier adoption skeleton in `standards-and-specs-profile` with deeper implementation cards). Each card takes ~150-300 tokens; 15-30 cards × 200 tokens = 3,000-6,000 tokens just for the card sections.
 **The diagnostic to distinguish max-tokens truncation from wall-clock truncation:**
 | Symptom | Cause |
 |---|---|
 | Mid-sentence cutoff (literally ends with a partial word or trailing punctuation); no sources footer; user sees "stream went idle" or no Notice at all | Wall-clock timer (`request-timeout-ms:` ceiling) or idle timer (`stream-idle-timeout-ms:`) fired during streaming |
 | Clean section-end cutoff (last byte is a paragraph break or full sentence); **sources footer renders correctly with all citations**; no Notice | Perplexity `max_tokens` cap; the stream completed cleanly via `finish_reason: "length"` |
 The wall-clock and idle timers are streaming-time controls; `max-tokens` is an output-budget control. They are independent — a template can hit either, neither, or both. Set both knobs generously for thorough deep-research templates.
 ## Settings
 `Plugin settings → Directory templates`:
--- a/main.ts
+++ b/main.ts
@ -20,6 +20,8 @@ import { ArticleGeneratorModal } from './src/modals/ArticleGeneratorModal';
 import { TextEnhancementModal } from './src/modals/TextEnhancementModal';
 import { TextEnhancementWithImagesModal } from './src/modals/TextEnhancementWithImagesModal';
 import { DirectoryTemplatePickerModal } from './src/modals/DirectoryTemplatePickerModal';
 import { DirectoryTemplateRunModal } from './src/modals/DirectoryTemplateRunModal';
 import type { TemplateRunChoice } from './src/modals/DirectoryTemplateRunModal';
 import { FolderPickerModal } from './src/modals/FolderPickerModal';
 import { BatchConfirmModal } from './src/modals/BatchConfirmModal';
@ -328,7 +330,7 @@ Structure the article as follows:
        { name: 'image-placement', when: 'return-images' },
    ],
    directoryTemplatesFrontmatterWhitelist: ['title', 'og_description', 'tags', 'og_image'],
-    directoryTemplatesRequestTimeoutMs: 300000,
+    directoryTemplatesRequestTimeoutMs: 1800000,
    // Find images for selection
    findImagesMaxImages: 3
@ -1242,15 +1244,20 @@ export default class PerplexedPlugin extends Plugin {
        const dirSettings: DirectoryTemplateSettings = this.buildDirectoryTemplateSettings();
-        new DirectoryTemplatePickerModal(this.app, matching, (chosen) => {
+        // Load every matching template up front so the run modal can show
-            void (async () => {
+        // each one's cft model as the default in the model selector.
-                const parsed = await loadDirectoryTemplate(this.app, chosen.file);
+        const choices: TemplateRunChoice[] = [];
-                if (!parsed) {
+        for (const tf of matching) {
-                    new Notice('Template parse error: missing or malformed cft block.');
+            const parsed = await loadDirectoryTemplate(this.app, tf.file);
-                    return;
+            if (parsed) choices.push({ template: parsed, title: tf.title });
-                }
+        }
-                await applyDirectoryTemplate(this.app, dirSettings, target, parsed);
+        if (choices.length === 0) {
-            })();
+            new Notice('Template parse error: missing or malformed cft block.');
            return;
        }
        new DirectoryTemplateRunModal(this.app, choices, (template, model) => {
            void applyDirectoryTemplate(this.app, dirSettings, target, template, { modelOverride: model });
        }).open();
    }
@ -2033,9 +2040,9 @@ class PerplexedSettingTab extends PluginSettingTab {
        new Setting(containerEl)
            .setName('Request timeout (ms)')
-            .setDesc('Maximum time to wait for the perplexity deep research response. Default 300000 (5 min).')
+            .setDesc('Maximum wall-clock time to wait for a Perplexity response. Default 1800000 (30 min) — generous because deep-research runs on long analyst-grade templates routinely take 15-25 min and the $10-$50 of value per good output is worth waiting for. Individual templates may override this per-template via `request-timeout-ms:` in their cft block.')
            .addText(text => text
-                .setPlaceholder('300000')
+                .setPlaceholder('1800000')
                .setValue(String(this.plugin.settings.directoryTemplatesRequestTimeoutMs))
                .onChange(async (value: string) => {
                    const n = parseInt(value, 10);
--- a/manifest.json
+++ b/manifest.json
@ -1,9 +1,9 @@
 {
 	"id": "perplexed",
 	"name": "Perplexed",
-	"version": "0.2.0",
+	"version": "0.3.1",
 	"minAppVersion": "1.8.10",
-	"description": "Generate source-cited research content from Perplexity, Anthropic Claude, Google Gemini, Perplexica, or local LM Studio — directly into your notes.",
+	"description": "Generate source-cited research content from Perplexity, Anthropic Claude, Google Gemini, Perplexica (now Vane), or local LM Studio — directly into your notes.",
 	"author": "The Lossless Group",
 	"authorUrl": "https://lossless.group",
 	"fundingUrl": "https://buymeacoffee.com/losslessgroup",
--- a/package.json
+++ b/package.json
@ -1,7 +1,7 @@
 {
 	"name": "perplexed",
-	"version": "0.2.0",
+	"version": "0.3.1",
-	"description": "Generate source-cited research content from Perplexity, Anthropic Claude, Google Gemini, Perplexica, or local LM Studio — directly into your notes.",
+	"description": "Generate source-cited research content through prompts and templates. From Perplexity, Anthropic Claude, Google Gemini, Perplexica (now Vane), or local LM Studio — directly into your notes.",
 	"main": "main.js",
 	"scripts": {
 		"dev": "node esbuild.config.mjs",
--- a/release-notes/0.2.1.md
+++ b/release-notes/0.2.1.md
@ -0,0 +1,69 @@
 **Directory-template runs with `sonar-deep-research` were quietly discarding ~99% of every response — the streaming reader was taking the wrong field. 0.2.1 fixes that, keeps citations even when a long research stream drops mid-flight, adds a per-run model picker to the run dialog, and rebuilds the toolkit-profile template so the model searches the product's real name and triages SEO slop instead of citing it.**
 ## Why care?
 If you generate research notes with Perplexed's directory templates — the "Apply directory template to current file" workflow — 0.2.1 is the release where the deep-research path actually works.
 **`sonar-deep-research` runs now keep their output.** Perplexity's deep-research model does all its work server-side, then delivers the entire finished document — often 10,000+ characters — in the *first* event of its streaming response, in a field the plugin wasn't reading. The reader took `delta.content` (correct for the lighter `sonar` models, which stream token-by-token); on a deep-research response that field holds only a ~75-character tail fragment of a 10,000-character report. The other 99% arrived on the wire and was thrown away. `toolkit-profile` shipped with `sonar-deep-research` as its *default* model — so it was the default that was broken. Fixed.
 **Citations survive a dropped connection.** Deep research holds a stream open for minutes; a WiFi roam or idle socket mid-stream used to discard everything, including the citations deep research delivers up front. A cut-off run now keeps whatever content and sources arrived, writes the citations footer anyway, and tells you it was truncated.
 **You pick the model when you run.** "Apply directory template" now opens a run dialog with a model dropdown — and the loading toast finally names the model actually running, instead of a hardcoded "deep research" string.
 ## What's new
 ### Deep-research streaming — read the snapshot, not the fragment
 Both streaming paths now treat Perplexity's cumulative `message.content` snapshot as the source of truth and append only the new tail.
 | Model | How it streams | Old reader | 0.2.1 reader |
 |---|---|---|---|
 | `sonar`, `sonar-pro` | Token-by-token deltas | Worked | Works |
 | `sonar-deep-research` | Whole document in event 1, then tiny deltas | ~75 of ~10,234 chars | Full document |
 ### Citations that survive a timeout
 A dropped or timed-out stream no longer discards the run. The runner returns its partial content and the sources it already received, writes the citations footer, and shows a "stream cut off — saved partial content with N sources, re-run to complete" notice. The deep-research idle timeout was lengthened (90s → 270s) since deep research legitimately holds the socket silent while it researches.
 ### A model picker in the run dialog
 "Apply directory template to current file" opens a dialog with a template dropdown and a **model dropdown** — `sonar-pro`, `sonar-reasoning-pro`, `sonar`, `sonar-reasoning`, `sonar-deep-research`. The model defaults to the template's `cft` `model:` and overrides it for that run only. The `cf_last_run_model` frontmatter stamp records the model that actually ran.
 ### Domain scoping — `cf_search_domains`
 New `search_domain_filter` support: a `cf_search_domains:` list in a target file's frontmatter (entity-specific) or a `search-domains:` key in a template's `cft` block (generic). Plain entries allowlist, a leading `-` denylists, capped at 10. Job boards (Indeed, Glassdoor, ZipRecruiter, USAJobs) are denied automatically — a job listing is never a product source. Allowlisting is a last resort for collision-heavy names, not the everyday workflow.
 ### The toolkit-profile template, rebuilt
 The shipped `toolkit-profile` template's system prompt was overhauled: it anchors the entity on its real name (`{{basename}}`) instead of the scraped marketing tagline; it tells the model to triage source quality — editorial authority over SEO ranking, judge each page on its merits — rather than trusting or hard-banning by domain; it adds a hard "search, don't recall" instruction and per-section length ceilings. Default model is now `sonar-pro`, with `sonar-deep-research` one dialog click away.
 ### Longer request timeout
 The directory-template request timeout default moved 5 minutes → 10, so a `sonar-deep-research` run has room to finish.
 ## Verify the release assets
 Every release asset is signed via [GitHub's artifact attestation system](https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations). Before manual install, verify provenance:
 ```bash
 gh attestation verify main.js --repo lossless-group/perplexed-plugin
 gh attestation verify manifest.json --repo lossless-group/perplexed-plugin
 gh attestation verify styles.css --repo lossless-group/perplexed-plugin
 ```
 Exit code 0 means the file came out of a legitimate build of the source repo.
 ## The Lossless Group plugin family
 | Plugin | What it does |
 |---|---|
 | [**Cite Wide**](https://community.obsidian.md/plugins/cite-wide) | Stable hex-identifier citations + URL-based dedupe + LLM-paste research conversion |
 | [**Image Gin**](https://community.obsidian.md/plugins/image-gin) | Recraft + Ideogram image generation, Magnific stock search, ImageKit CDN, drag-gate |
 | **Perplexed** *(this release)* | Source-cited research from Perplexity, Claude, Gemini, Perplexica, or LM Studio + per-directory templates |
 | [**Metafetch**](https://community.obsidian.md/plugins/metafetch) | Pull OpenGraph metadata into note frontmatter via OpenGraph.io or Microlink |
 If Perplexed saves you time, [buy us a coffee](https://buymeacoffee.com/losslessgroup). The plugins are free; the coffee keeps the next one coming.
 ## Full release narrative
 Marketing-grade narrative with the deeper context, the full eight-file changeset, and what's next: [`changelog/releases/0.2.1.md`](https://github.com/lossless-group/perplexed-plugin/blob/main/changelog/releases/0.2.1.md).
--- a/release-notes/0.3.0.md
+++ b/release-notes/0.3.0.md
@ -0,0 +1,59 @@
 **0.3.0 turns Perplexed into a serious analyst tooling layer for Venture Capital, Private Equity, and equities-trading workflows. Three new directory templates produce 6-9K-word cited analyst drafts in a single Perplexity Deep Research run. Underneath them, four infrastructure fixes turn deep-research from a brittle "might work" workflow into something you can put real analyst time behind.**
 ## Why care?
 If your work involves naming the actual companies in a market category, mapping who's playing where in an emerging market, or profiling the open specs an investment thesis depends on — 0.3.0 is the release where Perplexed becomes a serious analyst tooling layer, not just a notes-with-citations plugin.
 **Three new directory templates.** Each produces a 6-9K-word cited analyst draft in a single Perplexity Deep Research run. Each was designed for the kind of document a VC analyst, PE associate, or equities-trading desk strategist hands to a partner or IC:
 - **Market-category profiles** (`concepts/Market-Categories/`) name the companies in a category, sorted into three explicit financial-stage tiers — Incumbents (public / late-stage private / PE-owned), Challengers (Series C+ scale-ups), Innovators (Pre-Seed through Series B) — with separate *Why Now* / *What's Happening* sections covering CAGR and category-creation momentum, and a fully-sourced Industry Coverage section sub-grouped into Market Reports (Gartner, IDC, Forrester, ABI) / Industry Articles / Financial News (Bloomberg, FT, Pitchbook).
 - **Market-map drafts** (`lost-in-public/market-maps/`) produce the analyst memo itself — 4-8 sub-segments, 20-40 named innovator cards with funding stage and lead investor, Market Dynamics with cited sizing data and capital concentration, and a Frontier section with open questions framed as questions.
 - **Standards-and-specs profiles** (`Sources/Standards-and-Specs/`) turn an open protocol or convention into a strategic-context document — five-way authority typing (de-jure / consortium / vendor-led-open / community / de-facto), three-tier structural adoption framing plus notable holdouts, named editors, stewardship-transition stories, named public critics with their arguments. Surfaces what an analyst needs to know about a spec: is it surviving its originating vendor, who has it captured, who's working to break the capture.
 **Four infrastructure fixes** — each one solving a class of "deep-research silently produces broken output" failure mode that would otherwise have kept the analyst-grade workflow brittle:
 - **Per-chunk idle-timeout discipline.** The timer that decides "is this stream still alive?" is now armed per-chunk, not once per request. A healthy slow stream completes naturally regardless of total duration; a silently-stalled one fails in seconds, not at the wall-clock ceiling. Ported from the legacy `PerplexityModal` flow into the directory-template runtime.
 - **Per-template `request-timeout-ms:` cft override.** Becomes the optional absolute wall-clock ceiling (belt-and-suspenders backstop on top of the idle timer). Set to `0` to disable entirely and rely on idle-only safety — the recommended setting for the two new analyst-grade templates.
 - **Per-template `max-tokens:` cft override.** Fixes the silent-truncation pathology where Perplexity's default output cap (~8,192 tokens, ~6K words for `sonar-deep-research`) was ending the stream cleanly mid-template — full sources footer, no Notice, no truncation flag, looking like a healthy completion that just happened to be missing the back half. The three deep-research templates ship with `max-tokens: 24000` (~18K-word budget).
 - **Strengthened rendering-discipline partials.** `mermaid-discipline.md` rewritten with paired BAD/GOOD examples throughout, the `&amp;` escape rule, an explicit `\n` vs `<br/>` ban, a six-item self-check before emit, and a simplify-rather-than-break ethos. New `latex-discipline.md` partial covers Obsidian MathJax delimiters and `$` escaping. Both wired into `concept-profile.md`.
 ## Diagnostic vocabulary for streaming failures
 | Symptom | Cause |
 |---|---|
 | Mid-sentence cutoff (ends with a partial word); no sources footer; "stream went idle" Notice or no Notice at all | Wall-clock timer (`request-timeout-ms:`) or idle timer (`stream-idle-timeout-ms:`) fired during streaming |
 | Clean section-end cutoff; **sources footer renders correctly with all citations**; no Notice | Perplexity `max_tokens` cap; stream completed via `finish_reason: "length"` |
 The three streaming knobs (`request-timeout-ms`, `stream-idle-timeout-ms`, `max-tokens`) are independent — a template can hit any of them, all of them, or none. The three new deep-research templates set all three generously so analyst-grade runs land reliably.
 ## Editorial discipline diverges by template
 The three new deep-research templates share a common technical substrate but apply different editorial stances on big tech:
 - **Market-map** caps big tech at 1-of-5-10 in any sub-bucket to counteract training-data bias — the analyst's job is to name the named operators driving the curve.
 - **Standards-and-specs** names big tech freely in the Incumbents tier (large dominant implementations) and aggressively in the Innovators tier (where the next extensions come from). Suppression would distort the implementation landscape.
 - **Market-category** names big tech as the FIRST move in Incumbents because for a category profile, knowing the incumbents IS the work product.
 Each template's system prompt states its editorial stance explicitly so a future template-modifier doesn't accidentally cross-pollinate the wrong stance across templates.
 ## Upgrade notes
 **Vault template drift — the gotcha worth five seconds of attention.** If your `Content-Dev/Templates/concept-profile.md` was seeded before today, it may be missing the `{{include: mermaid-discipline}}` and `{{include: latex-discipline}}` directives. Without them, every concept-profile run generates Mermaid with no rendering rules in scope at all, producing broken diagrams 4-of-5 times. Open the file, find the *Defining and Describing* section, and insert both directives immediately after the "render a `mermaid` codefence here" instruction. The re-seed button won't add them — it only writes missing files.
 **Re-seed pulls in the new templates and the new partial.** The seven shipped templates plus the two partials are bundled into `main.js`. After upgrading, click *Settings → Directory templates → Re-seed templates* to write the new files into your vault. The button only adds files whose filenames don't already exist; nothing existing gets overwritten. You'll see `market-map-profile.md`, `standards-and-specs-profile.md`, `market-category-profile.md` appear in templates and `latex-discipline.md` appear in partials.
 **Backward compatibility.** All cft-block additions are opt-in. Templates that don't declare the new knobs inherit sensible defaults. The four pre-0.3.0 templates (concept, vocabulary, source, toolkit) run unchanged.
 **Plugin-level default request timeout bumped 10 min → 30 min.** Visible in *Settings → Directory templates → Request timeout (ms)*. If you had previously set a custom value, it's preserved.
 ## Deferred to 0.4.x
 Three follow-ups that surfaced during the 0.3.0 work cycle:
 - A `cf_finish_reason` frontmatter stamp on every run — would have made the max-tokens diagnostic instant instead of an investigation.
 - A "Continue directory template on current file" command — for runs cut off by max-tokens, would re-prompt with only the missing sections plus URL-keyed citation merge.
 - A "diff and patch user files" mode for re-seed — the vault-drift bug we caught is structural: re-seed only writes missing files, so partial/template improvements never reach users who've already seeded.
 ## Compatibility
 `minAppVersion: 1.8.10` — unchanged from 0.2.1. Desktop-only — unchanged. No new external dependencies; the three new templates and two partials are bundled into `main.js` at build time. The cleanup pipeline behavior (think-block wrap, image-marker swap, sources footer) is unchanged.
--- a/src/docs/partials/latex-discipline.md
+++ b/src/docs/partials/latex-discipline.md
@ -0,0 +1,18 @@
 **Obsidian MathJax discipline — Obsidian uses `$$...$$` for block math and `$...$` for inline math, NOT the `\[...\]` / `\(...\)` form.** Math wrapped in `\[...\]` or `\(...\)` renders as literal escape sequences, not as a formula.
 **Block math** uses `$$...$$`:
 - BAD:  `\[\text{CAGR} = (V_e / V_b)^{1/n} - 1\]`
 - GOOD: `$$\text{CAGR} = (V_e / V_b)^{1/n} - 1$$`
 **Inline math** uses `$...$`:
 - BAD:  `where \(n\) is the number of years`
 - GOOD: `where $n$ is the number of years`
 **Escape literal dollar signs as `\$` in prose.** If you mention a dollar amount anywhere in the document, escape the currency symbol so Obsidian does not pair it with another `$` and try to render the span between as inline math.
 - BAD:  `revenue grew from $500,000 to $2.5M`
 - GOOD: `revenue grew from \$500,000 to \$2.5M`
 This applies throughout the document, not just near math blocks. Two unescaped `$` characters in the same paragraph will be parsed as an inline-math span no matter how far apart they are.
--- a/src/docs/partials/mermaid-discipline.md
+++ b/src/docs/partials/mermaid-discipline.md
@ -1,12 +1,54 @@
-**Mermaid syntax discipline — read every rule before emitting a fence; broken mermaid is the #1 failure mode of these templates:**
+**Mermaid syntax discipline — read every rule before emitting a fence; broken mermaid is the #1 failure mode of these templates.** Mermaid is a strict parser. Every chart you emit must follow these rules or the page renders a syntax error instead of the diagram. Each rule below pairs a BAD example with a GOOD example — match the GOOD shape exactly.
- **Always quote node labels** with double quotes whenever the label contains ANY of: parentheses `( )`, equals `=`, plus `+`, comma `,`, colon `:`, slash `/`, backtick, brackets `[ ]`, braces `{ }`, angle brackets `< >`, pipe `|`, ampersand `&`, hash `#`, or LaTeX. Example: write `A["F(x) + x"]`, never `A[F(x) + x]`. When in doubt, quote.
+**Quote every multi-word node and edge label in double quotes.** The parser cannot handle `(`, `)`, `&`, `:`, `,`, `/`, `#`, `=`, `+`, `[`, `]`, `{`, `}`, `<`, `>`, `|`, backtick, or LaTeX syntax inside a bare label. When in doubt, quote.
- **Never put parentheses, math, or LaTeX inside an unquoted label.** No `\(...\)`, no `$...$`, no `H(x)=F(x)+x` bare — wrap in `"..."`.
+
- **Subgraph titles must be quoted and given an id:** `subgraph RB["Residual Block"] ... end`, never `subgraph Residual Block`.
+- BAD:  `A[Raw inputs (text, audio)]`
- **Edge labels** with special chars use the `-->|"label"|` form when the label contains punctuation; plain words can stay unquoted.
+- GOOD: `A["Raw inputs (text, audio)"]`
- **One statement per line.** No semicolons. No trailing whitespace inside labels.
+
- **Allowed shapes:** rectangle `A["x"]`, rounded `A("x")`, circle `A(("x"))`, rhombus `A{"x"}`. Pick one consistently; don't mix exotic shapes.
+- BAD:  `B -->|Sends data & status| C`
- **Direction header is required:** start with `flowchart LR` or `flowchart TD` (prefer `LR` for process flows, `TD` for hierarchies/taxonomies).
+- GOOD: `B -->|"Sends data &amp; status"| C`
- **Node ids are short alphanumerics** (`X`, `F1`, `Add`, `H`), distinct from labels. Never use spaces, parens, or punctuation in an id.
+
- **No HTML, no `<br>` outside quoted labels, no markdown inside labels.** If a label needs a line break, use `"line one<br/>line two"` inside quotes.
+**Escape `&` as `&amp;` inside quoted labels.** Bare `&` is interpreted as the start of an HTML entity even when the label is quoted.
- Before finalizing, mentally parse the fence: every `[`, `(`, `{` must close; every label containing punctuation must be wrapped in `"..."`.
+
 - BAD:  `D["Search & filter"]`
 - GOOD: `D["Search &amp; filter"]`
 **No nested double quotes.** The outer `"..."` is the only quoting layer the parser supports. Drop the inner quotes or rephrase.
 - BAD:  `B["Legal view<br/>\"Which laws apply?\""]`
 - GOOD: `B["Legal view<br/>Which laws apply?"]`
 **Line breaks inside labels are `<br/>`, never `\n`.** `\n` renders as the literal two characters backslash-n.
 - BAD:  `B1[Membrane potential\nintegrates spikes]`
 - GOOD: `B1["Membrane potential<br/>integrates spikes"]`
 **Subgraph titles with spaces need an ID plus a quoted display title.** A bare `subgraph My Title` breaks in stricter renderers.
 - BAD:  `subgraph Spiking neuron dynamics`
 - GOOD: `subgraph Spiking_neuron_dynamics ["Spiking neuron dynamics"]`
 **Quote decision-node (diamond) labels too.** Same rules as `[...]` labels apply to `{...}` shapes.
 - BAD:  `B2{Threshold reached?}`
 - GOOD: `B2{"Threshold reached?"}`
 **Allowed shapes:** rectangle `A["x"]`, rounded `A("x")`, circle `A(("x"))`, decision/rhombus `A{"x"}`. Pick one consistently per fence; don't mix exotic shapes.
 **Structural rules every fence must follow:**
 - Start with a direction header: `flowchart LR` for process flows, `flowchart TD` for hierarchies or taxonomies.
 - Node IDs are short alphanumerics (`X`, `F1`, `Add`, `H`) — distinct from labels, no spaces, no punctuation.
 - One statement per line. No semicolons. No trailing whitespace inside labels.
 - No HTML outside quoted labels. No markdown inside labels.
 **Self-check before emitting the ` ```mermaid ` block.** Mentally walk every node and edge label and confirm all six:
 1. Every multi-word label is wrapped in `"..."`.
 2. Every `&` inside a label is written `&amp;`.
 3. No label contains nested `"`.
 4. Every line break inside a label is `<br/>`, not `\n`.
 5. Every subgraph whose title contains a space uses the `id ["display title"]` form.
 6. Every `[`, `(`, `{` has a matching close.
 If you cannot satisfy all six, **simplify the labels** (drop the parenthetical sub-text, replace `&` with "and", shorten the phrasing) rather than emit a broken diagram. Do not emit a fence you cannot validate. A working diagram with shortened labels is always better than a broken diagram with the labels you wanted.
--- a/src/docs/templates/README.md
+++ b/src/docs/templates/README.md
@ -17,6 +17,9 @@ ## Shipped templates
 | `vocabulary-profile.md` | `Vocabulary/**` | Definitions of terms with disambiguation through an innovation-consulting lens. |
 | `source-profile.md` | `Sources/**` | Profiles of trusted sources — books, people, channels, publications, journals, reports, events. Adapts emphasis to the source's type. |
 | `toolkit-profile.md` | `Tooling/**` | Profiles of tools, products, platforms, frameworks. |
 | `market-map-profile.md` | `lost-in-public/market-maps/**`, `market-maps/**` | Analyst-grade market-map drafts — both Known Category (e.g., Humanoid Robots) and Thesis-Driven (e.g., Neural Network Hardware as Brains for Robotics). Runs on `sonar-deep-research`. Single-stage v1; multi-stage RAG + Claude-edit pass is planned. |
 | `standards-and-specs-profile.md` | `Sources/Standards-and-Specs/**`, `Standards-and-Specs/**` | Analyst-grade profiles of open specs and standards. Five-way authority typing (de-jure / consortium / vendor-led-open / community / de-facto). Three-tier structural adoption framing (incumbents / challengers / innovators) plus notable holdouts. Named editors, stewardship transitions, named critics. Runs on `sonar-deep-research` with idle-only timeout safety. |
 | `market-category-profile.md` | `concepts/Market-Categories/**`, `Market-Categories/**` | Concept-folder reference card for a named market category. Three-tier company landscape with explicit FINANCIAL-STAGE definitions: Incumbents (public / late-stage private / PE-owned) → Challengers (Series C+ scale-ups, recently public) → Innovators (Pre-Seed through Series B). Big tech belongs in Incumbents — no anti-incumbent cap here, unlike the market-map template. Separate Why Now / What's Happening sections covering CAGR + category-creation momentum. Industry Coverage section sub-grouped into Market Reports / Industry Articles / Financial News. Runs on `sonar-deep-research`. |
 ## How a template works
@ -59,7 +62,7 @@ # User Notes
 ### The three zones
 1. **Frontmatter** (top, between `---` lines) — carries `title`, `applies-to-paths` (array of glob patterns), and an optional `description`. The plugin uses `applies-to-paths` to match a template to a target file.
-2. **`cft` block** (a code fence with language `cft`) — YAML config: `provider`, `model`, `search-recency`, `return-citations`, `return-images`, plus a multi-line `system:` prompt. Anything above the `cft` block is treated as documentation and dropped from the request.
+2. **`cft` block** (a code fence with language `cft`) — YAML config: `provider`, `model`, `search-recency`, `return-citations`, `return-images`, optional `stream-idle-timeout-ms:` (per-chunk idle timer; defaults 270s for deep-research / 90s otherwise — the primary safety mechanism), `request-timeout-ms:` (optional absolute wall-clock ceiling; falls back to plugin-level *Request timeout (ms)*; set to `0` to disable and rely on idle-only), `max-tokens:` (optional Perplexity output-token budget override; defaults to Perplexity's per-model cap — ~8192 for sonar-deep-research, which silently truncates long analyst-grade templates), plus a multi-line `system:` prompt. Anything above the `cft` block is treated as documentation and dropped from the request.
 3. **Heading skeleton** (everything between the `cft` block's closing fence and the first `***`) — the user prompt. This is the markdown structure the model fills in. Bullets under each heading are *instructions to the model*, not literal output.
 The `***` divider terminates the user prompt. Anything below it (the User Notes zone) is for your own scratch work and never reaches the model.
--- a/src/docs/templates/concept-profile.md
+++ b/src/docs/templates/concept-profile.md
@ -60,6 +60,8 @@ # Defining and Describing {{basename}}
 - If this concept involves a process, hierarchy, taxonomy, or part-relationship that a diagram clarifies, render a `mermaid` codefence here. If a diagram does not add insight, omit it entirely — do not force one.
 {{include: mermaid-discipline}}
 {{include: latex-discipline}}
 - Write a one-sentence italicized lede (a zinger or kicker) that captures the core insight in plain language. Use markdown italics: `_..._`.
 - Then write a 2–4 sentence paragraph giving more context: what the concept is, when it applies, and why it matters.
--- a/src/docs/templates/market-category-profile.md
+++ b/src/docs/templates/market-category-profile.md
@ -0,0 +1,316 @@
 ---
 title: Market Category Profile (Analyst Draft)
 applies-to-paths:
  - "concepts/Market-Categories/**"
  - "Market-Categories/**"
 description: Generates an analyst-grade profile of a named market category — its definition, the forces that made it coherent right now, current CAGR and momentum, and the three-tier company landscape (incumbents / challengers / innovators) with explicit financial-stage definitions for each tier.
 date_created: 2026-05-27
 date_modified: 2026-05-27
 ---
 # About this template
 Use this for files under `concepts/Market-Categories/` whose body is empty or whose curated lead-in (a thesis paragraph or stake-in-the-ground take) has been authored but the analytical body is missing.
 A **market category profile** is the mental-model entry an innovation consultant draws on when asked "what is this market, who's playing in it, and where is it going?" Unlike a [[market-map-profile]] (which is a published analyst memo with sub-segments and lighthouse examples), this profile is a concept-folder entry — a reference card the curator returns to when reading or briefing on the category.
 The structure is opinionated about ONE thing in particular: the three-tier company landscape is **structural**, not editorial. Each tier has an explicit financial-stage definition, and the template asks the model to populate each tier separately:
 - **Incumbents** — large public companies, tech giants, late-stage private, PE-owned behemoths. Legacy footprint, named everywhere in the category. Big tech BELONGS HERE — do not suppress.
 - **Challengers** — well-funded scale-ups (Series C and beyond, or recently public via SPAC/IPO). Rapidly growing, hype-driven, eating share from the incumbents.
 - **Innovators** — Pre-Seed through Series B startups. Early-stage, often founder-led, novel-bet positioning, the frontier of the category.
 This is a different editorial discipline from the `market-map-profile` template — for a market map, the analyst is enumerating sub-segments and innovators within them, and big tech gets capped to 1 of 5-10 to counteract training-data over-representation. For a market-category profile, the goal is the *full* landscape of who plays in this market, sorted by financial stage. Both incumbents and innovators are first-class.
 This template runs on `sonar-deep-research` and skips image embedding by design — deep research returns better citation density and analytical length but unreliable image metadata. Banner / portrait / square imagery for category entries lives in frontmatter, generated separately (Ideogram).
 ```cft
 provider: perplexity
 model: sonar-deep-research
 return-citations: true
 return-images: false
 # Idle-only safety — ceiling disabled. The per-chunk idle timer (270s
 # for deep-research, inherited from the model-class default) is the
 # sole safety mechanism: as long as Perplexity sustains bytes, the run
 # is allowed to complete. A complete market-category profile can stretch
 # 6-9K words given the three-tier company landscape + named market
 # reports + What's Happening section; any wall-clock cap risks cutting
 # the tail. Set this to a positive value if you want a hard ceiling.
 request-timeout-ms: 0
 # Generous output-token budget. Perplexity's default for sonar-deep-research
 # (~8192 tokens, ~6K words) silently truncates this template's back half
 # by ending the stream cleanly with finish_reason: length mid-skeleton.
 # Bumped to 24000 (~18K-word budget) so the full three-tier landscape +
 # deeper cards + industry coverage section can land with headroom.
 max-tokens: 24000
 system: |
  You are writing the analyst-grade profile of a MARKET CATEGORY named
  "{{basename}}" for an innovation consultant's vault.
  A market-category profile is the mental-model entry an analyst returns
  to when asked "what is this market, who's in it, and where is it
  going?" It is NOT a published market map (those live in
  `lost-in-public/market-maps/` and run a different template). It IS a
  concept-folder reference card with disciplined three-tier company
  framing and explicit market-data sourcing.
  A CATEGORY PROFILE answers four questions a partner would ask:
  - WHAT is this market category — how are its boundaries drawn, what's
    in and what's out?
  - WHY NOW — what forces aligned to make this category coherent or
    expandable at this moment?
  - WHAT'S HAPPENING — what is the CAGR, the momentum, the category
    creation or coalescence dynamics right now, named with specific
    market-report figures?
  - WHO IS PLAYING in three explicit financial-stage tiers — incumbents,
    challengers, innovators — with named cited entities in each?
  THREE-TIER COMPANY FRAMING — STRUCTURAL, NOT EDITORIAL.
  Each tier has an explicit financial-stage definition. Sort each
  named company into exactly one tier:
  - INCUMBENTS — large public companies, tech giants, late-stage
    private companies (typically post-Series E or with $1B+ valuation
    and 10+ years of operation), and private-equity-owned behemoths.
    The defining trait is legacy footprint with huge market presence —
    these are the companies an enterprise buyer ALREADY has a contract
    with, even if not in this category yet. Big tech (Microsoft,
    Google, Amazon, Apple, Meta, Oracle, Salesforce, IBM, Adobe,
    SAP, Cisco, Nvidia post-2020) belongs HERE if it has a relevant
    offering in the category — DO NOT suppress.
  - CHALLENGERS — well-funded scale-ups, typically Series C through
    pre-IPO, or recently public via SPAC/IPO with under 7 years of
    operation. The defining traits are rapid growth, public hype
    (analyst coverage, founder press, conference keynotes), and the
    capital position to credibly threaten incumbent market share.
    "Recently IPO'd unicorn" usually belongs here, not in Incumbents.
  - INNOVATORS — Pre-Seed through Series B funded startups. The
    defining traits are early-stage funding, novel-bet positioning
    (often a contrarian thesis on the category's shape), and
    typically founder-led with under 100 employees. This is where
    the next category-redefinition will come from.
  For each tier, produce 5-8 named entities. Render each entry in
  `[Company Name](https://url) — one-line on their role in this category`
  form, then produce 2-3 deeper CARDS per tier for the most
  strategically significant entries (the ones a partner would
  expect to be briefed on by name).
  Frontmatter for "{{basename}}":
  {{frontmatter}}
  TIER-CARD SHAPE (deeper treatment for top entries per tier):
  Each deeper card follows this shape exactly. The emphasis SHIFTS by
  tier — for Incumbents the Footprint line is heaviest, for Innovators
  the Funding line is heaviest:
      #### [Company Name](https://homepage.url)
      **Stage**: public (EXCHANGE: TICKER) | late-stage private (last
      round date) | PE-owned (sponsor name + acquisition year) |
      scale-up | Series B (date) | Pre-Seed / Seed (date)
      **Funding**: total raised + most recent round + lead investor + year.
      For public companies: market cap (as of recent quarter) + last reported
      revenue. For PE-owned: known acquisition price if disclosed. Cite. [N]
      **Footprint**: revenue / employees / customer count / countries
      served — whatever's most visible. Heaviest for INCUMBENTS. Cite. [N]
      **Why they're in this category**: one specific sentence on what
      they ship into this market and what their angle is. Not marketing
      adjectives; a concrete differentiator or footprint claim. Cite. [N]
      **Coverage**: 1-2 references in trade press, founder podcasts, or
      analyst notes. Format: `[Outlet, Title](url)`. Cite. [N]
  INDUSTRY DATA AND COVERAGE — name the reports, not just the figures.
  The "Industry Coverage and Market Data" section is sub-grouped into
  three explicit subsections:
  - Market Reports — Gartner, IDC, Forrester, McKinsey, ABI Research,
    Bain, BCG, Deloitte Insights, Frost & Sullivan, Grand View Research,
    Mordor, Markets and Markets, Cabinet Office reports, etc. Name the
    REPORT (with title and year) AND the firm. Quote the figure plus the
    methodology framing (top-down vs bottom-up, geography, year range).
  - Industry Articles — specialized trade press (sector verticals from
    TechCrunch, The Information, Stratechery, sector-specific trade
    publications, named founder/operator blogs). Prefer NAMED journalists
    and operator-thinkers over generic outlet citations.
  - Financial News — Bloomberg, FT, WSJ, Pitchbook articles, Reuters,
    Crunchbase News, dealroom.co, Seeking Alpha, Tegus. These should be
    the source for funding-round figures, M&A activity, public-company
    earnings calls referencing the category.
  RESEARCH DISCIPLINE:
  - Use Perplexity's web search aggressively. For a market-category
    profile, breadth of named entities and cited market data both
    matter — you are populating a reference card that the analyst will
    return to dozens of times.
  - For every factual claim — funding round, market sizing, CAGR
    figure, revenue, market cap, customer name — append an inline
    numeric citation marker [1], [2], etc.
  - Quote phrasing from primary sources where useful (founder interviews,
    earnings notes, analyst report excerpts, regulatory filings).
  - Prefer primary surfaces: company homepage, S-1 filings, earnings
    calls, founder Twitter/X, technical blog posts, conference keynotes.
    Aggregator pages (Crunchbase summaries, PitchBook profiles,
    Wikipedia) are fallbacks for funding stage and headcount only.
  - Do NOT cite this Perplexity response itself, only the underlying
    sources.
  WHY NOW AND WHAT'S HAPPENING — DIFFERENT BEATS.
  These two sections are easy to conflate; keep them distinct:
  - WHY NOW asks: what FORCES aligned that made this category coherent
    (or expandable) right now? Technological unlocks, regulatory shifts,
    capital-formation patterns, customer-behavior shifts. The reader
    leaves Why Now understanding the ENABLING CONDITIONS.
  - WHAT'S HAPPENING asks: what is the current MOMENTUM — CAGR figures
    with sources, category-creation events (a defining IPO, a defining
    acquisition, a defining product launch that crystallized the
    category name), recent capital concentration. The reader leaves
    What's Happening understanding the CURRENT VELOCITY and where the
    capital is flowing.
  EDITORIAL STANCE:
  - This is a market-category profile, NOT a market-map analyst memo.
    There is no anti-incumbent cap — name the big tech in Incumbents,
    name the well-funded scale-ups in Challengers, name the Pre-Seed
    bets in Innovators. The three-tier sorting IS the discipline.
  - Where a company could plausibly fit in two tiers (a unicorn that
    just IPO'd; a late-stage private that's behaving like a scale-up),
    pick the tier that best matches the company's CURRENT behavior and
    note the ambiguity in the card's "Why they're in this category"
    line.
  - Surface CATEGORY DISPUTES — sentences in the prose where credible
    operators disagree about whether the category boundary should
    include or exclude a particular sub-area. Disputes are signal.
  LINKS AND WIKILINKS:
  - For company names, founder names, and source links, use
    `[Name](https://url)` form.
  - Do NOT invent `[[wikilink]]` syntax. The curator will promote
    names to vault wikilinks during the curation pass.
  - If you happen to know a canonical adjacent concept the curator
    has already vaulted (e.g., "Agentic Workspaces", "Compliance
    Automation"), surface the name as plain text in the Adjacent
    Concepts section so the curator can wikilink it later.
  CALIBRATION ON LENGTH:
  This is a deep-research run. Lean long, not short. A complete
  market-category profile is roughly 6,000-9,000 words of body, with
  15-24 named companies across three tiers (5-8 per tier) + 2-3 deeper
  cards per tier, a fully-populated Industry Coverage & Market Data
  section with 4-6 named reports / articles / news pieces per
  sub-grouping, and explicit cited CAGR / TAM figures in What's
  Happening. Better to over-enumerate and let the curator prune than
  to under-enumerate and leave the analyst guessing.
 ```
 # Snapshot
 - One-paragraph italicized lede (max 2 sentences) that captures the category in a single beat. Voice: an analyst introducing the category to a partner who's never heard of it. Use markdown italics: `_..._`.
 - Then the headline stat — one cited statistic that signals scale, velocity, or category momentum (current TAM, current CAGR, a recent landmark funding round, a defining exit). Format as a blockquote (`> "..."`).
 - Then 2-3 sentences orienting the reader: what is this category, what timeframe is the profile capturing, why is it worth a reference card right now.
 # What is this Market Category?
 - 3-5 sentences defining the category. What problems do its products solve? What customer is it sold to? What does the category EXCLUDE — what does the boundary leave out that a naive reader might wrongly include?
 - One sentence on where the boundary is fuzzy or disputed — name the specific edges where operators disagree about whether something belongs.
 # Why Now?
 - 3-5 cited bullets, each one a specific FORCE that aligned to make this category coherent or expandable in the current quarter. The reader should leave understanding the ENABLING CONDITIONS.
 - Force types to consider: technological unlocks (a capability crossing a cost / latency / accuracy threshold), regulatory or standards shifts, capital-formation patterns (a fund vintage, an exit precedent that opened the LP appetite), customer-behavior shifts, an open-source release that lowered the floor for new entrants.
 - Cite each force. Where possible, quote a founder, analyst, or operator who named the force in public.
 # What's Happening?
 What's the current momentum, in cited numbers and named events.
 - **CAGR and TAM:** 2-3 cited bullets covering the current CAGR estimate, TAM, and forecast year range — naming both the figure AND the report it comes from. Where two credible sources disagree (Gartner says $X by Y, IDC says $Z by Y), surface the disagreement explicitly.
 - **Category creation events:** 2-3 cited bullets covering the defining moments — a landmark IPO that crystallized the category name; a defining acquisition that signaled incumbent recognition; a defining product launch that became the category's reference implementation; a defining analyst-report-naming-the-category event.
 - **Capital concentration:** 2-3 cited bullets covering where funding has concentrated by tier (e.g., "$3B raised by Series B-D scale-ups in the category in 2025, led by [named funds]"), with named lead investors where public.
 # Market Incumbents
 Large public companies, tech giants, late-stage private (post-Series E or $1B+ valuation with 10+ years of operation), or private-equity-owned behemoths. Legacy with huge market footprint. Big tech belongs here when they have a relevant offering — do not suppress.
 - 5-8 named entities. Format: `[Company Name](https://url) — one-line on their role and footprint in this category`. Cite each.
 - After the flat list, produce 2-3 deeper TIER-CARDS for the most strategically significant entries per the card shape in the system prompt. For Incumbents, the **Footprint** line is the heaviest field — market cap, revenue, customer count, geographic reach.
 # Market Challengers
 Well-funded scale-ups (Series C and beyond, or recently public via SPAC/IPO with under 7 years of operation). Rapidly growing, hype-driven, public analyst coverage, capital position to credibly threaten incumbent share.
 - 5-8 named entities. Same format as Incumbents.
 - After the flat list, produce 2-3 deeper TIER-CARDS for the most strategically significant. For Challengers, both **Funding** and **Footprint** matter equally — funding shows the war chest, footprint shows the traction.
 # Market Innovators
 Pre-Seed through Series B funded startups. Early-stage, often founder-led, novel-bet positioning, typically under 100 employees. This is where the next category-redefinition will come from.
 - 5-8 named entities. Same format as Incumbents.
 - After the flat list, produce 2-3 deeper TIER-CARDS for the most strategically significant. For Innovators, the **Funding** line is the heaviest field — round size, lead investor, date, total raised, and the contrarian thesis the round capitalized.
 # Industry Coverage and Market Data
 The sourcing layer — where to read about this category and where the figures came from. Group into three subsections:
 ## Market Reports
 - 4-6 named market reports. Format: `**[Report Title, Year](url)** — Firm — one-line on the report's signature finding or methodology. [N]`
 - Prefer specialized analyst firms (ABI Research, Forrester, Gartner, IDC, Frost & Sullivan, Grand View Research) over generalist business publications.
 - Where a report's headline figure has been quoted in this profile (CAGR, TAM), the citation should resolve to the report itself, not a press release recapping it.
 ## Industry Articles
 - 4-6 named articles, blog posts, or essays from specialized trade press and operator-thinkers. Format: `**[Title](url)** — Outlet / Author — one-line on the angle. [N]`
 - Prefer named journalists and operator-bloggers over generic outlet citations. Founder posts on Substack / personal blog count and are often the most useful.
 ## Financial News Sources
 - 4-6 named financial news pieces covering funding rounds, M&A activity, public-company earnings calls referencing the category, or sell-side analyst notes. Format: `**[Title](url)** — Outlet — one-line on what it reports. [N]`
 - Prefer Bloomberg, FT, WSJ, Reuters, Pitchbook, Crunchbase News, dealroom.co for primary funding/M&A signal. Use Seeking Alpha / Motley Fool / Yahoo Finance only for public-company earnings recaps when the primary source isn't available.
 # Frontier and Open Questions
 - 4-6 bullets, each a specific open question about where the category is going. Frame each as a question, not a statement.
 - Pair each question with a one-sentence note on which tier (Incumbents / Challengers / Innovators) or which named operators are most likely to drive the resolution.
 - Surface category-boundary disputes explicitly — where credible operators disagree about whether the category should expand to include adjacent functions, or contract to focus on a narrower core.
 # Adjacent Concepts and Categories
 - 4-8 plain-text concept names (no wikilink syntax — the curator wikilinks during curation) that an operator working in this category would want to explore next.
 - Mix of: adjacent market categories (other categories this one borders), foundational concepts (mental models the category sits on), and vocabulary terms (specific jargon the curator may want to define in `Vocabulary/`).
 - Format: `- <Concept Name> — one-line on how it adjoins this category`.
 ***
 # User Notes
 Anything below the `***` line is excluded from the request. Use this zone for:
 - The stake-in-the-ground take or thesis paragraph you want to fold into the model's context. To do that, move the paragraph ABOVE the `***` divider before running the template.
 - Curator's wikilink resolution notes during the curation pass — which named companies, founders, or adjacent concepts have vault entries that need linking back.
 - Tuning notes on which sections came back thin and need a re-run.
 - Hand-curated notes on the category's relevance to your portfolio or thesis.
 ## Relationship to other shipped templates
 - **`market-map-profile`** — for published analyst-grade maps in `lost-in-public/market-maps/`. That template produces a memo with sub-segments and lighthouse examples in flowing analyst prose. THIS template (market-category-profile) is a concept-folder reference card with the disciplined three-tier financial-stage framing.
 - **`concept-profile`** — for general `concepts/**` entries. The market-category template targets the narrower `concepts/Market-Categories/` sub-folder and overrides the general concept template's editorial stance (which suppresses big tech) — for a category profile, naming the incumbents IS the goal.
 - **`toolkit-profile`** — for individual tools/products/platforms in `Tooling/**`. Many companies named in this template's tiers will have their own toolkit-profile entries that the curator wikilinks back to.
 ## Multi-stage roadmap (deferred)
 This v1 template is intentionally single-stage. The deferred multi-stage version mirrors the market-map and standards-and-specs roadmaps, tracked in `context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md`:
 1. **RAG pre-flight** — pull canonical Lossless sources adjacent to this category (vaulted company profiles in `Tooling/`, adjacent vaulted categories, prior commentary in `concepts/`) as primed context.
 2. **Perplexity research stage** — deep research with the RAG context as primer, eliminating wikilink invention and letting the model name actual vault entries.
 3. **Claude editing stage** — editorial pass that enforces tier discipline, prunes over-enumeration, sharpens tier boundaries (especially for the unicorn-just-IPO'd ambiguity), and emits the final `[[wikilink]]` form.
--- a/src/docs/templates/market-map-profile.md
+++ b/src/docs/templates/market-map-profile.md
@ -0,0 +1,244 @@
 ---
 title: Market Map (Analyst Draft)
 applies-to-paths:
  - "lost-in-public/market-maps/**"
  - "market-maps/**"
 description: Generates an analyst-grade draft of a market map — either a Known Category (e.g., Humanoid Robots, Light-based Computing) or a Thesis-driven map (e.g., Neural Network Hardware as `Brains` for Robotics) that traverses known categories.
 date_created: 2026-05-26
 date_modified: 2026-05-26
 ---
 # About this template
 Use this for files under `lost-in-public/market-maps/` (or any top-level `market-maps/`) whose body is empty or whose curated lead-in (Topics, Lighthouse Examples) has been authored but the analytical body is missing.
 A **market map** is the draft a well-paid analyst would hand to a partner: not an encyclopedia entry, not marketing copy. It explains who is doing what in a category, why now, who funded them, how segments differ, and what the open questions are. Two flavors are supported:
 1. **Known Category** — Humanoid Robots, Light-based Computing, Quantum Computing. The taxonomy is roughly settled; the work is enumerating innovators within established sub-segments and explaining the current frontier.
 2. **Thesis-Driven** — "Neural Network Hardware as `Brains` for Robotics." The thesis traverses multiple known categories under a hypothesis. The work is making the thesis legible, then enumerating innovators from each adjacent category that the thesis pulls in.
 The heading skeleton works for both flavors. The model picks up the flavor from the file's `title`, `tags`, and any thesis paragraph the user has pre-authored above the body.
 This template runs on `sonar-deep-research` and skips image embedding by design — deep research returns better citation density and analytical length but unreliable image metadata. Banner / portrait / square imagery for market maps is generated separately (Ideogram) and lives in frontmatter.
 ```cft
 provider: perplexity
 model: sonar-deep-research
 return-citations: true
 return-images: false
 # 40-minute absolute wall-clock ceiling — the belt-and-suspenders cap.
 # The primary safety is the per-chunk idle timer (270s for deep-research,
 # inherited from the model-class default), which lets a slow-but-healthy
 # stream complete naturally while killing a silently-stalled one fast.
 # This ceiling is the "do not run longer than 40 min under any
 # circumstances" cap on top of that. Market-map deep-research runs
 # routinely produce 6-8K-word drafts with 20-40 named innovators across
 # 4-8 sub-segments; a complete run is worth $10-$50 of analyst time, so
 # the cap is generous. Set this key to 0 to disable the ceiling entirely
 # and rely on idle-only safety.
 request-timeout-ms: 2400000
 # Generous output-token budget. Perplexity's default for sonar-deep-research
 # (~8192 tokens, ~6K words) silently truncates market-map drafts mid-skeleton
 # by ending the stream cleanly with finish_reason: length. Bumped to 24000
 # (~18K-word budget) so a complete map can land with headroom for 20-40
 # innovator cards across 4-8 sub-segments plus Market Dynamics, Frontier,
 # and Adjacent Concepts.
 max-tokens: 24000
 system: |
  You are writing the analyst-grade draft of a MARKET MAP titled "{{basename}}".
  A market map is the draft a well-paid analyst hands to a partner. It is not
  an encyclopedia entry, not a marketing post, not a listicle. It explains:
  - WHO is doing what in this market — named companies, named founders, named
    research labs, with funding stage and primary URL.
  - HOW the market segments — what natural sub-buckets exist, and which
    innovators sit in which sub-bucket.
  - WHY NOW — what changed (technology unlock, regulatory shift, capital
    flow, customer behavior) that made this market legible.
  - WHAT IS DISPUTED — where credible operators disagree about category
    boundaries, winner archetypes, or thesis viability.
  Determine the FLAVOR of map from the title and frontmatter:
  - KNOWN CATEGORY (e.g., "Quantum Computing is Confusing", "Humanoid Robots",
    "Agentic AI in Fintech") — the category name is settled. Your job is to
    enumerate sub-segments and innovators within each, and explain the frontier.
  - THESIS-DRIVEN (e.g., "Neural Network Hardware as Brains for Robotics",
    "Blockchain and Web3 Institutional Invasions") — a hypothesis traverses
    known categories. Your job is to make the thesis legible, then enumerate
    innovators from each adjacent category the thesis pulls in.
  Frontmatter for "{{basename}}":
  {{frontmatter}}
  RESEARCH DISCIPLINE:
  - Use Perplexity's web search aggressively. For a market map, breadth of
    named entities matters more than depth on any single one.
  - For every factual claim — funding round, founding year, customer name,
    market sizing, product capability — append an inline numeric citation
    marker [1], [2], etc. corresponding to the search-result order.
  - Quote phrasing from primary sources where useful (founder interviews,
    investor blog posts, earnings notes, academic abstracts).
  - Prefer primary surfaces: company homepage, founder Twitter/X, technical
    blog posts, conference talks, investor announcements. Aggregator pages
    (Crunchbase, PitchBook summaries) are fallbacks for funding stage only.
  - Do NOT cite this Perplexity response itself, only the underlying sources.
  EDITORIAL STANCE — attribute innovation correctly:
  Markets are pioneered by startups, academics, research labs, and indie
  practitioners — NOT by tech giants. Training data over-represents
  incumbents. Counteract this systematically:
  - Treat big tech (Microsoft, Google, Amazon, Apple, Meta, Oracle, Salesforce,
    IBM post-1990s, Nvidia post-2020) as ADOPTERS or POPULARIZERS in this
    market unless the entry documents an originating research-lab paper or
    a heyday-era origination story (Bell Labs, Xerox PARC, DeepMind, OpenAI's
    early years).
  - In every Lighthouse Examples sub-bucket, cap big-tech entries at 1 of
    5–10. Prefer Series A-C startups, seed-stage frontier bets,
    open-source projects, indie practitioners, research labs.
  - In Market Dynamics, name the named operators driving the curve, not
    the incumbents profiting from it.
  - Where an incumbent IS the originator, say so explicitly with the
    research paper or product release that documents the origination.
  INNOVATOR CARD SHAPE:
  Each named innovator under "Innovator Profiles" follows this shape exactly:
      #### [Innovator Name](https://homepage.url)
      **Offering**: one-sentence description of what they do that is specific
      enough that a partner could repeat it back. Use product names, customer
      names, and category boundaries. Cite. [N]
      **Funding**: stage and round size if disclosed (e.g., "$30M Series A,
      DN Capital, 2024"). "Undisclosed" or "Bootstrapped" if no public data.
      **Why they matter**: one sentence on what makes them distinctive — the
      technical bet, the GTM angle, the team's background. Not marketing
      adjectives; a specific differentiator. Cite. [N]
      **Coverage**: 1-2 references in trade press, founder podcasts, or
      analyst notes if available. Format: `[Outlet, Title](url)`. Cite. [N]
  Aim for 3-7 innovators per sub-bucket. If a sub-bucket has fewer than 3
  credible named entities, MERGE it into an adjacent sub-bucket rather than
  padding with weak entries.
  LINKS AND WIKILINKS:
  - For innovator names and source links, use `[Name](https://url)` form.
  - Do NOT invent `[[wikilink]]` syntax. The curator will promote names to
    vault wikilinks during the curation pass. If you happen to know a
    canonical concept this market touches (e.g., "Compliance AI", "Agentic
    Workspaces"), surface the concept name as plain text in the Adjacent
    Concepts section so the curator can wikilink it later.
  CALIBRATION ON LENGTH:
  This is a deep-research run. Lean long, not short. A complete market map
  is roughly 4,000-8,000 words of body, with 20-40 named innovators across
  4-8 sub-buckets, a summary table per sub-bucket where useful, and explicit
  funding-trend / adoption-pattern data in Market Dynamics. Better to over-
  enumerate and let the curator prune than to under-enumerate.
 ```
 # Market Snapshot
 - One-paragraph italicized lede (max 2 sentences) that captures the punchy thesis of this market. Voice: the analyst opening their memo. Use markdown italics: `_..._`.
 - Then the headline stat — one quoted statistic from a credible source that signals scale or velocity, with inline citation. Format the quote as a blockquote (`> "..."`).
 - Then 2-3 sentences orienting the reader: what is this market, what is the timeframe, why is it worth a map right now.
 # The Question this Map Answers
 - One paragraph (3-5 sentences) stating the question this map clarifies for an operator or investor reading it.
 - If the file is a KNOWN CATEGORY map, frame the question as "what shape has this category settled into, and where is the frontier."
 - If the file is a THESIS-DRIVEN map, state the thesis explicitly in one sentence, then explain which adjacent categories the thesis traverses and why the traversal is non-obvious.
 # Why Now
 - 3-5 bullets, each one a specific unlock that explains why this market is mappable in the current quarter and would have been premature 18 months ago.
 - Unlock types to consider: a technical capability crossing a threshold (cost, latency, accuracy), a regulatory or standards shift, a capital-formation pattern (a fund vintage, an exit precedent), a customer-behavior shift, an open-source release that lowered the floor.
 - Cite each unlock. Where possible, quote a founder, researcher, or operator who named the unlock.
 # Map of the Market — Sub-Segments
 - Identify 4-8 natural sub-segments that partition this market. For a thesis-driven map, the sub-segments are the adjacent categories the thesis traverses.
 - Give each sub-segment a 1-2 sentence definition: what falls inside, what falls outside, and what distinguishes it from its neighbors.
 - This section is the TABLE OF CONTENTS for the Innovator Profiles section below. Sub-segment names here must match section headings below.
 # Lighthouse Examples
 - For each sub-segment, list 5-10 lighthouse innovators in `[Name](url) — one-line description` form. These are the names a partner would expect to hear in this category — recognized leaders, well-funded operators, frontier bets the analyst would brief on.
 - Group under `## <Sub-segment name>` subheadings matching the Map of the Market above.
 - This is a flat reference list. The deeper analysis lives in Innovator Profiles below.
 - Use the editorial-stance cap: at most 1 of 5-10 in any sub-bucket may be big tech.
 # Innovator Profiles
 For each sub-segment from the Map of the Market, produce a `## <Sub-segment name>` heading and under it, 3-7 innovator cards in the format defined in the system prompt:
    #### [Innovator Name](https://homepage.url)
    **Offering**: ...
    **Funding**: ...
    **Why they matter**: ...
    **Coverage**: ...
 - Order within each sub-segment from most-established to most-frontier (seed-stage / stealth at the end).
 - After the innovator cards in each sub-segment, render a single summary table with columns: `Innovator | Stage | Differentiator | Primary Customer`. The table lets a partner skim the sub-segment without reading every card.
 # Media, Voices, and Coverage
 - 6-12 bullets covering the publications, podcasts, YouTubers, analysts, and individual operator-thinkers who shape the conversation about this market.
 - Format: `**Name** — Platform — one-line note on their angle / why they are worth following`. Include a primary URL link.
 - Sub-group with `## Publications`, `## Podcasts & YouTube`, `## Analysts & Operator-Thinkers` if the list is long enough to warrant it.
 - Prefer specialized trade press over generalist business press. Prefer named operator-bloggers over corporate marketing surfaces.
 # Market Dynamics
 ## Sizing and Growth
 - 2-4 cited bullets covering: TAM / current market size, projected CAGR, the report or analyst behind each number.
 - Be skeptical of single-source sizing claims; where two credible sources disagree, surface the disagreement.
 ## Adoption Patterns and Barriers
 - 2-4 cited bullets covering: what percentage of the addressable buyer base has adopted, what the canonical barriers are (procurement cycle, regulatory uncertainty, technical readiness, talent shortage), and where the adoption curve is bending.
 ## Capital Flow
 - 2-4 cited bullets covering: where the funding has concentrated by sub-segment, who the active funds are (named partners where public), and any recent exit, acquisition, or IPO that reset valuation expectations in the category.
 # Frontier and Open Questions
 - 4-7 bullets, each one a specific open question that a partner reading this map would want the analyst to think about next.
 - Frame each as a question, not a statement: "Will agent-to-agent micropayments require a separate settlement rail, or will existing card networks absorb the use case?" not "Settlement rails are evolving."
 - Pair each question with a one-sentence note on which innovators or research streams are likely to produce the answer.
 # Adjacent Concepts and Maps
 - 4-8 plain-text concept names (no wikilink syntax — the curator wikilinks during curation) that an operator working in this market would want to explore next.
 - Mix of: adjacent market maps (other categories this one borders), foundational concepts (the mental models this market sits on), and vocabulary terms (the specific jargon the curator may want to define in `Vocabulary/`).
 - Format: `- <Concept Name> — one-line on why it adjoins this map`.
 ***
 # User Notes
 Anything below the `***` line is excluded from the request. Use this zone for:
 - The thesis paragraph (for thesis-driven maps) you want the model to fold into the system context. To do that, move the paragraph ABOVE the `***` divider — into the Market Snapshot or Question section — before running the template.
 - Hand-curated `:::tool-showcase` blocks pointing to vault tools you want to feature.
 - Tuning notes, prior model outputs, and iteration history while you refine the template for your domain.
 ## Multi-stage roadmap (deferred)
 This v1 template is intentionally single-stage: one Perplexity Deep Research run produces the full draft, the curator promotes Lighthouse Examples to `:::tool-showcase` blocks and `[Name](url)` references to `[[wikilink]]` references during the curation pass.
 The deferred multi-stage version will run:
 1. **RAG pre-flight** — pull canonical Lossless sources for this market (tools under `Tooling/`, concepts under `concepts/`, prior market maps that overlap) and feed them as context to the research stage. This eliminates the wikilink-invention problem and lets the model name actual vault entries.
 2. **Perplexity research stage** — deep research with the RAG context as a primer, producing the draft this v1 template produces.
 3. **Claude editing stage** — an editorial pass that enforces the analyst voice, prunes the over-enumeration, restructures sub-segment boundaries where the data demands, and emits the final `[[wikilink]]` form.
 The plumbing for stage 3 (the Claude orchestrator) lives in `claudeService.ts`. The plumbing for stage 1 (RAG over the Lossless corpus) is partially built via the `chroma` MCP server — the missing piece is a per-template `rag-context:` block in the cft fence that names which collections to query and how many results to inject. Spec lives in `context-v/specs/` — open it before starting the multi-stage build.
--- a/src/docs/templates/standards-and-specs-profile.md
+++ b/src/docs/templates/standards-and-specs-profile.md
@ -0,0 +1,385 @@
 ---
 title: Standard or Spec Profile (Analyst Draft)
 applies-to-paths:
  - "Sources/Standards-and-Specs/**"
  - "Standards-and-Specs/**"
 description: Generates an analyst-grade profile of an open spec or standard — covering authorship, current stewardship, ecosystem position, three-tier adoption, named critics, and stewardship transitions. Tuned for the strategist's lens, not the implementer's.
 date_created: 2026-05-27
 date_modified: 2026-05-27
 ---
 # About this template
 Use this for files under `Sources/Standards-and-Specs/` whose body is empty or whose curated lead-in (a thesis paragraph, a stake-in-the-ground take) has been authored but the analytical body is missing.
 A **spec profile** is the draft an innovation consultant hands to a partner who has just asked "should we care about this spec?" It is not a tutorial, not implementer documentation, not marketing for the spec. It explains who created it, who steers it now, what coordination problem it solves, who has implemented it across three tiers (incumbents / challengers / innovators), who has explicitly declined, and what the public critics are saying.
 The template handles five flavors of authority:
 1. **De-jure** — formal standards body with quasi-legal authority (W3C, IETF/RFC, ISO, IEEE, ECMA, NIST).
 2. **Industry consortium** — non-vendor-controlled multi-stakeholder body (Khronos, Linux Foundation, OpenJS, OASIS, CNCF).
 3. **Vendor-led-open** — single vendor created and primarily maintains it but has published under an open license (MCP from Anthropic, OpenAPI's early years at Wordnik).
 4. **Community** — published by an individual or informal group with no parent org (llms.txt by Jeremy Howard, AGENTS.md after the OpenAI handoff).
 5. **De-facto** — never formally published but treated as a spec because the dominant implementation defines it (README convention, package.json shape, the curl interface).
 The model classifies the authority type from the spec name, primary URL, and search results before drafting Identity & Status. Every downstream section's treatment depends on which type.
 This template runs on `sonar-deep-research` and skips image embedding by design — deep research returns better citation density and analytical length but unreliable image metadata. Visual identity for specs (logos, diagrams) lives in frontmatter, generated separately.
 ```cft
 provider: perplexity
 model: sonar-deep-research
 return-citations: true
 return-images: false
 # Idle-only safety — ceiling disabled. The per-chunk idle timer (270s for
 # deep-research, inherited from the model-class default) is the sole
 # safety mechanism: as long as Perplexity sustains bytes, the run is
 # allowed to complete. A complete spec profile can stretch 6-9K words
 # given the three-tier adoption section + named editors + critique
 # framing; any wall-clock cap risks cutting the tail. Set this to a
 # positive value (e.g., 3600000 for 60 min) if you want a hard ceiling.
 request-timeout-ms: 0
 # Generous output-token budget. Perplexity's default for sonar-deep-research
 # is ~8192 tokens (~6K words), which silently truncates this template's
 # back half by ending the stream cleanly with finish_reason: length
 # mid-skeleton (looks like a healthy completion except the last several
 # sections never appear). Bumped to 24000 (~18K-word budget) so the full
 # 6-9K-word draft can land with headroom for the three-tier adoption
 # section, deeper implementation cards, named critics, and the frontier.
 max-tokens: 24000
 system: |
  You are writing the analyst-grade profile of a STANDARD or SPEC named
  "{{basename}}" for an innovation consultant's vault.
  This is NOT an implementer's reference. Your reader is a strategist,
  founder, or operator who needs to understand the spec's significance,
  governance, and adoption landscape — not how to write a conformant
  implementation. Conformance matrices, MUST/SHOULD/MAY discipline, and
  wire-format details are out of scope. Stewardship, adoption tiers,
  political fault lines, and named critics are in scope.
  A SPEC PROFILE answers four questions a partner would ask:
  - WHAT is this spec, who wrote it, and who currently steers it?
  - WHY does it matter — what does it unlock, what has shifted because
    it exists, what has died because it exists?
  - WHO is using it (in three tiers: incumbents, challengers, innovators)
    and who is publicly NOT using it?
  - WHERE is the frontier — pending disputes, likely next versions, the
    political fault lines that will shape the spec's evolution?
  CLASSIFY THE AUTHORITY TYPE FIRST.
  Every spec falls into one of these five governance models, and your
  treatment of every section downstream depends on which one:
  - DE-JURE — formal standards body with legal or quasi-legal authority
    (W3C, IETF/RFC, ISO, IEEE, ECMA, NIST). Editors are publicly named
    in the spec itself; working group archives are public; decisions
    follow formal procedure.
  - INDUSTRY CONSORTIUM — non-vendor-controlled multi-stakeholder body
    (Khronos, Linux Foundation, OpenJS Foundation, OASIS, CNCF).
    Members are named companies; governance is documented in foundation
    bylaws.
  - VENDOR-LED-OPEN — a single vendor created and primarily maintains
    the spec but has published it under an open license and accepts
    community contributions (MCP from Anthropic, OpenAPI's early years
    at Wordnik).
  - COMMUNITY — published by an individual or informal group with no
    parent org (llms.txt by Jeremy Howard, AGENTS.md in its current
    form after the OpenAI handoff). Stewardship is by reputation and PR.
  - DE-FACTO — never formally published as a spec but treated as one
    by the ecosystem because the dominant implementation defines the
    behavior (the README convention, package.json's shape, the curl
    interface). Stewardship is implicit and shifts with the dominant
    implementation.
  Determine the authority type from the spec name, its primary URL, and
  search results before drafting Identity & Status. State the type
  explicitly in the Identity & Status section AND in the Snapshot
  callout line.
  Frontmatter for "{{basename}}":
  {{frontmatter}}
  FRONTMATTER FIELDS TO SURFACE FOR LATER PROMOTION:
  A curator promotes the named-creator and named-steward findings into
  structured frontmatter after the draft is written. To make that pass
  easy, surface these explicitly in the Snapshot lede callout AND in
  full detail in the Identity & Status section:
  - created_by — list of authors at inception (people OR orgs; prefer
    named persons + their affiliation at inception where public)
  - created_year — four-digit year of first public release
  - original_publisher — the org that first published it (if different)
  - maintained_by — current stewards (people, orgs, or "community")
  - stewardship_type — one of: de-jure | consortium | vendor-led-open
    | community | de-facto
  RESEARCH DISCIPLINE:
  - Use Perplexity's web search aggressively. For a spec profile,
    accuracy on governance and adoption matters more than encyclopedic
    breadth.
  - For every factual claim — author name, version date, implementation,
    transition story, critique, license terms — append an inline numeric
    citation marker [1], [2], etc. corresponding to the search-result
    order.
  - Quote phrasing from primary sources where useful: the spec text
    itself, editor blog posts, working-group meeting minutes, GitHub
    issues that captured the dispute, conference talks where the
    editors presented the spec.
  - Prefer primary surfaces: the spec's canonical URL, the working
    group's mailing list archive or GitHub repo, the editors' own
    posts, the implementing-org's announcements. Aggregator pages
    (Wikipedia, blog roundups, "X explained" articles) are fallbacks
    for orientation only.
  - Do NOT cite this Perplexity response itself, only the underlying
    sources.
  EDITORIAL STANCE — name the humans, not just the orgs:
  Specs are written by people. Training data over-represents the
  sponsoring org's logo and under-represents the named editors who
  actually drafted the text and made the design choices. Counteract
  this systematically:
  - In `created_by` and `maintained_by`, prefer NAMED PERSONS where
    public. "Anthropic" is acceptable; "Anthropic + David Soria Parra
    + Justin Spahr-Summers" is what an analyst wants.
  - In Governance & Stewardship, name the editors, working group
    chairs, and sponsoring partners with their roles. Not "the
    working group decided" — name the chair and the decision.
  - For DE-JURE specs, the editor names are on the cover of the spec
    itself; surface them.
  - For VENDOR-LED-OPEN specs, name both the originating team (the
    people who shipped it) AND the cross-vendor partners who have
    since contributed substantively.
  - For COMMUNITY specs, the originator's identity is the spec's
    political center; name them prominently.
  THREE-TIER ADOPTION FRAMING — this is STRUCTURAL, not editorial:
  The Adoption section is partitioned into three sub-buckets, and each
  has its own discipline:
  - INCUMBENTS — the dominant implementations that the ecosystem
    treats as canonical or near-canonical. This includes the
    reference implementation (if there is one), the most-deployed
    OSS implementation, and the commercial implementations from
    market leaders. 4-8 named entries. Big tech BELONGS HERE if it
    has implemented the spec — do not suppress.
  - CHALLENGERS — production-grade alternative implementations that
    compete with the incumbents on completeness or performance,
    often from mid-sized companies or well-funded startups. 4-8
    entries. These are the implementations that keep the incumbents
    honest.
  - INNOVATORS — early-stage, experimental, or research
    implementations exploring the spec's edges, novel extensions, or
    unusual integration patterns. Often single-developer or
    research-lab projects. 4-8 entries. This is where the next
    extensions will come from.
  Within each bucket, render entries in `[Name](url) — one-line
  description` form. After the three buckets, surface NOTABLE HOLDOUTS
  — orgs that explicitly declined to implement, forked, or are running
  incompatible alternatives — as a separate short paragraph. The
  holdouts are often as informative as the adopters about where the
  spec's design choices have made enemies.
  IMPLEMENTATION CARD SHAPE (deeper treatment for top entries):
  For the top 2-3 implementations PER TIER — the ones a partner would
  expect to be briefed on by name — produce a deeper card:
      #### [Implementation Name](https://homepage.url)
      **Steward**: the org or individual maintaining this implementation,
      with one-line context on their relationship to the spec (authoring
      participant, early adopter, late convert, fork). Cite. [N]
      **Coverage of the spec**: which parts of the spec are implemented,
      which are explicitly not, any extensions added beyond spec. Cite. [N]
      **Adoption signal**: who uses this implementation in production
      (named customer, named project, GitHub stars / downloads if
      relevant). Cite. [N]
      **Why it matters**: one sentence on the strategic significance —
      the distribution channel it commands, the conformance bar it sets,
      the political coalition it represents. Cite. [N]
  STEWARDSHIP TRANSITIONS — surface these explicitly:
  When the created-by org and the maintained-by org are different, that
  transition is itself a story worth its own paragraph. Canonical
  examples to model the depth of treatment on:
  - AGENTS.md: OpenAI → community (via Sourcegraph)
  - OpenAPI: Wordnik → SmartBear → Linux Foundation (OpenAPI Initiative)
  - HTTP: Tim Berners-Lee/CERN → IETF / W3C
  - JSON: Douglas Crockford → IETF (RFC 8259) + Ecma (ECMA-404) — two
    parallel stewards
  In Governance & Stewardship, when a transition has happened, give it
  a dedicated paragraph: WHEN the transition was announced, WHO handed
  to WHOM, WHAT triggered the handoff, what changed in governance pace
  or direction as a result. Cite the transition announcement.
  CRITIQUE — be candid, do not adjudicate:
  A spec without public critics is either (a) too young to have
  attracted critique, or (b) so dominant that nobody bothers. For any
  spec older than 18 months, there should be at least 2-3 publicly
  named critics worth surfacing in Critique & Open Disputes. Name them,
  link to their critique, summarize their argument in one sentence
  each. Do not rebut — the analyst's job here is to surface the
  disagreement so the partner reading the memo knows the political
  landscape, not to settle it.
  LINKS AND WIKILINKS:
  - For implementation names, editor names, and source links, use
    `[Name](https://url)` form.
  - Do NOT invent `[[wikilink]]` syntax. The curator will promote
    names to vault wikilinks during the curation pass. If you happen
    to know a canonical adjacent spec the curator has already vaulted
    (e.g., "JSON-RPC", "OAuth 2.0", "OpenAPI"), surface the name as
    plain text in the Adjacent Specs section so the curator can
    wikilink it later.
  CALIBRATION ON LENGTH:
  This is a deep-research run. Lean long, not short. A complete spec
  profile is roughly 6,000-9,000 words of body, with 12-24 named
  implementations across three tiers + holdouts, 2-3 deeper cards per
  tier for the most strategically significant implementations, a
  documented stewardship transition where relevant, and 2-4 named
  critics in Critique & Open Disputes. Better to over-enumerate and
  let the curator prune than to under-enumerate and leave the analyst
  guessing.
 ```
 # Snapshot
 - One-paragraph italicized lede (max 2 sentences) that captures the spec's significance in a single beat. Voice: the analyst opening their memo. Use markdown italics: `_..._`.
 - Immediately after the lede, a one-line bold callout in this exact shape: `**Created by** {name(s)} ({year}) · **Maintained by** {name(s)} · **Type:** {de-jure | consortium | vendor-led-open | community | de-facto}`. This is the spec's identity at a glance.
 - Then the headline stat or framing quote — one cited statement (an adoption number, a developer-survey result, a notable adopter's framing of why they picked it, a critic's signature objection) from a credible source. Format as a blockquote (`> "..."`).
 - Then 2-3 sentences orienting the reader: what is this spec, what does it constrain or enable, and why is it worth profiling right now.
 # The Question this Spec Answers
 - One paragraph (3-5 sentences) stating the coordination failure, interop gap, or pain point this spec was created to address.
 - What was the world like in the period before this spec existed? What were people doing instead — bespoke per-vendor integrations, fragmented forks, lock-in to proprietary APIs?
 - What does the spec's existence allow that wasn't possible (or was prohibitively expensive) before? Be specific — name the kinds of products, integrations, or workflows that the spec made tractable.
 # Identity & Status
 - **Full name** and commonly-used abbreviation.
 - **Type:** protocol / data format / API / behavior spec / schema / process spec / RFC / convention.
 - **Authority type:** state explicitly which of the five — de-jure / industry consortium / vendor-led-open / community / de-facto — and cite the evidence.
 - **Created by:** named persons and orgs at inception, with affiliations as of the inception date.
 - **Created year:** four-digit year of first public release.
 - **Original publisher:** if different from creators.
 - **Maintained by:** current stewards (people, orgs, foundation, or "community").
 - **Current version** and **lifecycle stage** — use whatever taxonomy the spec itself uses (Draft / Working / Recommendation / Stable / Deprecated; or numbered milestones; or "alive / abandoned" for community specs).
 - **License:** publishing license of the spec text itself (CC-BY? MIT? bespoke?) and patent-grant terms if relevant.
 - **Canonical URL** of the spec.
 # Why It Matters
 What this spec unlocks, and what has shifted because it exists.
 - 3-5 cited bullets covering WHAT IT UNLOCKS: the interop, coordination, or portability story that the spec enables. Be specific about what kinds of integrations, products, or customer escapes from lock-in become possible. Where possible, quote a founder, editor, or implementer who named the unlock.
 - 3-5 cited bullets covering WHAT IMPACT IT HAS HAD (or is having): named adopters that shipped because of it, named projects that died because of it (or that the spec rendered unnecessary), named market shifts in vendor positioning, named regulatory or procurement-policy changes that referenced the spec. Surface both adoption AND resistance — who explicitly chose to ignore or fork, and why their stated reasoning matters.
 - If the spec is too young to have demonstrable impact yet (under 12 months from first public release), say so explicitly and pivot to predicted impact based on the early-adoption signals — naming the early-adopting orgs and quoting their stated reasons for adopting early.
 # Position in the Ecosystem Stack
 - **What it depends on** — the layers underneath that the spec assumes. Name the underlying specs or de-facto conventions (e.g., MCP depends on JSON-RPC and HTTP; A2A depends on HTTP/SSE; AGENTS.md depends on markdown's conventions). For each, cite where the dependency is documented in the spec text.
 - **What depends on it** — the layers that have been built on top of this spec by other authors. 4-8 named downstream specs, protocols, or conventions with one-line each on how they extend or consume this spec.
 - **Companion specs** — specs deliberately designed to work alongside this one, often by the same authoring group (e.g., OAuth 2.0 companions: PKCE, dPoP; MCP companions: capability negotiation specs). One paragraph naming 3-5 companions.
 - **Strategic positioning:** one sentence — what part of the stack does this spec colonize, and why does that position matter for whoever wants to compete with the dominant implementations?
 # Lineage
 - **Predecessors** — earlier specs or de-facto conventions in the same problem space. For each, one-line on what they got wrong, what they got right that this spec inherited, and why the ecosystem ultimately moved on.
 - **Parallel efforts** — concurrent specs from other authoring groups solving the same coordination problem differently. Name 2-4, with one-line each on their distinguishing bet and current adoption signal relative to this spec.
 - **Likely successors** — emerging specs or research efforts that may eventually supersede this one, OR an explicit statement that no successor is visible yet. If a successor exists, name its authoring group and current status.
 # Governance & Stewardship
 - **Editors / chairs / sponsoring partners** — named individuals with their roles. For DE-JURE specs, lift the editor list from the spec's cover. For VENDOR-LED-OPEN specs, name both the originating team and the cross-vendor contributors who have committed substantively. For COMMUNITY specs, the originator's identity is the political center — name them prominently.
 - **Where decisions get made** — the mailing list, GitHub org, working-group meeting, or governance forum. Cite the URL of the public archive.
 - **Pace** — release cadence, date of last meaningful update, typical time between major versions.
 - **Versioning policy** — semver / calendar / named milestones / RFC numbering / "live spec, no versions."
 - **STEWARDSHIP TRANSITIONS** — if `created_by` and `maintained_by` differ, this gets its own paragraph: when the transition was announced, who handed to whom, what triggered the handoff (loss of vendor interest, community pressure, foundation absorption, fork-takeover), what changed in pace or direction as a result. Cite the transition announcement post or RFC.
 - **Political fault lines** — 2-3 sentences naming the working-group disputes (or implementer disputes) that have been publicly archived. Where is the coalition holding together and where is it under strain? Cite the GitHub issue, mailing-list thread, or talk where the dispute is most visible.
 # Adoption — by Tier
 The three-tier framing is structural. Each tier has its own subsection with its own discipline. After the three tiers, surface notable holdouts as a separate short paragraph.
 ## Incumbents
 - 4-8 dominant implementations the ecosystem treats as canonical or near-canonical. The reference implementation belongs here if there is one. Big tech belongs here if it has implemented the spec — do not suppress.
 - Format: `[Implementation Name](https://url) — one-line on what they ship and their relationship to the spec`. Cite each.
 - After the flat list, produce 2-3 deeper IMPLEMENTATION CARDS for the most strategically significant entries per the card shape defined in the system prompt.
 ## Challengers
 - 4-8 production-grade alternative implementations that compete with the incumbents on completeness, performance, or coverage of optional spec capabilities. Often from mid-sized companies or well-funded startups.
 - Same format as Incumbents, with 2-3 deeper cards for the most significant.
 ## Innovators
 - 4-8 early-stage, experimental, or research implementations exploring the spec's edges, novel extensions, or unusual integration patterns. Often single-developer projects, research-lab outputs, or open-source projects with small but engaged communities.
 - Same format as Incumbents, with 2-3 deeper cards for the most significant.
 ## Notable Holdouts
 - One short paragraph (3-6 sentences) naming 2-5 orgs or projects that explicitly declined to implement, forked the spec, or are running incompatible alternatives. Name each holdout, link to where they stated their position, summarize their reasoning in one sentence. The holdouts are often as informative as the adopters about where the spec's design choices have made enemies.
 # Critique & Open Disputes
 - 2-4 named critics (people or orgs) with their argument summarized in one sentence each. Format: `**Critic Name** ([affiliation](url)) — "their argument in one sentence" [N]`.
 - 1-2 sentences on the known limitations the editors themselves have admitted (in mailing-list posts, GitHub issues, conference Q&A).
 - 1-2 sentences on the working-group fault lines that are publicly archived — where members have voted against each other, where a proposal was withdrawn under pressure, where a fork was threatened.
 - Do NOT rebut the critique here. Surface the disagreement; do not adjudicate it. The partner reading the memo will form their own view.
 # Frontier & Open Questions
 - 4-6 bullets, each a specific open question that the working group, implementers, or critics are actively debating. Frame each as a question, not a statement.
 - Pair each question with a one-sentence note on which editors, working groups, or implementations are most likely to drive the resolution.
 - Where pending RFCs / extensions / next-version drafts are public, link to them and note their current status (proposed / under review / rejected / merged).
 # Media, Voices, and Coverage
 - 6-12 bullets covering the editors' own posts and talks, the credible critics' coverage, the implementer blogs that document real-world deployment, and the podcasts/conferences where the spec is regularly debated.
 - Format: `**Name** — Platform — one-line note on their angle / why they are worth following`. Include a primary URL link.
 - Sub-group with `## Editor & Maintainer Voices`, `## Implementer Coverage`, `## Critic Coverage`, `## Conferences & Working Group Forums` if the list is long enough to warrant it.
 - Prefer specialized voices over generalist tech press. Prefer the editors' own posts over recapped coverage.
 # Adjacent Specs and Standards
 - 4-8 plain-text spec names (no wikilink syntax — the curator wikilinks during curation) that an operator working with this spec would want to explore next.
 - Mix of: predecessor and successor specs, companion specs, competing specs from parallel efforts, and foundational specs this one depends on.
 - Format: `- <Spec Name> — one-line on how it adjoins this spec`.
 ***
 # User Notes
 Anything below the `***` line is excluded from the request. Use this zone for:
 - Hand-curated notes on the spec's relevance to your own work, portfolio, or thesis.
 - Curator's wikilink resolution notes during the curation pass — which named editors, implementations, or adjacent specs have vault entries that need linking back.
 - Iteration history while you refine the template for specific spec types.
 - Tuning notes on which sections came back thin and need a re-run.
 ## Multi-stage roadmap (deferred)
 This v1 template is intentionally single-stage: one Perplexity Deep Research run produces the full draft. The curator promotes implementation names to `[[wikilink]]` form, adds vault-specific cross-references to other vaulted specs, and adds `:::tool-showcase` blocks for vault-tracked implementations during the curation pass.
 The deferred multi-stage version mirrors the market-map roadmap and is tracked in `context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md`:
 1. **RAG pre-flight** — pull canonical Lossless sources adjacent to this spec (other vaulted specs in `Sources/Standards-and-Specs/`, the `open-specs-and-standards` study, any prior commentary in `concepts/` or `Tooling/`) as primed context.
 2. **Perplexity research stage** — deep research with the RAG context as primer, eliminating the wikilink-invention problem and letting the model name actual vault entries.
 3. **Claude editing stage** — an editorial pass that enforces the analyst voice, prunes over-enumeration, sharpens the three-tier adoption boundaries, and emits the final `[[wikilink]]` form.
--- a/src/docs/templates/toolkit-profile.md
+++ b/src/docs/templates/toolkit-profile.md
@ -3,46 +3,130 @@
 applies-to-paths:
  - "Tooling/**"
 description: Generates a structured profile for a company / service / app / open-source repo.
 date_created: 2026-05-09
 date_modified: 2026-05-22
 ---
 # About this template
-Use this for files under `Tooling/` whose body is empty or near-empty. Run "Apply directory template to current file" while viewing the target. The runtime sends the heading skeleton below as the user prompt to Perplexity Deep Research; the model returns markdown that follows the structure with inline citations.
+Use this for files under `Tooling/` whose body is empty or near-empty. Run "Apply directory template to current file" while viewing the target — a dialog lets you pick the Perplexity model (the `model:` below is the default). The runtime sends the heading skeleton below as the user prompt; the model returns markdown that follows the structure with inline citations.
 Model guidance: `sonar-pro` (default) does a fast, clean, single-pass grounded search. `sonar-deep-research` does genuine multi-query research across many sources — pick it from the dialog when you want depth and can wait a few minutes. Avoid `sonar-reasoning-pro` here: reasoning models spend their budget thinking rather than searching and dump a large `<think>` block into the file.
 The bullets under each heading are *instructions to the model*, not literal content. The model fills each section based on those instructions.
 ## Scoping search to the right entity
 Many tool/company names collide with unrelated namesakes (e.g. `NATS` the messaging system vs. `NATS` the UK air traffic authority). Two levers keep the model on-target:
 - The `system:` prompt above hard-pins the entity to its **name** (`{{basename}}`, the filename) and `{{url}}` — *not* the scraped `{{title}}`, which is usually a marketing tagline (e.g. Kubernetes' title is "Production-Grade Container Orchestration"). It then tells the model to **triage source quality itself** — favour editorial authority over SEO ranking, judge each page on its merits — rather than trusting the top search hits. Job postings are denied automatically by the runtime (Indeed, Glassdoor, ZipRecruiter, USAJobs), since a job listing is never a product source; everything else is the model's judgment call.
 - Allowlisting is a last resort, not the workflow — curating domains by hand is the same labour as searching yourself. Use it **only** when a name is so collision-prone that entity-anchoring and triage still can't recover it (e.g. `NATS` the messaging system vs. `NATS` the UK air traffic authority). For those cases, add a **`cf_search_domains:`** list to the *target file's* frontmatter (not this template — this template is generic). Plain entries allowlist; a leading `-` denylists; max 10 total. Example for a `NATS.md` profiling `nats.io`:
  ```yaml
  cf_search_domains:
    - nats.io
    - docs.nats.io
    - github.com
    - synadia.com
    - cncf.io
  ```
  An allowlist restricts Perplexity to exactly those domains, so the wrong entity cannot come back. A template-wide `search-domains:` key in the `cft` block above works the same way if you ever want it applied to every file.
 ```cft
 provider: perplexity
-model: sonar-deep-research
+model: sonar-pro
 search-recency: month
 return-citations: true
 return-images: false
 system: |
-  You are a research analyst profiling "{{title}}". Use Perplexity's web search
+  You are a research analyst writing a factual profile of ONE specific
-  aggressively for every section. For every factual claim, append an inline
+  entity and no other: the product, service, or company known as
-  numeric citation marker like [1], [2] corresponding to the search-result order.
+  "{{basename}}", whose canonical website is {{url}}. Note that "{{title}}"
-  Quote phrasing from primary sources where useful. Prefer first-party sources,
+  is that website's tagline or page heading — useful context, but it is
-  official filings, and reputable industry publications. Use the entity's
+  NOT the entity's name. Search for, and reason about, "{{basename}}".
-  existing metadata as starting context.
+
  SEARCH, do not recall. Your training knowledge is stale and is not
  citable. Run a fresh web search for every section — value proposition,
  architecture, history, funding, pricing, competitors, recent news — as
  separate, specific queries. Base every claim on what the search results
  actually say, not on what you already know. A profile assembled from
  memory instead of search results is a failed profile.
  DISAMBIGUATION — read this before searching. Software, company, and
  product names collide constantly. Many search results that match the
  name "{{basename}}" will be about a DIFFERENT entity that merely shares
  the name — a namesake company, a person, a place, a government body, an
  acronym. Before using any result, confirm it is about the exact entity
  at {{url}}. Discard anything about an unrelated same-named entity, no
  matter how highly it ranks. Anchor every search query on "{{basename}}"
  together with its own domain.
  SOURCE QUALITY — you are the editor; judge every source, do not just
  take the top results. Search rankings are SEO-driven, so the highest
  hits are often content marketing that ranks well but carries little
  authority. Triage the full set of results you get and cite by merit.
  Higher authority — lead with these: the entity's own site, docs, GitHub
  org, and changelog; official filings, regulatory documents, and standards
  bodies; established journalism and trade press with real editorial
  standards (e.g. The Economist, the FT, IEEE Spectrum, The Register, Ars
  Technica); recognized analyst firms; primary interviews and conference
  talks by the entity's own people.
  Lower authority — use only when nothing better covers a fact, and never
  as the sole basis for a significant claim: thin "what is X" listicles and
  content farms; scraper or aggregator pages; pages that exist to rank for
  a keyword rather than to inform. Note: a consulting firm's or a vendor's
  blog post is perfectly fine when that specific page is substantive,
  first-hand, and accurate — judge the page on its merits, not the domain
  it sits on. The only hard reject is a page about a different same-named
  entity.
  When sources conflict, prefer the more authoritative and more recent one.
  If a section genuinely has no credible source, write "No reliable source
  found" rather than padding it with low-authority filler.
  LENGTH AND ECONOMY — this matters as much as accuracy. The output is a
  scannable reference card, NOT an essay or a research report. Be terse
  and fact-dense:
  - Treat the sentence and paragraph counts in each section's instructions
    as hard ceilings, not suggestions. "2-3 sentences" means at most three
    sentences. "One short paragraph" means one. Never exceed them.
  - One fact per sentence. Cut all filler: no throat-clearing, no
    restating the question, no "it is worth noting that", no concluding
    sentence that re-summarizes what you just wrote.
  - Prefer bullets and tables over prose wherever a section allows it.
  - A section with little substance should be SHORT. Never pad a thin
    section to look thorough — a short correct section beats a long padded
    one. Brevity is the goal; length is not evidence of quality.
  - Output only the finished profile. Do not narrate your research
    process, your reasoning, or your search strategy.
  For every factual claim, append an inline numeric citation marker like
  [1], [2] corresponding to the order of the search results. Quote phrasing
  from primary sources where useful.
  Entity metadata for grounding:
  {{frontmatter}}
 ```
-# Features
+# Value Proposition & Features
 - Summarize the value proposition in 2-3 sentences. 
 - Describe the core product features in 2–3 sentences each.
 - Bullet 5–8 features in priority order.
 - If the product has a notable architecture, data flow, or pipeline that a diagram clarifies (e.g., an ingest → process → serve flow, an agent/orchestrator topology, a multi-tier deployment shape), render a `mermaid` codefence describing it. If a diagram does not add insight, omit it entirely — do not force one.
 {{include: mermaid-discipline}}
 ## Screenshots
- If three official screenshots are publicly available, list their URLs as bullets.
+- If three official screenshots are publicly available, list their URLs wrapped in `![Alt Text](url)`, each one on a new line.
- For each, write a 1-sentence caption. If none are publicly available, write "No publicly available screenshots."
+- For each, write a 1-sentence caption. If none are publicly available, skip.
 ## Product Roadmap / Announcements
 - Start with leading text `As of {today's date},`
 - Public roadmap items and product announcements from the past 6 months.
 - Use dated bullets, most recent first. Cite each item.
 ## Recent Developments
- News and developments from the past 90 days. Cite sources inline.
+- News and developments from the past 90 days. Cite sources inline, preserve related reference definitions in response.
 # History and Origin Story
 - Founding story, founders, key inflection points. One short paragraph.
@ -58,12 +142,15 @@ ## Notable Team Members
 # Market Sizing
 ## Category, Market Size, and Category Growth
 - Define what category or categories this toolkit/tooling entry (the app, service, or company) is likely in based on available data. 
 - Detail any estimates of market size and category growth, with priority for established analyst firms, consulting groups, and financial journalism.
 ## Pricing
 - Markdown table of pricing tiers if published.
 - Note "no public pricing" if not.
 ## Revenue Trajectory Estimates
- Estimated or reported revenue / ARR. Cite source per figure.
+- Estimated or reported revenue / ARR. If not available, skip. Cite source per figure.
 # Competitive Landscape
@ -73,6 +160,9 @@ ## Who it's for, who it's not for
 ## Viable Alternatives
 - 3–5 alternatives, one bullet each, with a brief rationale.
 ## Competitor Table
 - Create a Markdown GFM table, with competitors, their names wrapped in markdown links, on the left, and a brief description on the right. 
 ***
 # User Notes
--- a/src/modals/DirectoryTemplateRunModal.ts
+++ b/src/modals/DirectoryTemplateRunModal.ts
@ -0,0 +1,125 @@
 import { Modal, Setting, type App } from 'obsidian';
 import type { ParsedTemplate } from '../services/directoryTemplateService';
 export interface TemplateRunChoice {
    template: ParsedTemplate;
    title: string;
 }
 // Perplexity models offered in the run modal, in recommendation order.
 const MODEL_CHOICES: { id: string; label: string }[] = [
    { id: 'sonar-pro', label: 'Sonar Pro — grounded search + citations, fast' },
    { id: 'sonar-reasoning-pro', label: 'Sonar Reasoning Pro — adds chain-of-thought' },
    { id: 'sonar', label: 'Sonar — lightweight, fastest' },
    { id: 'sonar-reasoning', label: 'Sonar Reasoning — lightweight + reasoning' },
    { id: 'sonar-deep-research', label: 'Sonar Deep Research — exhaustive, minutes-long' },
 ];
 function cftModel(choice: TemplateRunChoice): string {
    const m = choice.template.cftConfig['model'];
    return typeof m === 'string' && m.length > 0 ? m : 'sonar-pro';
 }
 /**
 * Run dialog for "Apply directory template to current file": pick which
 * template to apply (when more than one matches) and which Perplexity model
 * to run it with. The model defaults to the template's cft `model:` but the
 * selection here overrides it for this run only.
 */
 export class DirectoryTemplateRunModal extends Modal {
    private readonly choices: TemplateRunChoice[];
    private readonly onRun: (template: ParsedTemplate, model: string) => void;
    private selectedIndex = 0;
    private selectedModel: string;
    constructor(
        app: App,
        choices: TemplateRunChoice[],
        onRun: (template: ParsedTemplate, model: string) => void,
    ) {
        super(app);
        if (choices.length === 0) {
            throw new Error('DirectoryTemplateRunModal requires at least one template choice');
        }
        this.choices = choices;
        this.onRun = onRun;
        this.selectedModel = cftModel(this.selected());
    }
    /** The currently selected choice. choices is non-empty (checked in ctor). */
    private selected(): TemplateRunChoice {
        const c = this.choices[this.selectedIndex];
        if (c) return c;
        const first = this.choices[0];
        if (first) return first;
        throw new Error('DirectoryTemplateRunModal: choices unexpectedly empty');
    }
    onOpen(): void {
        this.render();
    }
    onClose(): void {
        this.contentEl.empty();
    }
    private render(): void {
        const { contentEl } = this;
        contentEl.empty();
        contentEl.createEl('h3', { text: 'Apply directory template' });
        if (this.choices.length > 1) {
            new Setting(contentEl)
                .setName('Template')
                .setDesc('Which template to apply to this file.')
                .addDropdown((dd) => {
                    this.choices.forEach((c, i) => {
                        dd.addOption(String(i), c.title);
                    });
                    dd.setValue(String(this.selectedIndex));
                    dd.onChange((v) => {
                        this.selectedIndex = Number(v);
                        // Reset the model to the newly chosen template's default.
                        this.selectedModel = cftModel(this.selected());
                        this.render();
                    });
                });
        } else {
            new Setting(contentEl)
                .setName('Template')
                .setDesc(this.selected().title);
        }
        const cftDefault = cftModel(this.selected());
        new Setting(contentEl)
            .setName('Model')
            .setDesc(`Perplexity model for this run. Template default: ${cftDefault}.`)
            .addDropdown((dd) => {
                for (const m of MODEL_CHOICES) dd.addOption(m.id, m.label);
                // Keep an unlisted cft model selectable if the template names one.
                if (!MODEL_CHOICES.some((m) => m.id === this.selectedModel)) {
                    dd.addOption(this.selectedModel, this.selectedModel);
                }
                dd.setValue(this.selectedModel);
                dd.onChange((v) => {
                    this.selectedModel = v;
                });
            });
        new Setting(contentEl)
            .addButton((b) => b
                .setButtonText('Run')
                .setCta()
                .onClick(() => {
                    const chosen = this.selected();
                    const model = this.selectedModel;
                    this.close();
                    this.onRun(chosen.template, model);
                }))
            .addButton((b) => b
                .setButtonText('Cancel')
                .onClick(() => {
                    this.close();
                }));
    }
 }
--- a/src/services/directoryTemplateService.ts
+++ b/src/services/directoryTemplateService.ts
@ -1,3 +1,5 @@
 import * as https from 'node:https';
 import type { IncomingMessage } from 'node:http';
 import type { App } from 'obsidian';
 import { normalizePath, Notice, parseYaml, stringifyYaml, TFile } from 'obsidian';
@ -43,6 +45,8 @@ interface PerplexityPayload {
    return_images: boolean;
    return_related_questions: boolean;
    search_recency_filter?: string;
    search_domain_filter?: string[];
    max_tokens?: number;
 }
 export interface PerplexitySource {
@ -413,13 +417,51 @@ export function interpolate(text: string, ctx: InterpolationContext): string {
    });
 }
 // Job boards and recruiting sites are never a credible source for a product
 // profile, yet they rank highly for company names — especially names that
 // collide with a large employer (e.g. "NATS" also = National Air Traffic
 // Services). Denied on every run UNLESS the run declares an explicit
 // allowlist: an allowlist already restricts results, and Perplexity caps
 // search_domain_filter at 10 entries, so deny entries would only burn slots.
 const JOB_BOARD_DENYLIST = [
    '-indeed.com',
    '-glassdoor.com',
    '-ziprecruiter.com',
    '-usajobs.gov',
 ];
 // Perplexity's hard cap on search_domain_filter length.
 const SEARCH_DOMAIN_CAP = 10;
 /**
 * Normalize a cft `search-domains:` value or a target file's
 * `cf_search_domains:` frontmatter value into a clean string list. Accepts a
 * YAML array or a comma-separated string. A leading `-` marks a denylist
 * entry; everything else is an allowlist entry.
 */
 function parseDomainList(raw: unknown): string[] {
    const items = Array.isArray(raw)
        ? raw
        : typeof raw === 'string'
            ? raw.split(',')
            : [];
    return items
        .filter((x): x is string => typeof x === 'string')
        .map((s) => s.trim())
        .filter((s) => s.length > 0);
 }
 function buildPayload(
    template: ParsedTemplate,
    systemPrompt: string,
    userPrompt: string,
    targetDomains: string[] = [],
    modelOverride?: string,
 ): PerplexityPayload {
    const cfg = template.cftConfig;
-    const model = typeof cfg.model === 'string' ? cfg.model : 'sonar-deep-research';
+    const model = (modelOverride && modelOverride.length > 0)
        ? modelOverride
        : (typeof cfg.model === 'string' ? cfg.model : 'sonar-pro');
    const recency = typeof cfg['search-recency'] === 'string'
        ? cfg['search-recency']
        : undefined;
@ -438,6 +480,39 @@ function buildPayload(
        return_related_questions: false,
    };
    if (recency) payload.search_recency_filter = recency;
    // search_domain_filter: template-level `search-domains:` (generic) + the
    // target file's `cf_search_domains:` frontmatter (entity-specific) + the
    // built-in job-board denylist. Declared entries win over the denylist on
    // truncation to the 10-domain cap.
    const declared = [
        ...parseDomainList(cfg['search-domains']),
        ...targetDomains,
    ];
    const hasAllowlist = declared.some((d) => !d.startsWith('-'));
    const merged = hasAllowlist ? declared : [...declared, ...JOB_BOARD_DENYLIST];
    const domainFilter = [...new Set(merged)].slice(0, SEARCH_DOMAIN_CAP);
    if (domainFilter.length > 0) {
        payload.search_domain_filter = domainFilter;
    }
    // Per-template `max-tokens:` override — Perplexity's default output cap
    // (~8192 for sonar-deep-research) silently truncates long analyst-grade
    // templates by ending the stream cleanly with finish_reason: "length"
    // mid-skeleton. The symptom looks identical to a healthy run except
    // that the back half of the section skeleton never appears. Templates
    // that legitimately want 6-9K-word bodies must bump this explicitly.
    // Accept number or numeric-string; ignore non-positive / non-numeric.
    const maxTokensRaw = cfg['max-tokens'];
    const maxTokens = typeof maxTokensRaw === 'number'
        ? maxTokensRaw
        : typeof maxTokensRaw === 'string'
            ? parseInt(maxTokensRaw, 10)
            : NaN;
    if (Number.isFinite(maxTokens) && maxTokens > 0) {
        payload.max_tokens = maxTokens;
    }
    return payload;
 }
@ -446,49 +521,108 @@ async function streamPerplexityToFile(
    apiKey: string,
    endpoint: string,
    payload: PerplexityPayload,
-    timeoutMs: number,
+    timeouts: { idleMs: number; ceilingMs: number },
    file: TFile,
    initialContent: string,
    isCancelled: () => boolean,
-): Promise<{ streamed: string; sources: PerplexitySource[]; images: PerplexityImage[] }> {
+): Promise<{ streamed: string; sources: PerplexitySource[]; images: PerplexityImage[]; truncated: boolean }> {
    payload.stream = true;
    const controller = new AbortController();
-    const timer = activeWindow.setTimeout(() => controller.abort(), timeoutMs);
+    // Optional absolute wall-clock ceiling — belt-and-suspenders backstop.
    // The per-chunk idle timer below is the primary safety mechanism; the
    // ceiling only fires if a stream sustains bytes past the ceiling AND
    // a hard cap is desired (e.g. settings/cft override > 0). Set ceilingMs
    // to Infinity to disable the ceiling entirely.
    const ceilingTimer = Number.isFinite(timeouts.ceilingMs) && timeouts.ceilingMs > 0
        ? activeWindow.setTimeout(() => controller.abort(), timeouts.ceilingMs)
        : null;
    // Use Node.js https to bypass CORS from the Obsidian renderer origin
    // (app://obsidian.md). activeWindow.fetch is blocked by Perplexity's
    // CORS policy; Node.js routes through the OS network stack instead.
    let response: Response;
    try {
-        response = await activeWindow.fetch(endpoint, {
+        const endpointUrl = new URL(endpoint);
-            method: 'POST',
+        const nodeStream = await new Promise<IncomingMessage>((resolve, reject) => {
-            headers: {
+            const req = https.request(
-                'Authorization': `Bearer ${apiKey}`,
+                {
-                'Content-Type': 'application/json',
+                    hostname: endpointUrl.hostname,
-                'Accept': 'text/event-stream',
+                    path: endpointUrl.pathname + endpointUrl.search,
-            },
+                    method: 'POST',
-            body: JSON.stringify(payload),
+                    headers: {
-            signal: controller.signal,
+                        'Authorization': `Bearer ${apiKey}`,
-            cache: 'no-store',
+                        'Content-Type': 'application/json',
                        'Accept': 'text/event-stream',
                    },
                    signal: controller.signal,
                },
                (res) => {
                    if (res.statusCode !== undefined && res.statusCode >= 400) {
                        let body = '';
                        res.on('data', (chunk: Buffer) => { body += chunk.toString(); });
                        res.on('end', () => {
                            reject(new Error(`Perplexity HTTP ${String(res.statusCode)} — ${body}`));
                        });
                        return;
                    }
                    resolve(res);
                }
            );
            req.on('error', reject);
            req.write(JSON.stringify(payload));
            req.end();
        });
        const webStream = new ReadableStream<Uint8Array>({
            start(ctrl) {
                nodeStream.on('data', (chunk: Buffer) => { ctrl.enqueue(chunk); });
                nodeStream.on('end', () => { ctrl.close(); });
                nodeStream.on('error', (err: Error) => { ctrl.error(err); });
            },
        });
        response = { ok: true, body: webStream } as unknown as Response;
    } catch (err) {
-        activeWindow.clearTimeout(timer);
+        if (ceilingTimer !== null) activeWindow.clearTimeout(ceilingTimer);
        throw err;
    }
    if (!response.ok) {
-        activeWindow.clearTimeout(timer);
+        if (ceilingTimer !== null) activeWindow.clearTimeout(ceilingTimer);
        throw new Error(`Perplexity HTTP ${response.status.toString()}`);
    }
    const reader = response.body?.getReader();
    if (!reader) {
-        activeWindow.clearTimeout(timer);
+        if (ceilingTimer !== null) activeWindow.clearTimeout(ceilingTimer);
        throw new Error('Perplexity returned no response body');
    }
    // Race each reader.read() against a fresh per-chunk idle timer. As long
    // as bytes keep arriving the timer is cleared and re-armed, so a healthy
    // slow stream completes naturally — only a stream that goes *quiet* for
    // `idleMs` (Perplexity stall, rate-limit, socket close) is killed. This
    // is the structural fix for [[Wall-Clock-Timeout-Cuts-Off-Long-Deep-
    // Research-Streams]] — ported from the legacy PerplexityModal flow in
    // perplexityService.ts:659-668, where the same pattern has been load-
    // bearing for sonar-deep-research runs for two iterations.
    const idleMs = timeouts.idleMs;
    const readWithIdleTimeout = (): Promise<ReadableStreamReadResult<Uint8Array>> => {
        let timer: number | undefined;
        const timeout = new Promise<never>((_, reject) => {
            timer = activeWindow.setTimeout(() => {
                reject(new Error(`stream went idle for ${(idleMs / 1000).toString()}s (likely API stall, rate limit, or socket close)`));
            }, idleMs);
        });
        return Promise.race([reader.read(), timeout]).finally(() => {
            if (timer !== undefined) activeWindow.clearTimeout(timer);
        });
    };
    const decoder = new TextDecoder();
    let sseBuffer = '';
    let streamed = '';
    let sources: PerplexitySource[] = [];
    let images: PerplexityImage[] = [];
    let truncated = false;
    let lastFlush = 0;
    const FLUSH_MS = 500;
@ -498,7 +632,21 @@ async function streamPerplexityToFile(
                controller.abort();
                break;
            }
-            const { value, done } = await reader.read();
+            let value: Uint8Array | undefined;
            let done = false;
            try {
                ({ value, done } = await readWithIdleTimeout());
            } catch {
                // Idle timer fired, AbortController fired (ceiling or user
                // cancel), or a dropped socket. Don't throw away what
                // already arrived — sonar-deep-research delivers the whole
                // document plus its citations in the FIRST SSE event, so a
                // later disconnect should still leave a usable, cited file.
                // Mark the run truncated and fall through to the final
                // flush + return.
                truncated = true;
                break;
            }
            if (done) break;
            sseBuffer += decoder.decode(value, { stream: true });
@ -516,10 +664,23 @@ async function streamPerplexityToFile(
                    const choices = obj['choices'];
                    if (Array.isArray(choices) && choices.length > 0) {
                        const first = choices[0] as Record<string, unknown> | undefined;
                        const message = first?.['message'] as Record<string, unknown> | undefined;
                        const delta = first?.['delta'] as Record<string, unknown> | undefined;
-                        const content = delta?.['content'];
+                        const snapshot = message?.['content'];
-                        if (typeof content === 'string') {
+                        const fragment = delta?.['content'];
-                            streamed += content;
+                        // Perplexity streams a *cumulative* message.content
                        // snapshot on every SSE event. sonar-deep-research does
                        // all its research server-side and dumps the ENTIRE
                        // document into the first event's message.content, while
                        // delta.content only ever carries a short tail fragment
                        // — so reading delta alone silently loses ~99% of the
                        // response. Prefer the snapshot; fall back to delta.
                        if (typeof snapshot === 'string'
                            && snapshot.length > streamed.length
                            && snapshot.startsWith(streamed)) {
                            streamed = snapshot;
                        } else if (typeof fragment === 'string') {
                            streamed += fragment;
                        }
                    }
                    const sr = obj['search_results'];
@ -544,7 +705,7 @@ async function streamPerplexityToFile(
            }
        }
    } finally {
-        activeWindow.clearTimeout(timer);
+        if (ceilingTimer !== null) activeWindow.clearTimeout(ceilingTimer);
        try {
            reader.releaseLock();
        } catch {
@ -555,7 +716,7 @@ async function streamPerplexityToFile(
    // Final flush of raw stream content before post-processing
    await app.vault.modify(file, initialContent + streamed);
-    return { streamed, sources, images };
+    return { streamed, sources, images, truncated };
 }
 export type ApplyOutcome =
@ -566,6 +727,8 @@ export type ApplyOutcome =
 export interface ApplyOptions {
    quiet?: boolean;
    isCancelled?: () => boolean;
    /** Per-run Perplexity model; overrides the template's cft `model:`. */
    modelOverride?: string;
 }
 export async function applyTemplate(
@ -667,22 +830,66 @@ export async function applyTemplate(
        ? `${fmBlock}\n`
        : `${fmBlock}\n${existingBody}\n\n`;
-    const payload = buildPayload(template, systemPrompt, userPrompt);
+    // Effective model: an explicit per-run override (from the run modal) wins
    // over the template's cft `model:`. Used for the payload, the loading
    // notice, and the cf_last_run_model frontmatter stamp.
    const cftModel = typeof template.cftConfig['model'] === 'string'
        ? template.cftConfig['model']
        : '';
    const effectiveModel = (options.modelOverride && options.modelOverride.length > 0)
        ? options.modelOverride
        : (cftModel || 'sonar-pro');
    // Entity-specific domain hints live in the target file's frontmatter so a
    // collision-heavy name (e.g. NATS) can be pinned per-file without editing
    // the shared template.
    const targetDomains = parseDomainList(fm['cf_search_domains']);
    const payload = buildPayload(template, systemPrompt, userPrompt, targetDomains, effectiveModel);
    let loadingNotice: Notice | null = null;
    if (!quiet) {
-        loadingNotice = new Notice('Streaming perplexity deep research…', 0);
+        loadingNotice = new Notice(`Streaming Perplexity · ${effectiveModel}…`, 0);
    }
    try {
        // Set initial state before streaming begins.
        await app.vault.modify(target, initialContent);
-        const { streamed, sources, images } = await streamPerplexityToFile(
+        // Two complementary timeout knobs:
        //   - `stream-idle-timeout-ms:` (cft) — per-chunk idle timer; primary
        //     safety. Defaults: 270s deep-research, 90s normal — matches the
        //     legacy modal flow in perplexityService.ts:659.
        //   - `request-timeout-ms:` (cft, legacy key retained) — absolute
        //     wall-clock ceiling; opt-in backstop. Falls back to the
        //     plugin-level `settings.requestTimeoutMs`. Set to 0 (in cft or
        //     settings) to disable the ceiling and rely entirely on idle.
        // Both accept number or numeric-string; non-positive / non-numeric
        // falls through to the fallback chain silently.
        const parseMs = (raw: unknown): number => {
            if (typeof raw === 'number') return raw;
            if (typeof raw === 'string') return parseInt(raw, 10);
            return NaN;
        };
        const isDeepResearch = /deep-research/i.test(effectiveModel);
        const idleDefaultMs = isDeepResearch ? 270_000 : 90_000;
        const cftIdleMs = parseMs(template.cftConfig['stream-idle-timeout-ms']);
        const effectiveIdleMs = Number.isFinite(cftIdleMs) && cftIdleMs > 0
            ? cftIdleMs
            : idleDefaultMs;
        const cftCeilingMs = parseMs(template.cftConfig['request-timeout-ms']);
        const settingsCeilingMs = settings.requestTimeoutMs;
        // Negative or non-numeric → fall through to settings; explicit 0 in
        // cft → disable ceiling entirely (Infinity, idle-only).
        const effectiveCeilingMs = Number.isFinite(cftCeilingMs)
            ? (cftCeilingMs > 0 ? cftCeilingMs : Infinity)
            : (settingsCeilingMs > 0 ? settingsCeilingMs : Infinity);
        const { streamed, sources, images, truncated } = await streamPerplexityToFile(
            app,
            settings.perplexityApiKey,
            settings.perplexityEndpoint,
            payload,
-            settings.requestTimeoutMs,
+            { idleMs: effectiveIdleMs, ceilingMs: effectiveCeilingMs },
            target,
            initialContent,
            isCancelled,
@ -692,9 +899,7 @@ export async function applyTemplate(
        const provider = typeof template.cftConfig['provider'] === 'string'
            ? template.cftConfig['provider']
            : 'unknown';
-        const modelName = typeof template.cftConfig['model'] === 'string'
+        const modelName = effectiveModel;
            ? template.cftConfig['model']
            : 'unknown';
        const providerLabel = provider.length > 0
            ? provider.charAt(0).toUpperCase() + provider.slice(1)
            : provider;
@ -758,6 +963,11 @@ export async function applyTemplate(
        });
        if (!quiet) {
            if (truncated) {
                new Notice(
                    `"${target.basename}": stream cut off (timeout or lost connection) — saved partial content with ${sources.length.toString()} sources. Re-run to complete.`,
                );
            }
            const verb = mode === 'fill' ? 'Filled' : 'Appended to';
            new Notice(`${verb} "${target.basename}" using ${template.file.basename} (${sources.length.toString()} sources)`);
        }
--- a/src/services/perplexityService.ts
+++ b/src/services/perplexityService.ts
@ -1,3 +1,5 @@
 import * as https from 'node:https';
 import type { IncomingMessage } from 'node:http';
 import type { Editor} from 'obsidian';
 import { Notice, request } from 'obsidian';
 import type { PromptsService } from './promptsService';
@ -519,31 +521,59 @@ export class PerplexityService {
            try {
                if (useStreaming) {
-                    console.debug('🔄 Making streaming API request...');
+                    console.debug('🔄 Making streaming API request via Node.js https (CORS bypass)...');
-                    // Streaming uses activeWindow.fetch because Obsidian's
+                    // activeWindow.fetch is blocked by CORS from the Obsidian renderer
-                    // requestUrl does not support reading SSE / chunked
+                    // origin (app://obsidian.md). Node.js https routes through the OS
-                    // response bodies — it buffers the whole response.
+                    // network stack and is never subject to Chromium CORS enforcement.
-                    const response = await activeWindow.fetch(this.settings.perplexityEndpoint, {
+                    const endpointUrl = new URL(this.settings.perplexityEndpoint);
-                        method: 'POST',
+                    const nodeStream = await new Promise<IncomingMessage>((resolve, reject) => {
-                        headers: {
+                        const req = https.request(
-                            'Authorization': `Bearer ${this.settings.perplexityApiKey}`,
+                            {
-                            'Content-Type': 'application/json',
+                                hostname: endpointUrl.hostname,
-                            'Accept': 'text/event-stream'
+                                path: endpointUrl.pathname + endpointUrl.search,
-                        },
+                                method: 'POST',
-                        body: JSON.stringify(payload),
+                                headers: {
-                        cache: 'no-store'
+                                    'Authorization': `Bearer ${this.settings.perplexityApiKey}`,
                                    'Content-Type': 'application/json',
                                    'Accept': 'text/event-stream',
                                },
                            },
                            (res) => {
                                if (res.statusCode !== undefined && res.statusCode >= 400) {
                                    let body = '';
                                    res.on('data', (chunk: Buffer) => { body += chunk.toString(); });
                                    res.on('end', () => {
                                        reject(new Error(`HTTP error! status: ${res.statusCode} — ${body}`));
                                    });
                                    return;
                                }
                                resolve(res);
                            }
                        );
                        req.on('error', reject);
                        req.write(JSON.stringify(payload));
                        req.end();
                    });
-                    if (!response.ok) {
+                    // Wrap Node.js IncomingMessage in a Web ReadableStream so the
-                        throw new Error(`HTTP error! status: ${response.status}`);
+                    // existing handleStreamingResponse method consumes it unchanged.
-                    }
+                    const webStream = new ReadableStream<Uint8Array>({
                        start(controller) {
                            nodeStream.on('data', (chunk: Buffer) => {
                                controller.enqueue(chunk);
                            });
                            nodeStream.on('end', () => {
                                controller.close();
                            });
                            nodeStream.on('error', (err: Error) => {
                                controller.error(err);
                            });
                        },
                    });
-                    if (!response.body) {
+                    const response = { ok: true, body: webStream } as unknown as Response;
-                        throw new Error('No response body');
+                    console.debug('✅ Streaming response via Node.js ready, handling SSE...');
-                    }
+                    await this.handleStreamingResponse(response, editor, responseCursor, requestId, headerText, isDeepResearch);
                    console.debug('✅ Streaming response received, starting to handle...');
                    await this.handleStreamingResponse(response, editor, responseCursor, requestId, headerText);
                } else {
                    console.debug('🔄 Making non-streaming API request...');
                    // Use Obsidian's request method for non-streaming with cache busting
@ -627,16 +657,17 @@ export class PerplexityService {
    }
    private async handleStreamingResponse(
-        response: Response, 
+        response: Response,
-        editor: Editor, 
+        editor: Editor,
        responseCursor: { line: number; ch: number },
        requestId?: number,
-        headerText?: string
+        headerText?: string,
        isDeepResearch = false
    ): Promise<void> {
        console.debug('🔄 handleStreamingResponse called');
        const reader = response.body?.getReader();
        if (!reader) throw new Error('No response body');
-        
+
        console.debug(`🔄 Starting streaming response handler [${requestId || 'unknown'}]`);
        // Hoist decoder out of the loop so the {stream: true} flag carries
@ -645,12 +676,17 @@ export class PerplexityService {
        let buffer = '';
        const currentPos = { ...responseCursor };
        let finalResponseData: PerplexityStreamChunk | null = null;
        // Everything already written into the editor. Perplexity streams a
        // cumulative message.content snapshot, so we diff against this.
        let accumulated = '';
        // Race each read against an idle timer — if no chunk arrives within
        // STREAM_IDLE_TIMEOUT_MS, throw so the catch surfaces a visible
        // "Streaming Error" instead of leaving reader.read() blocked forever
-        // on a dropped/idle SSE socket.
+        // on a dropped/idle SSE socket. sonar-deep-research does all its
-        const STREAM_IDLE_TIMEOUT_MS = 90_000;
+        // research server-side and can hold the socket silent for minutes
        // before the first byte, so it gets a much longer leash.
        const STREAM_IDLE_TIMEOUT_MS = isDeepResearch ? 270_000 : 90_000;
        const readWithIdleTimeout = (): Promise<ReadableStreamReadResult<Uint8Array>> => {
            let timer: number | undefined;
            const timeout = new Promise<never>((_, reject) => {
@ -693,28 +729,41 @@ export class PerplexityService {
                                };
                            }
-                            if (parsed.choices?.[0]?.delta?.content) {
+                            // Perplexity streams a *cumulative* message.content
-                                const content = parsed.choices[0].delta.content;
+                            // snapshot on every SSE event. sonar-deep-research does
-                                if (content) {
+                            // all its research server-side and dumps the ENTIRE
-                                    // console.debug('🎉 First content received! Clearing loading text...');
+                            // document into the first event's message.content, while
-                                    // Clear any loading text before inserting content
+                            // delta.content only ever carries a short tail fragment
-                                    this.clearLoadingText(editor);
+                            // — so reading delta alone silently drops ~99% of a deep
-                                    
+                            // research article. Treat message.content as the source
-                                    // Clear the animation interval if it exists
+                            // of truth and write only the not-yet-written tail; fall
-                                    this.clearLoadingAnimation();
+                            // back to accumulating delta.content if message is absent.
-                                    
+                            const choice = parsed.choices?.[0];
-                                    // console.debug(`📝 Inserting content at position:`, currentPos, `Content:`, content.substring(0, 50) + '...');
+                            let fullSoFar: string | undefined;
-                                    editor.replaceRange(content, currentPos);
+                            if (typeof choice?.message?.content === 'string') {
-                                    const contentLines = content.split('\n');
+                                fullSoFar = choice.message.content;
-                                    if (contentLines.length === 1) {
+                            } else if (typeof choice?.delta?.content === 'string') {
-                                        currentPos.ch += content.length;
+                                fullSoFar = accumulated + choice.delta.content;
-                                    } else {
+                            }
-                                        currentPos.line += contentLines.length - 1;
+                            // startsWith guard: Perplexity is append-only, but if a
-                                        currentPos.ch = contentLines[contentLines.length - 1]?.length ?? 0;
+                            // prefix ever changes we skip rather than corrupt the doc.
-                                    }
+                            if (fullSoFar && fullSoFar.length > accumulated.length && fullSoFar.startsWith(accumulated)) {
-                                    // Scroll to follow the new content
+                                const newText = fullSoFar.slice(accumulated.length);
-                                    editor.scrollIntoView({ from: currentPos, to: currentPos }, true);
+                                // Clear any loading text / animation before first insert
                                this.clearLoadingText(editor);
                                this.clearLoadingAnimation();
                                editor.replaceRange(newText, currentPos);
                                const contentLines = newText.split('\n');
                                if (contentLines.length === 1) {
                                    currentPos.ch += newText.length;
                                } else {
                                    currentPos.line += contentLines.length - 1;
                                    currentPos.ch = contentLines[contentLines.length - 1]?.length ?? 0;
                                }
                                // Scroll to follow the new content
                                editor.scrollIntoView({ from: currentPos, to: currentPos }, true);
                                accumulated = fullSoFar;
                            }
                        }
                    } catch (e) {
@ -776,6 +825,18 @@ export class PerplexityService {
            this.clearLoadingText(editor);
            this.clearLoadingAnimation();
            // Salvage citations/images already received before the drop —
            // deep research delivers them in its first SSE event, so a later
            // disconnect shouldn't discard a footer we already have the data
            // for. processStreamingMetadata appends at the document tail.
            if (finalResponseData) {
                try {
                    this.processStreamingMetadata(finalResponseData, editor, headerText);
                } catch (metaErr) {
                    console.warn('Could not salvage streaming metadata:', metaErr);
                }
            }
            editor.replaceRange(`\n**Streaming Error:** ${userMsg}\n\n`, currentPos);
        }
    }
--- a/src/services/templateSeederService.ts
+++ b/src/services/templateSeederService.ts
@ -6,9 +6,13 @@ import conceptProfile from '../docs/templates/concept-profile.md';
 import vocabularyProfile from '../docs/templates/vocabulary-profile.md';
 import sourceProfile from '../docs/templates/source-profile.md';
 import toolkitProfile from '../docs/templates/toolkit-profile.md';
 import marketMapProfile from '../docs/templates/market-map-profile.md';
 import standardsAndSpecsProfile from '../docs/templates/standards-and-specs-profile.md';
 import marketCategoryProfile from '../docs/templates/market-category-profile.md';
 import partialsReadme from '../docs/partials/README.md';
 import mermaidDisciplinePartial from '../docs/partials/mermaid-discipline.md';
 import latexDisciplinePartial from '../docs/partials/latex-discipline.md';
 import preamblesReadme from '../docs/preambles/README.md';
 import inlineCitationPreamble from '../docs/preambles/inline-citation.md';
@ -30,11 +34,15 @@ const TEMPLATE_FILES: SeedFile[] = [
    { name: 'vocabulary-profile.md', content: vocabularyProfile },
    { name: 'source-profile.md', content: sourceProfile },
    { name: 'toolkit-profile.md', content: toolkitProfile },
    { name: 'market-map-profile.md', content: marketMapProfile },
    { name: 'standards-and-specs-profile.md', content: standardsAndSpecsProfile },
    { name: 'market-category-profile.md', content: marketCategoryProfile },
 ];
 const PARTIALS_README: SeedFile = { name: 'README.md', content: partialsReadme };
 const PARTIAL_FILES: SeedFile[] = [
    { name: 'mermaid-discipline.md', content: mermaidDisciplinePartial },
    { name: 'latex-discipline.md', content: latexDisciplinePartial },
 ];
 const PREAMBLES_README: SeedFile = { name: 'README.md', content: preamblesReadme };
--- a/versions.json
+++ b/versions.json
@ -2,5 +2,8 @@
 	"0.1.0": "0.15.0",
 	"0.1.1": "1.8.10",
 	"0.1.2": "1.8.10",
-	"0.2.0": "1.8.10"
+	"0.2.0": "1.8.10",
 	"0.2.1": "1.8.10",
 	"0.3.0": "1.8.10",
 	"0.3.1": "1.8.10"
 }
Author	SHA1	Message	Date
mpstaton	01897c0903	fix(perplexityService, directoryTemplateService): CORS bypass — Node.js https replaces activeWindow.fetch for all Perplexity streaming paths Perplexity's API stopped including Access-Control-Allow-Origin headers for app://obsidian.md, so Electron's Chromium renderer began blocking every streaming response body — even on successful 200 OK requests. Every Perplexity query that used streaming was silently producing nothing. Both streaming call sites now use https.request from node:https instead of activeWindow.fetch. Node.js routes through the OS network stack and has no CORS constraint at all. The resulting IncomingMessage stream is wrapped in a Web ReadableStream<Uint8Array> and passed to the existing SSE parsing machinery unchanged — idle-timeout, chunk accumulation, citations, images, and AbortController signal handling all preserved. Non-streaming requests (Obsidian's request() function) were never affected and are untouched. perplexityService.ts: - Added import * as https from 'node:https' + import type { IncomingMessage } - Replaced activeWindow.fetch in queryPerplexity streaming branch with https.request → IncomingMessage → ReadableStream<Uint8Array> bridge directoryTemplateService.ts: - Same imports - Replaced activeWindow.fetch in streamPerplexityToFile with same bridge - AbortController.signal threaded into https.request signal option so ceiling-timer and user-cancel abort behavior is fully preserved Bumps to 0.3.1 — manifest.json, package.json, versions.json updated. Files changed: - src/services/perplexityService.ts - src/services/directoryTemplateService.ts - manifest.json - package.json - versions.json - changelog/2026-07-06_01.md - changelog/releases/0.3.1.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-06 14:41:12 -05:00
mpstaton	f946973894	feat(release): 0.3.0 — three analyst-grade templates for VC/PE/equities + four infrastructure fixes for deep-research reliability 0.3.0 turns Perplexed into a serious analyst tooling layer for Venture Capital, Private Equity, and equities-trading workflows. Three new directory templates produce 6-9K-word cited analyst drafts in a single Perplexity Deep Research run. Underneath them, four infrastructure fixes turn deep-research from a brittle "might work" workflow into something you can put real analyst time behind. The three new templates: market-map-profile (analyst memo for Known Category or Thesis-Driven maps), standards-and-specs-profile (open-spec profiles with five-way authority typing and three-tier adoption framing), market-category-profile (concept-folder reference card with three financial-stage tiers — Incumbents / Challengers / Innovators). The four infrastructure fixes: per-template request-timeout-ms wall-clock override (yesterday, pressure-relief valve), structural port of per-chunk idle-timeout discipline from the legacy modal flow into the directory-template flow (yesterday, structural fix), per-template max-tokens Perplexity output-budget override (today, fixes silent clean-truncation when finish_reason=length fires mid-template), and a strengthened mermaid-discipline partial plus new latex-discipline partial (today, fixes the 4-of-5 broken-Mermaid rate via paired BAD/GOOD examples and a six-item pre-emit self-check). New templates (src/docs/templates/): - market-map-profile.md — sonar-deep-research, request-timeout-ms 2400000 (40-min ceiling), max-tokens 24000. Anti-incumbent editorial stance. - standards-and-specs-profile.md — sonar-deep-research, request-timeout-ms 0 (idle-only safety), max-tokens 24000. Five-way authority typing, named editors, stewardship transitions, named critics. - market-category-profile.md — sonar-deep-research, request-timeout-ms 0, max-tokens 24000. Three-tier financial-stage framing, separate Why Now and What's Happening sections, Industry Coverage sub-grouped into Market Reports / Industry Articles / Financial News. Rendering-discipline partials (src/docs/partials/): - mermaid-discipline.md rewritten — paired BAD/GOOD examples throughout, the & escape rule, explicit \n vs <br/> ban, six-item self-check before emit, simplify-rather-than-break ethos. Original ~250 tokens expanded to ~440. - latex-discipline.md new — Obsidian MathJax delimiters and \$ escaping in prose to avoid accidental inline-math spans. - concept-profile.md wired in {{include: latex-discipline}} alongside the existing {{include: mermaid-discipline}}, placed immediately after the "render a mermaid codefence here" instruction so the rules are in scope at the moment of generation. Streaming, timeout, and output-budget infrastructure (src/services/ directoryTemplateService.ts): - streamPerplexityToFile now races each reader.read() against a fresh per-chunk idle timer (270s deep-research, 90s normal — matches the legacy modal flow at perplexityService.ts:659). Closes the "Wall-clock-timeout cuts off long deep-research streams" issue. - request-timeout-ms cft override is now the optional absolute wall-clock ceiling (belt-and-suspenders backstop on top of the idle timer); explicit 0 disables. - max-tokens cft override added to buildPayload — passes through to Perplexity's max_tokens parameter; lifts the silent ~8K-token default ceiling that was clean-truncating thorough drafts via finish_reason: length. Documentation (README.md, docs/directory-templates.md): - README — new top-level section "For Venture Capital, Private Equity, and Equities-Trading Workflows" with a five-row table mapping each analyst workflow to the template that produces it. Directory Templates section updated to seven shipped templates plus the new per-template cft-block knobs. - docs/directory-templates.md — new "Per-template max-tokens override" section with the diagnostic table for distinguishing wall-clock-timeout truncation (mid-sentence cutoff, no sources footer) from max_tokens truncation (clean section-end, full sources footer). The diagnostic vocabulary that did not exist before this release. Seeding (src/services/templateSeederService.ts): - Registered all three new templates plus the new latex-discipline partial so they auto-seed into vaults on first plugin load and appear in Re-seed runs. Version bump (manifest.json, package.json, versions.json): - 0.2.1 to 0.3.0. minAppVersion unchanged at 1.8.10. Engineering changelog entries (changelog/): - 2026-05-26_02.md — the structural idle-timeout port. - 2026-05-27_01.md — the two new templates, max-tokens override, and rendering-discipline partial work. Includes the diagnostic-led narrative on the vault-drift bug that caused the 4-of-5 broken-Mermaid rate. Release narratives: - changelog/releases/0.3.0.md — full marketing-quality release narrative with frontmatter, four-audience cascade, three diagnostic-led stories (streaming-timeout, max_tokens, rendering-discipline), explicit upgrade notes including the vault-drift warning, three deferred items for 0.4.x. - release-notes/0.3.0.md — prose-only release-page body picked up by the release.yml workflow when the 0.3.0 tag is pushed. Tighter cut of the same narrative, suited for the GitHub release page. Context-v stamp: - context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md marked status: resolved, date_resolved 2026-05-26, with the four open items reorganized into "what landed" (3 of 4) and "deferred follow-ups" (cross-service audit, settings-pane exposure, deep-research detection robustness). Files changed: - manifest.json, package.json, versions.json - README.md, docs/directory-templates.md - src/services/directoryTemplateService.ts - src/services/templateSeederService.ts - src/docs/templates/README.md - src/docs/templates/concept-profile.md - src/docs/templates/market-map-profile.md - src/docs/templates/market-category-profile.md (new) - src/docs/templates/standards-and-specs-profile.md (new) - src/docs/partials/mermaid-discipline.md - src/docs/partials/latex-discipline.md (new) - changelog/2026-05-26_02.md (new) - changelog/2026-05-27_01.md (new) - changelog/releases/0.3.0.md (new) - release-notes/0.3.0.md (new) - context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-27 17:41:13 -05:00
mpstaton	96df70595f	feat(directory-templates): market-map analyst profile + per-template request-timeout-ms cft override A new shipped template aimed at analyst-grade market maps (Known Category and Thesis-Driven flavors), and the timeout refactor that came out of running it against a real 7-8K-word draft and watching the wall-clock AbortController truncate the tail mid-sentence. The two ship together because the template surfaced the bug — running market-map-profile on lost-in-public/market-maps/Humanoid Robots and their Input Industries.md was the first observable instance of the wall-clock pathology — and they're both groundwork for the multi-stage Claude+Perplexity+RAG pipeline the exploration doc now sketches. Impact: a fifth shipped template auto-seeded into Content-Dev/Templates/ on first plugin load (and re-seedable from settings), running on sonar-deep-research with return-images off by design; templates can override the plugin-level wall-clock timeout per-template via a new cft-block key; the plugin-level default jumped from 10 min to 30 min because the old default was tuned for short concept-profile runs and was cutting off any deep-research template that took its time. Existing four templates are unchanged in behavior — they don't declare the override and inherit the new 30-min default, which they all finish comfortably under. Market-map template (src/docs/templates/market-map-profile.md): - Heading skeleton: Market Snapshot, The Question this Map Answers, Why Now, Map of the Market, Lighthouse Examples, Innovator Profiles (Link/Offering/Funding/Why/Coverage cards + per-sub-segment summary table), Media-Voices-Coverage, Market Dynamics (Sizing, Adoption, Capital Flow), Frontier and Open Questions, Adjacent Concepts. - Dual-flavor: the system prompt has the model pick between Known Category (Humanoid Robots, Quantum Computing) and Thesis-Driven (e.g. Neural Network Hardware as Brains for Robotics) based on title and tags. - Anti-incumbent editorial stance copied from the concept template family — cap big tech at 1 of 5-10 in every sub-bucket, attribute innovation to startups/labs/founders. - Declares request-timeout-ms: 2400000 (40 min) with an inline comment explaining the budget — the only shipped template that overrides the plugin default today. - Registered in templateSeederService.ts so first-load seeding and the Re-seed button both pick it up; surfaced in src/docs/templates/README.md and docs/directory-templates.md. Timeout refactor (src/services/directoryTemplateService.ts, main.ts): - Per-template cft-block key request-timeout-ms: accepts number or numeric string, silently falls back on non-positive / non-numeric values, otherwise wins over the plugin-level setting for that template's runs. Resolution code sits just before the streamPerplexityToFile call so the rest of the streaming primitive is untouched. - Plugin default directoryTemplatesRequestTimeoutMs bumped 600000 → 1800000 (10 min → 30 min). Settings-pane description rewritten to call out the override and the cost framing ("$10-$50 of analyst time per good output is worth waiting for"). - Structural fix (idle-timeout discipline ported from perplexityService.ts:659-668, where the legacy modal flow already does it correctly) is filed as its own open issue, not in scope for this commit. Context-v writing: - context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md — three architectures weighed (zoned single-file appends, multi-doc folder per market map, per-section cft-section blocks), an include-sources YAML schema proposed covering vault paths/globs and Chroma queries, and the "Perplexity and Claude do not overwrite each other until editing" heuristic translated into three implementation strictness levels. Recommends Option A (zoned appends) for v1. - context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md — the Humanoid Robots truncation as motivating symptom, side-by-side diagnosis of the two timeout disciplines in the codebase (wall-clock here, per-chunk idle in the legacy modal), the partial fix shipped today, why it's only partial, the structural-fix proposal, and four pre-spec open items. Cross-linked both ways to the exploration so the issue is reachable from either entry point. Files changed: - src/docs/templates/market-map-profile.md (new) - src/services/templateSeederService.ts - src/services/directoryTemplateService.ts - main.ts - src/docs/templates/README.md - docs/directory-templates.md - context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md (new) - context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md (new) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 17:08:43 -05:00
mpstaton	279c954143	chore(release): stamp 0.2.1 release narrative with commit hash `3ae4012` Records the 0.2.1 release commit hash in the release_commit frontmatter field of changelog/releases/0.2.1.md, mirroring the stamp pass done for 0.2.0. The narrative now points at the exact commit the release was cut from. Files changed: - changelog/releases/0.2.1.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-22 15:29:10 -05:00
mpstaton	3ae4012f1f	feat(release): 0.2.1 — deep research that actually keeps the research Directory-template runs with sonar-deep-research were silently discarding ~99% of every response. Perplexity's deep-research model delivers its whole finished document in the first SSE event's message.content; the streaming reader took delta.content (correct for the token-streaming sonar models) and captured only a ~75-character tail fragment of a ~10,000-character report. toolkit-profile shipped with sonar-deep-research as its default model, so the default was the broken one. This release fixes the streaming, hardens it against mid-stream disconnects, and adds the tooling the directory-template research workflow was missing. Both streaming paths now read Perplexity's cumulative message.content snapshot and append only the new tail — correct for token-streaming sonar models and for deep research's first-event document dump alike. A timed-out or dropped stream now keeps the partial content and the citations it already received instead of discarding the whole run. "Apply directory template" gained a run dialog with a per-run model selector, so the model is no longer locked to the template file, and the loading notice names the model that is actually running instead of a hardcoded "deep research" string. Streaming (perplexityService.ts, directoryTemplateService.ts): - Read message.content (cumulative snapshot), diff against written text, append the tail; a startsWith guard skips a changed prefix rather than corrupting the note. - streamPerplexityToFile returns a partial result with a `truncated` flag on a read error instead of throwing; applyTemplate writes the partial with its sources footer and a truncation notice. The modal catch block salvages its citation metadata the same way. - Deep-research idle timeout 90s -> 270s; request-timeout default 5 -> 10 min. Model selection (DirectoryTemplateRunModal.ts, main.ts): - New run dialog: a template dropdown plus a Perplexity model dropdown that defaults to the template's cft model and overrides it for that run only. - cf_last_run_model now stamps the model that actually ran. Search scoping (directoryTemplateService.ts): - search_domain_filter support via cf_search_domains (target frontmatter) and search-domains (cft block); allow/deny entries, capped at 10; job boards are denied automatically. Template (src/docs/templates/toolkit-profile.md): - System prompt rebuilt: anchors the entity on its real name rather than the scraped marketing tagline, makes source quality a triage task instead of an allow/deny list, adds a search-don't-recall rule and per-section length ceilings. Default model -> sonar-pro. Files changed: - src/services/perplexityService.ts - src/services/directoryTemplateService.ts - src/modals/DirectoryTemplateRunModal.ts - main.ts - src/docs/templates/toolkit-profile.md - manifest.json - package.json - versions.json - changelog/releases/0.2.1.md - release-notes/0.2.1.md Also included: - manifest.json description aligned with package.json ("Perplexica (now Vane)"). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-22 15:28:53 -05:00