diff --git a/changelog/2026-05-26_01.md b/changelog/2026-05-26_01.md new file mode 100644 index 0000000..9a36bef --- /dev/null +++ b/changelog/2026-05-26_01.md @@ -0,0 +1,202 @@ +--- +title: "Market-map analyst template + the per-template timeout override the long deep-research it needs surfaced as load-bearing" +lede: "A fifth shipped directory template aimed at analyst-grade market maps (Known Category and Thesis-Driven flavors) — the kind of document a partner pays a $150K analyst three days to produce." +date_work_started: 2026-05-26 +date_work_completed: 2026-05-26 +date_created: 2026-05-26 +date_modified: 2026-05-26 +publish: true +category: Changelog +tags: + - Directory-Templates + - Market-Maps + - Deep-Research + - Streaming-Timeouts + - cft-Block + - Perplexity-sonar-deep-research + - Multi-Stage-Generation +authors: + - Michael Staton +augmented_with: + - Claude Code on Claude Opus 4.7 (1M context) +files_changed: + - src/docs/templates/market-map-profile.md + - src/services/templateSeederService.ts + - src/services/directoryTemplateService.ts + - src/docs/templates/README.md + - docs/directory-templates.md + - main.ts + - context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md + - context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md +--- + +# A template for analyst-grade market maps, and the timeout the work it produces actually needs + +## Why care? + +If you write market maps — the kind of document where you enumerate every credible operator in a category, segment them, name the funders, surface the open questions, and hand it to a partner — Perplexed now ships a directory template that produces a respectable analyst draft in one pass. Drop a file into `lost-in-public/market-maps/` (or any top-level `market-maps/`), give it a title and a one-line lede, run *Apply directory template to current file*, and you get back a 6-8K-word draft on `sonar-deep-research` with named innovators, per-segment funding tables, media coverage, market dynamics, and a frontier-questions section. Then you spend your time on the curation pass — promoting `[Name](url)` references to `[[wikilink]]`s back into your vault, layering in your `:::tool-showcase` blocks, tightening the analyst voice — instead of on the enumeration grind. The framing is straightforward: an analyst-day at $150K-a-year rates is $600 of someone's time, the API run costs a few dollars, and the rest is your judgment doing what your judgment does best. + +If you author *any* directory template that runs on a deep-research model, the timeout refactor is the more load-bearing change. The plugin's wall-clock cap before today was 10 minutes — perfectly reasonable for `concept-profile` and `vocabulary-profile`, fatal for anything that legitimately needs 20-30 minutes to produce its long analytical body. Today's market-map run on *Humanoid Robots and their Input Industries* was the first observable instance: ~7,500 words of body produced, then a hard stop mid-sentence in the *Frontier and Open Questions* section as the `AbortController` fired at the ceiling. The fix: per-template `request-timeout-ms:` cft-block override, plugin-level default bumped to 30 min, and an open issue filed for the structural fix (port the idle-timeout discipline the legacy modal flow already uses). + +## What's new? + +Three shipped behaviors: + +``` +zz-cf-lib/ +├── templates/ +│ ├── concept-profile.md +│ ├── vocabulary-profile.md +│ ├── source-profile.md +│ ├── toolkit-profile.md +│ └── market-map-profile.md (new — runs on sonar-deep-research) +├── partials/ (existing — referenced via {{include: name}}) +└── preambles/ (existing — auto-attached to every request) +``` + +- **`market-map-profile.md`** is auto-seeded into `Content-Dev/Templates/` on first plugin load and re-seedable from settings via the *Re-seed templates* button. It matches `lost-in-public/market-maps/**` and `market-maps/**` so both your `lost-in-public/`-organized vault and a top-level `market-maps/` folder work. +- **`request-timeout-ms:` cft-block override** lets a template declare its own wall-clock budget that wins over the plugin-level setting for that template's runs. `market-map-profile.md` declares `2400000` (40 min); the other four don't declare it and inherit the new default. +- **Plugin-level default bumped 600000 → 1800000** (10 min → 30 min). Settings-pane description rewritten to call out the override and the cost framing ("$10-$50 of analyst time per good output is worth waiting for"). + +Two context-v artifacts captured at the same time, because the market-map work raised both as the next-natural moves: + +- **`context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md`** — opens the explorations folder for this submodule. Weighs three architectures for evolving the lean v1 template into a cooperative pipeline (zoned single-file appends, multi-doc folder per market map, per-section `cft-section` blocks); proposes an `include-sources:` YAML schema covering vault paths/globs and Chroma queries; translates the "Perplexity and Claude do not overwrite each other until editing" framing into three implementation strictness levels. Recommends Option A (zoned appends) for v1. +- **`context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md`** — the Humanoid Robots truncation as motivating symptom, side-by-side diagnosis of the two timeout disciplines in this codebase (wall-clock here, per-chunk idle in the legacy modal), the partial fix shipped today, why it's only partial, and the structural-fix proposal. + +## How the market-map template thinks + +A market map is the draft a well-paid analyst hands to a partner. It is not an encyclopedia entry, not a marketing post, not a listicle. The template's system prompt commits to this framing explicitly and asks the model to determine the *flavor* of the map from the title and frontmatter: + +- **Known Category** — Humanoid Robots, Quantum Computing, Agentic AI in Fintech. The taxonomy is roughly settled; the work is enumerating innovators within established sub-segments and explaining the frontier. +- **Thesis-Driven** — Neural Network Hardware as Brains for Robotics, Blockchain and Web3 Institutional Invasions. A hypothesis traverses multiple known categories. The work is making the thesis legible, then enumerating innovators from each adjacent category the thesis pulls in. + +The skeleton works for both flavors. Sections (in order): + +1. **Market Snapshot** — italicized lede + headline-stat blockquote + orienting paragraph. +2. **The Question this Map Answers** — what this map clarifies for an operator or investor reading it. +3. **Why Now** — 3-5 cited unlocks (technical thresholds, regulatory shifts, capital patterns, customer-behavior shifts) that make the market mappable in the current quarter. +4. **Map of the Market — Sub-Segments** — 4-8 sub-segments with 1-2 sentence definitions. Doubles as the TOC for the next section. +5. **Lighthouse Examples** — 5-10 named operators per sub-segment in `[Name](url) — one-line` form, flat reference list. +6. **Innovator Profiles** — 3-7 deeper cards per sub-segment in a strict Link / Offering / Funding / Why-they-matter / Coverage shape, followed by a per-sub-segment summary table. +7. **Media, Voices, and Coverage** — publications, podcasts, YouTubers, analysts, operator-thinkers. +8. **Market Dynamics** — Sizing & Growth, Adoption Patterns & Barriers, Capital Flow. +9. **Frontier and Open Questions** — 4-7 specific questions phrased as questions. +10. **Adjacent Concepts and Maps** — plain-text concept names (no wikilinks invented; the curator wikilinks during curation). + +The anti-incumbent editorial stance from the concept-profile family carries over: big tech is treated as adopters/popularizers, not innovators, unless a research-lab paper or heyday-era origination supports otherwise. Cap big tech at 1 of 5-10 in every sub-bucket. The market-map prompt phrases this in terms specific to category mapping — "name the named operators driving the curve, not the incumbents profiting from it." + +Image-return is off by design for this template. `sonar-deep-research` returns better citation density and analytical length than `sonar-pro` but its image metadata is unreliable; market-map imagery (banner, portrait, square) is generated separately via Ideogram and lives in frontmatter, so the per-section image markers the other templates use aren't load-bearing here. + +## How the timeout override works + +The directory-template flow has always armed a single wall-clock `setTimeout` at fetch time, racing the entire stream against `settings.requestTimeoutMs`: + +```ts +const controller = new AbortController(); +const timer = activeWindow.setTimeout(() => controller.abort(), timeoutMs); +``` + +This worked fine when the entire suite of shipped templates ran on `sonar-pro` and finished in 2-3 minutes. It doesn't work when a single template legitimately needs 25-30 minutes. The naïve fix would have been to bump `settings.requestTimeoutMs` to something generous for everyone, but that punishes the short-template case (a 90-second concept run that stalls silently shouldn't take 30 minutes to surface). So we put the dial where the cft already lives — inside the template itself: + +```yaml +provider: perplexity +model: sonar-deep-research +return-citations: true +return-images: false +request-timeout-ms: 2400000 # 40 minutes for this template specifically +system: | + ... +``` + +Resolution sits in `directoryTemplateService.ts` just before the `streamPerplexityToFile` call: + +```ts +const cftTimeoutRaw = template.cftConfig['request-timeout-ms']; +const cftTimeoutMs = typeof cftTimeoutRaw === 'number' + ? cftTimeoutRaw + : typeof cftTimeoutRaw === 'string' + ? parseInt(cftTimeoutRaw, 10) + : NaN; +const effectiveTimeoutMs = Number.isFinite(cftTimeoutMs) && cftTimeoutMs > 0 + ? cftTimeoutMs + : settings.requestTimeoutMs; +``` + +Number or numeric-string both work. Non-positive / non-numeric values are silently ignored and the plugin-level default takes over — the template author can't accidentally lock themselves out of the run with a typo. + +Plugin-level default bumped because the old 10-minute ceiling was clearly tuned for the original four templates and not for any subsequent deep-research template anyone might write. 30 minutes is the conservative new default: comfortably above what any of the existing four templates need, generous enough that most new deep-research templates won't need to override, and visible-enough in the settings pane that a user who hits a longer-running case can find the dial. + +## The bug that surfaced this — Humanoid Robots, line 400 + +The first real-world run of `market-map-profile` was `lost-in-public/market-maps/Humanoid Robots and their Input Industries.md`. The model produced a draft that looked respectable through nine of the ten skeleton sections, then terminated mid-sentence inside the tenth: + +``` +Will the evolution of robot-as-a-service (R +``` + +That trailing `(R` is the last byte written to the file. The *Adjacent Concepts and Maps* heading never appeared. Frontmatter stamps (`cf_last_run`, `cf_last_run_model`) landed correctly — the run did make it to its `processFrontMatter` post-step — so what was lost was streamed body content that hadn't arrived when the `AbortController` fired. Classic wall-clock cut-off shape. + +The directory-template flow uses a single wall-clock timeout. The *legacy* modal flow (`perplexityService.ts:659-668`) has used a per-chunk idle-timeout for several iterations: + +```ts +const STREAM_IDLE_TIMEOUT_MS = isDeepResearch ? 270_000 : 90_000; +const readWithIdleTimeout = (): Promise<...> => { + let timer: number | undefined; + const timeout = new Promise((_, reject) => { + timer = window.setTimeout(() => { + reject(new Error(`stream went idle for ${...}s ...`)); + }, STREAM_IDLE_TIMEOUT_MS); + }); + return Promise.race([reader.read(), timeout]).finally(() => { + if (timer !== undefined) window.clearTimeout(timer); + }); +}; +``` + +The idle-timeout pattern is structurally correct. Per `reader.read()`, a fresh timeout is armed; as long as bytes keep arriving, the timer keeps getting cleared and re-armed. The stream is only killed if it goes *quiet* for N seconds. Slow-but-healthy completes; stalled-but-silent fails fast. + +We didn't port the idle-timeout pattern into the directory-template flow today, because the structural refactor warrants its own design and its own commit, not a side-quest in a template-shipping changelog. Today's fix is the pressure-relief valve: a bigger ceiling, plus a per-template dial. The structural fix is filed at `context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md` with a 4-step concrete plan, a key-rename proposal (`stream-idle-timeout-ms:`), and a backstop-or-no-backstop open question. + +## Under the hood — the things that earned their own design decisions + +**Both flavors with one template, not two.** The Known Category vs Thesis-Driven distinction is real, but it lives in the *content* of the map, not its *structure*. A Known Category map and a Thesis-Driven map both want sub-segments, both want innovator profiles, both want the same frontier-questions framing. The system prompt names the flavor difference once and asks the model to pick from `{{title}}` and `{{frontmatter}}` signals. Two templates would have produced more drift than value. + +**No invented wikilinks.** The model doesn't know what's in your vault. If it tries to emit `[[Tooling/AI-Toolkit/Agentic AI/Crew AI]]` from training data, it'll get the path wrong half the time and the basename wrong the rest of the time. So the prompt is explicit: use `[Name](url)` form for every entity, leave wikilink promotion to the curator. The future multi-stage pipeline (per the exploration doc) will inject canonical vault content via an `include-sources:` block at request time and authorize the model to emit `[[wikilink]]`s for the entries it's been shown — but that's v2. + +**`return-images: false` because deep-research's image metadata isn't reliable enough.** This is a documented limit on `sonar-deep-research` we've eaten before in the concept-profile changelog (2026-05-12). Market-map imagery doesn't lean on inline images anyway — `banner_image`, `portrait_image`, `square_image` in frontmatter is the canonical surface, generated separately via Ideogram. + +**The override key is named `request-timeout-ms:`, not `stream-idle-timeout-ms:`.** That'll likely change when the structural fix lands and the underlying primitive moves from wall-clock to idle-timeout. We chose the misnomer-tolerant present-day name on purpose: the cft schema can rename keys with a compatibility alias in one release, and naming the new key for what it actually does *today* keeps the surface honest until the underlying mechanism changes. + +**Length not trimmed.** The market-map template's system prompt is over 1,000 words of guidance. The skeleton with bullet instructions is another 1,000+. A first instinct was to trim — most prompts of that length are over-engineered — but every section of the template's prompt is doing real work: the editorial stance, the innovator-card shape, the dual-flavor detection, the wikilink discipline, the calibration on length ("lean long, not short"). We left it long. The cost is a longer first-time-readable system prompt for vault users; the benefit is fewer regenerations and a draft that's closer to a partner-ready document. + +## Files touched + +``` +plugin-modules/perplexed/ +├── src/docs/templates/ +│ ├── market-map-profile.md (new) +│ └── README.md (cft-block-keys list now mentions request-timeout-ms) +├── src/services/ +│ ├── directoryTemplateService.ts (request-timeout-ms override resolution) +│ └── templateSeederService.ts (market-map-profile registered as 5th seeded template) +├── docs/ +│ └── directory-templates.md (shipped-templates table now has market-map row; +│ new "Per-template timeout override" section) +├── context-v/ +│ ├── explorations/ (new folder; opens with the multi-stage doc) +│ │ └── Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md +│ └── issues/ +│ └── Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md +└── main.ts + ├── directoryTemplatesRequestTimeoutMs default: 600000 → 1800000 + └── settings-pane description rewritten with the override + cost framing +``` + +Pointer-bumped at the parent: `content-farm` commit `e31b982` folds in the perplexed pointer (`a44f23b`) and a 3-line drag-along addition to `context-v/plans/Create-a-Study-of-the-Best-Obsidian-Plugins.md` (`neural-composer` by oscampo flagged as a study-candidate plugin). + +## What's next + +- **Run the rebuilt plugin against Humanoid Robots in append mode** to extend the truncated tail. With the 40-minute ceiling on the market-map template, the *Frontier and Open Questions* section should finish and *Adjacent Concepts and Maps* should land for the first time. +- **The structural idle-timeout fix.** The wall-clock primitive in `directoryTemplateService.ts:511` has a known better replacement living in the same codebase (`perplexityService.ts:659-668`). The plan and the open design questions live at `context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md`. Worth doing before the multi-stage spec lands, because stage 3 (Claude editorial on a 7-8K-word draft) will hit the same pathology as stage 2 (Perplexity research). +- **The multi-stage Claude+Perplexity+RAG spec.** The exploration doc commits to a tentative direction (Option A — zoned single-file appends with `include-sources:` and `editor:` keys added to the cft schema). The next move is a spec in `context-v/specs/` pinning down the schema additions, the zone-delimiter format, the resolution order for `include-sources`, and the migration path for the existing four templates. Then a chunked prompt in `context-v/plans/`. Then `market-map-profile` becomes the reference implementation. +- **The anti-incumbent editorial stance is now duplicated across four templates.** `concept-profile`, `vocabulary-profile`, `source-profile`, and now `market-map-profile` all carry a variation. Per the partials/preambles work that landed 2026-05-19, this is the next obvious extraction — `partials/editorial-stance-anti-incumbent.md` referenced via `{{include: editorial-stance-anti-incumbent}}`. Not done today; flagged. diff --git a/context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md b/context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md new file mode 100644 index 0000000..a594689 --- /dev/null +++ b/context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md @@ -0,0 +1,346 @@ +--- +title: "Multi-Stage Cooperative Claude + Perplexity with RAG" +lede: "What if a market-map draft were the output of three agents — a RAG pre-flight that names what to include, a Perplexity research run that finds the rest, and a Claude editorial pass that never steps on either — instead of one single-shot prompt?" +date_created: 2026-05-26 +date_modified: 2026-05-26 +authors: + - Michael Staton +augmented_with: + - Claude Opus 4.7 (1M context) +semantic_version: 0.0.0.1 +tags: + - Exploration + - Perplexed + - Multi-Stage-Generation + - Agentic-Orchestration + - RAG + - Chroma + - Market-Maps + - Claude + - Perplexity-Deep-Research +status: Open +related: + - "[[market-map-profile]]" + - "[[Using-Files-as-Prompt-Outlines]]" + - "[[Partials-And-Preambles-For-Perplexed-Templates]]" + - "[[Getting-Claude-to-Respond-With-Research]]" + - "[[Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams]]" +--- + +# Multi-Stage Cooperative Claude + Perplexity with RAG + +A market map is the kind of document where one-shot generation hits its limit. The lean v1 `market-map-profile` template ([`src/docs/templates/market-map-profile.md`](../../src/docs/templates/market-map-profile.md)) produces a respectable analyst draft in a single Perplexity deep-research run, but every analyst who has read one of our existing market maps notices the same gaps: the wikilinks back to our own `Tooling/` and `concepts/` aren't there, the segmentation occasionally misses a sub-bucket we cover heavily in the vault, and the editorial voice drifts from "well-paid analyst" toward "encyclopedia summary" without a second pass. The fix is multi-stage, but multi-stage has more design surface than it looks: who runs first, how they hand off, where the RAG context lives, and — the question that motivated this doc — **how do we keep Claude and Perplexity from overwriting each other's work?** + +## The question + +We want to evolve `market-map-profile` (and, by precedent, any other research-heavy template) from a one-shot Perplexity run into a cooperative pipeline that includes: + +1. A **RAG pre-flight** that pulls canonical Lossless source material (vault tools, concepts, prior market maps; Chroma-indexed corpus chunks) and bakes it into the prompt context as named, citeable inputs. +2. A **Perplexity research stage** that does what `sonar-deep-research` does today, but primed with the RAG context so it cites our own canonical entries by name rather than generating shadow versions of them. +3. A **Claude editorial stage** that enforces analyst voice, prunes over-enumeration, restructures sub-segments where the data demands, emits `[[wikilink]]`s for entries that exist in the vault, and surfaces "the question this map answers" framing. + +The specific design questions: + +- **How do RAG-provided sources get expressed in a template?** The user articulated this as an `include_sources:` argument — a list of sources to include, distinct from `search-domains:` (which constrains where Perplexity searches the open web). What's the surface shape — a list of file paths, a Chroma query spec, both? +- **How do the three stages share a target file without clobbering each other?** Perplexity streams text into the file body. Claude's editorial pass needs to *edit* that text. If Claude runs while Perplexity is still streaming, or if Claude edits the same region Perplexity wrote, we get a race or a confused output. +- **Single file or multi-document folder?** A market map could remain one large markdown file with sectioned output, or it could become a folder where each sub-segment is its own document with a manifest at the root. The latter is more agentic-friendly (each section is independently re-runnable) but breaks the single-file mental model that every other directory template uses. +- **Where does the RAG index live, and how does it stay fresh?** The Chroma collections in [`lossless-monorepo/CLAUDE.md`](../../../../CLAUDE.md) (`context-vigilance-corpus`, `lossless-changelog`, `claude-code-sessions`, `claude-code-tool-traces`) are reachable via the `chroma` MCP server inside Claude Code, but the perplexed plugin runs inside Obsidian — it can't reach an MCP server. The plugin would need its own client to the same Chroma instance, OR the RAG pre-flight runs *outside* the plugin and writes its output into the target file before the plugin's Perplexity stage triggers. + +## Why we don't already know + +Three reasons. + +**One — the existing `cft` template surface is intentionally minimal.** The `ParsedTemplate` interface ([`src/services/directoryTemplateService.ts:31`](../../src/services/directoryTemplateService.ts)) gives a template four things: a config block, a system prompt, a user skeleton, and the four shipped templates ride that surface comfortably because each is *one* provider call. The moment we add a second provider — Claude after Perplexity, or a RAG client before either — we need to decide whether the template is still one file (with multiple stages declared inside) or whether it's a *pipeline* with a per-stage manifest. We have no precedent in this codebase for the latter. + +**Two — `include_sources` isn't a Perplexity API field, and we haven't decided what it means.** `search_domain_filter` constrains where Perplexity searches; it doesn't inject our content into the prompt. What the user is gesturing at — "a list of sources to include" — is closer to **retrieval-augmented generation**: pull canonical text from named documents (or a Chroma vector query), splice it into the prompt as named, citeable inputs, then let the model lean on it. We already have the splice primitive (`{{include: name}}` partials, [`directoryTemplateService.ts:83`](../../src/services/directoryTemplateService.ts)) — but partials are pure snippets, content-agnostic. RAG sources need different handling: they want to be named, attributed, and bounded so the model knows what it's quoting. + +**Three — we don't yet know whether the editing stage should run as an *edit* or as an *additive pass*.** The user's framing — *"Perplexity and Claude do not overwrite each other until editing"* — implies a model where Perplexity writes its draft in one zone of the file, Claude writes in another, and only at the explicit edit-phase do they consolidate. That's a real design choice with consequences for the file shape, the cft block schema, and how the user knows where they are in the pipeline. + +## Options + +Three architectures to weigh. They are not mutually exclusive — Option C is closer to a generalization of A and B than a distinct third path. + +### Option A — Sequential single-file appends with named zones + +The template stays one markdown file in `zz-cf-lib/templates/`. The cft block grows new keys but the file shape stays familiar. + +Pipeline: + +1. **RAG pre-flight (in-plugin).** Plugin reads `include_sources:` from the cft block, resolves it (vault paths and/or Chroma queries), splices the retrieved text into the system prompt as a named context block. No file writes happen at this stage — RAG context flows straight into the prompt. +2. **Perplexity stage.** Runs as today: streams body content into the target file, stamps `cf_last_run` / `cf_last_run_model`, appends `# Sources` footer. Output lands in a zone delimited by `` and `` HTML comments. +3. **Claude editorial stage.** Reads the entire file, runs Claude with a specific edit prompt (`edit-system:` from the cft block, or a separate `claude` block), writes the edited version into a NEW zone delimited by `` and ``. The Perplexity zone is preserved untouched as provenance; the Claude zone is what the published doc draws from. + +cft block grows: + +```yaml +provider: perplexity +model: sonar-deep-research +include-sources: + - vault: "Tooling/AI-Toolkit/Agentic AI/**" + - vault: "concepts/Explainers for AI/Agentic Workspaces.md" + - chroma: + collection: context-vigilance-corpus + query: "{{title}} market map prior art" + n_results: 5 +system: | + ... + +editor: + provider: anthropic + model: claude-opus-4-7 + pass: editorial-restructure + system: | + You are editing the Perplexity-drafted market map below into final form... +``` + +**Pros:** + +- One file per market map; consistent with every other directory template. +- The zoned-append discipline (Perplexity zone, Claude zone) makes provenance auditable — you can read exactly what each agent produced. +- Both stages re-runnable independently. Re-run Perplexity to refresh the source data; re-run Claude to refresh the edit without re-burning a deep-research credit. +- `include-sources` becomes a structured field that handles both vault-path and Chroma-query forms, with the resolution logic centralized in the plugin. + +**Cons:** + +- One file gets long fast. Market maps are already 4-8K words; doubling the body to keep both zones means the on-disk file is hefty. Obsidian handles it, but readability degrades. +- The Chroma client has to run inside the Obsidian plugin. That means embedding a Chroma HTTP client in `main.js`, configuring the Chroma URL in plugin settings, and handling the failure mode where Chroma is unreachable. +- The HTML-comment zone markers are fragile — a user editing the file in another pane could break them, and at that point the re-run logic doesn't know where to write. +- Claude's editorial output sitting in a sibling zone means downstream rendering (the public site) needs to know to draw from the Claude zone, not the Perplexity zone. That's a coupling that didn't exist before. + +### Option B — Multi-document folder, section per file, with a manifest + +Each market map becomes a folder, not a file. `lost-in-public/market-maps/Quantum Computing is Confusing/` rather than `Quantum Computing is Confusing.md`. Inside the folder: + +``` +Quantum Computing is Confusing/ +├── _manifest.md # frontmatter only — title, lede, banner_image, etc. +├── 00-snapshot.md # Market Snapshot section +├── 01-question.md # The Question this Map Answers +├── 02-why-now.md # Why Now +├── 03-map.md # Map of the Market — Sub-Segments +├── 04-lighthouse.md # Lighthouse Examples +├── 05-profiles-superconducting.md # one file per sub-segment +├── 05-profiles-photonic.md +├── 05-profiles-trapped-ion.md +├── 06-media.md +├── 07-dynamics.md +├── 08-frontier.md +├── 09-adjacent.md +└── _sources.md # consolidated sources footer +``` + +Each `NN-*.md` file is independently re-runnable. The template becomes a *pipeline manifest* that names which agent runs on which file and in what order. The cft format extends to declare per-file stages. + +Pipeline: + +1. **Scaffold.** A new command — `Initialize market map folder` — creates the folder structure from a template manifest, populating each section file's frontmatter with its position in the pipeline. +2. **Per-section RAG + Perplexity.** Each section file declares its own `include-sources:` and runs its own targeted Perplexity query. Sub-segment files (`05-profiles-*.md`) get the most aggressive RAG injection (vault tools matching the sub-segment). +3. **Section-level Claude pass.** Claude edits each section file independently against a section-specific edit prompt. +4. **Roll-up.** A final command stitches the section files into one rendered markdown file for publication, or the public site renders the folder directly via a folder-aware renderer. + +**Pros:** + +- Each section is independently re-runnable. The biggest pain in a one-shot 6K-word generation is "the trapped-ion sub-segment is thin, I'd like to re-run just that one." Folder form makes that natural. +- RAG context can be tightly scoped per section — the Lighthouse Examples section gets a different vault query than the Media section. With one mega-prompt, RAG context is one-size-fits-all. +- Per-section editorial passes are cheaper. Editing a 600-word section is faster and tighter than editing a 6K-word document. +- The folder maps cleanly to how an actual analyst works — they don't write the whole memo in one sitting; they research the sub-segments first, then sequence the framing prose. +- Agentic orchestration becomes natural: a future orchestrator-agent can iterate over the section files, decide which need refresh, and dispatch jobs in parallel. + +**Cons:** + +- Breaks every existing convention in the perplexed plugin and the vault. The four shipped templates target one file each; making market maps target a folder is a tier shift. +- The existing `applies-to-paths` glob ([`directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts)) matches a file against a template; we'd need parallel `applies-to-folders` matching, or a new command entirely. +- Cross-section consistency is harder. The Innovator Profiles section needs to reference the same sub-segment names declared in the Map of the Market section. With separate files, drift across sections becomes likely. +- The published-on-the-web rendering needs a folder-aware renderer. Astro Knots can do this (it already renders nested content), but every existing market map is a single file and migrating them is its own project. +- Users edit market maps as one document in their head. A folder breaks that mental model. + +### Option C — Per-section refresh blocks inside a single file + +Take the multi-`cft` idea floated in the directory-templates doc's *Known limits and open items* and run with it. The template file stays one markdown file. The skeleton declares per-section refresh prompts via dedicated `cft` blocks inline under each H1. + +```markdown +# Market Snapshot + +```cft-section +provider: perplexity +model: sonar-deep-research +include-sources: [...] +prompt: | + Write the Market Snapshot section per the instructions below. +``` + +- Bullet instructions for this section... + +# Innovator Profiles + +```cft-section +provider: perplexity +model: sonar-deep-research +include-sources: + - vault: "Tooling/AI-Toolkit/Agentic AI/**" +prompt: | + Write the Innovator Profiles section. Use RAG content to name actual + vault tools where possible. +``` + +- Bullet instructions... +``` + +A header-level `cft` block stays at the top of the template declaring the editorial stage (Claude pass) that runs across the whole file at the end. + +Pipeline: + +1. **Section-by-section Perplexity runs.** The plugin walks each `cft-section` block and runs its own targeted query. Section content lands under that H1. +2. **File-level Claude editorial pass.** After all sections are filled, Claude reads the assembled file and writes the edited version per the file-level `editor:` block. + +**Pros:** + +- One file per market map (keeps Option A's familiarity). +- Per-section RAG and Perplexity targeting (gets Option B's tight scoping). +- Re-runnable per section — invoke `Refresh section` on the active section's H1 to re-run just that `cft-section`. +- Sub-segment-aware RAG: each section's `include-sources` can be tuned to what that section needs. + +**Cons:** + +- The `cft-section` schema is novel; we'd have to design the multi-block parser, the per-section command surface, and the interaction with the existing one-`cft` flow. +- The file is longer at-rest because it carries every section's prompt inline. This is a feature (visible provenance) and a bug (clutter) at the same time. +- "Append vs. fill" semantics become per-section. The mode logic in `applyTemplate` gets more states. + +## The `include_sources` sub-exploration + +This is the design call most likely to set the trajectory of everything else. The user named it as a key argument, and it sits at the intersection of three primitives we already have: + +| Primitive | What it does today | What's missing for `include_sources` | +|---|---|---| +| `search-domains:` ([`directoryTemplateService.ts:486`](../../src/services/directoryTemplateService.ts)) | Constrains Perplexity's open-web search to a domain allowlist + a job-board denylist | Doesn't inject content into the prompt; only filters where the model searches | +| `{{include: name}}` partials ([`directoryTemplateService.ts:83`](../../src/services/directoryTemplateService.ts)) | Splices a named partial's body into the prompt, content-agnostic | No naming/attribution wrapper for the spliced content; no Chroma-query form | +| Chroma collections (`context-vigilance-corpus` et al, [`lossless-monorepo/CLAUDE.md`](../../../../CLAUDE.md)) | Section-chunked vault content indexed and queryable via the `chroma` MCP server | Only reachable from inside Claude Code today, not from inside Obsidian plugin runtime | + +Proposed shape for `include-sources:` in the cft block: + +```yaml +include-sources: + # Vault path — read the file, splice its body into the prompt as a named, + # attributed context block. Glob form is allowed; matched files are + # concatenated with their basenames as the attribution label. + - vault: "concepts/Explainers for AI/Agentic Workspaces.md" + - vault: "Tooling/AI-Toolkit/Agentic AI/**" + max_files: 10 # cap for glob expansion + body_only: true # strip each file's frontmatter (default true) + + # Chroma query — run a semantic search against a named collection, + # splice the top N results into the prompt with their source_path + # and source_repo_slug as attribution. + - chroma: + collection: context-vigilance-corpus + query: "{{title}} market segmentation" + n_results: 5 + where: # optional metadata filter + source_repo_slug: "lossless" + + # External URL — fetch and inline. Lowest priority because we can't + # cache, but useful for one-off canonical references the user wants + # baked in. Requires explicit opt-in in settings. + - url: "https://www.lossless.group/projects/gallery/agentic-workspaces" + max_chars: 8000 +``` + +How the spliced content lands in the prompt: + +``` +## Canonical sources (use these as primary attributions where applicable) + +### Source: Tooling/AI-Toolkit/Agentic AI/Crew AI.md + + +### Source: concepts/Explainers for AI/Agentic Workspaces.md + + +### Source: Chroma (context-vigilance-corpus) — n=3 + + +## Search the open web for additional sources beyond the canonical set above. +``` + +This shape solves the "named, attributed" problem (the model knows what it's quoting and can cite by vault path), keeps glob expansion bounded (`max_files`), and degrades gracefully when Chroma is unreachable (skip the chroma entries with a warning notice; vault entries still resolve). + +Open sub-questions: + +- **Does `include-sources` belong at the template level or the section level?** Template-level is simpler. Section-level is more powerful but only matters if we go Option B or C. +- **How big can the canonical-sources block grow before we OOM the prompt?** `sonar-deep-research` has generous context but not infinite. A vault glob matching 100 files is too much. The `max_files` cap matters; defaults should be conservative (5-10). +- **Is the Chroma client an in-plugin dependency or an out-of-plugin pre-flight?** If we ship a Chroma client in `main.js`, plugin size grows and we need to surface Chroma URL configuration in settings. If we keep RAG pre-flight outside the plugin (a CLI that writes a `_rag-context.md` partial the plugin then includes), the plugin stays lean but the workflow gains an extra step. + +## "Perplexity and Claude do not overwrite each other until editing" + +This is the design heuristic that distinguishes Option A's zoned-append from a naive sequential pipeline. Three implementations of the heuristic, in increasing strictness: + +1. **Soft separation (zones).** Perplexity writes into one zone, Claude writes into another. Both zones live in the same file. Provenance is preserved; the published doc draws from the Claude zone but the Perplexity zone is readable. +2. **Hard separation (files).** Perplexity writes its draft to `.perplexity.md`. Claude reads that file and writes its edit to `.md` (or vice versa, depending on which is canonical). The original is preserved untouched. +3. **Strict additive (no edits, only appendages).** Both agents append; neither edits. The final document is a stitched sequence of their outputs with explicit attribution. Useful if we want to preserve "the analyst's draft" and "the editor's reorganization" as separate readable artifacts. + +The strict additive form (3) has appeal — it makes the multi-agent collaboration audit-trail-perfect — but it's hostile to the actual editorial goal, which is *to merge the two voices into one coherent document*. Soft separation (1) hits the right tradeoff for a v1: provenance is preserved, the final published form is one voice, and re-running either stage doesn't destroy the other's work. + +The "until editing" clause is where the user's framing gets sharp. **The editing stage is the only point at which the two agents' outputs are consolidated.** Before that, they're parallel artifacts. This is why Option A's zoned append is the most faithful interpretation: Perplexity and Claude do their work in parallel zones; the editing stage (which Claude itself performs) is when the consolidation happens, producing the canonical Claude zone. + +## Live example surfacing the timeout issue + +The first real-world run of the v1 `market-map-profile` template — `lost-in-public/market-maps/Humanoid Robots and their Input Industries.md`, executed 2026-05-26 — produced a ~7,500-word draft that terminated mid-sentence inside *Frontier and Open Questions* because the directory-template runtime's wall-clock `AbortController` fired before the stream completed. That truncation is the motivating use case for the **per-template `request-timeout-ms:` cft-block override** we shipped on the same day as this exploration (default bumped to 30 min, market-map-profile overridden to 40 min). The pressure-relief valve is in place; the structural fix — porting the idle-timeout discipline from `perplexityService.ts:659-668` into `streamPerplexityToFile` — is captured as its own issue at [[Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams]] and feeds back into this exploration's *Open items* below. + +This is the shape of validation we want for the multi-stage pipeline overall: ship the lean v1, run it on a real document, surface the structural gaps as their own context-v issues (where they get the attention they deserve), and let the issues feed into the eventual spec. + +## Findings so far + +From this exploration session, the [partials-and-preambles issue](../issues/Partials-And-Preambles-For-Perplexed-Templates.md) the architecture builds on, and the [wall-clock-timeout issue](../issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md) surfaced by the first real market-map run: + +- **The `{{include: name}}` partial primitive is already structurally close to what `include-sources` needs.** What's missing is (a) the per-source attribution wrapper, (b) the Chroma-query form, and (c) the multi-source named-block formatting in the prompt. The partial loader's depth and cycle guards transfer directly. +- **The cft config is already a structured YAML surface.** Adding `include-sources:` (a list) and `editor:` (an object) doesn't break the existing parser; the four shipped templates remain valid because they don't declare either key. +- **`search-domains:` and `include-sources:` are complementary, not redundant.** Domains constrain where Perplexity searches the open web; include-sources injects canonical content into the prompt directly. A future market-map run would likely use both: include-sources to seed our canonical entries, search-domains to keep open-web search restricted to credible analyst sources (Bloomberg, FT, sector trade press, founder blogs). +- **Claude can already stream to a file** ([`claudeService.ts:48`](../../src/services/claudeService.ts) `queryClaude`). The editorial stage doesn't need a new transport — it needs a different prompt construction path and a different write-zone strategy. +- **The `editor:` block can be opt-in.** Templates that don't declare it stay single-stage and behave exactly as they do today. The four shipped templates would stay single-stage; only `market-map-profile` and any future research-heavy templates would declare an `editor:`. +- **Chroma reachability from inside Obsidian is the long pole.** The MCP server only exists in Claude Code; the Obsidian plugin runs in Electron and needs its own HTTP client to a Chroma instance, OR we keep RAG pre-flight as an out-of-band step that writes a context partial the plugin then includes. + +## Tentative direction + +Lean toward **Option A (zoned single-file appends) with the `include-sources:` and `editor:` keys added to the cft schema**, and keep RAG pre-flight as an *in-plugin* feature with Chroma queries optional (vault-path includes always work; Chroma includes degrade to a Notice when unreachable). + +Reasoning: + +- Option A is the smallest delta from the existing template surface. The four shipped templates remain unchanged; market-map-profile gains two new keys. +- Vault-path includes get us most of the RAG win without the Chroma-reachability problem. The dominant value of "include our canonical tools and concepts" is satisfied by globbed vault includes; Chroma adds discoverability over the wider corpus but isn't load-bearing for the first iteration. +- Zoned appends preserve the "Perplexity and Claude do not overwrite each other until editing" heuristic literally: each agent writes its zone, the editing stage consolidates. If we want hard separation later, we can promote the zones into separate files without redesigning the schema. +- Option B (multi-doc folder) is the right end-state for sufficiently complex maps but is a tier shift that should be earned, not pre-emptively designed. +- Option C (per-section cft blocks) is the right *generalization* but is also the right way to over-design. Defer until we've shipped Option A and felt the actual pain. + +Defer for a follow-up spec: + +- The Chroma in-plugin client work. Ship vault-path includes first; revisit Chroma after we know whether the in-plugin client size cost is worth the discoverability gain. +- The multi-doc folder form (Option B). Track as a possible v3 once we have 10+ market maps and can feel the cross-section consistency pain. +- The per-section `cft-section` block (Option C). Track as a possible v2 alternative if Option A's zoned appends turn out to be insufficient for sub-segment scoping. + +## Open items before we promote to a spec + +- [ ] Confirm zone delimiter shape. HTML comments are tolerant in Obsidian and Astro Knots renderers; named-anchor headings (``) work too. Pick one and document. +- [ ] Decide whether `editor:` is a block or a list. A list (`editor: [{ provider: anthropic, ... }, { provider: anthropic, pass: copyedit, ... }]`) would let us chain multiple Claude passes; a block keeps the single-pass v1 simpler. +- [ ] Spec the `include-sources` resolution order and de-duplication policy. If a vault glob and a Chroma query both surface the same source file, do we splice it twice? +- [ ] Decide attribution-block format. Does each included source get a top-level H2 in the prompt context, or a fenced block, or YAML-front-matter-wrapped? Trade off model-comprehension against prompt-token economy. +- [ ] Decide what `cf_last_run` stamps look like for a multi-stage run. One timestamp per stage (`cf_last_run_perplexity`, `cf_last_run_editor`)? A composite? This matters for the "is this map stale?" query later. +- [ ] Verify Chroma collections are queryable from a non-MCP HTTP client. The [chroma-local skill](../../../../context-v/skills/chroma-local/SKILL.md) covers `ChromaClient`/`HttpClient` setup; should be a 30-minute spike. +- [ ] Resolve [[Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams]] before the multi-stage spec lands — the current wall-clock primitive cuts off long stage-2 (Perplexity) runs and will cut off long stage-3 (Claude editorial) runs too, since the editor's per-section reasoning on a 7-8K-word draft is itself a long generation. The structural fix (idle-timeout discipline) belongs upstream of multi-stage, not as part of it. + +## Outcome + +Open. When this resolves, the expected artifacts are: + +1. A spec in [`context-v/specs/`](../specs/) — `Multi-Stage-Templates-with-Include-Sources-and-Editor.md` (working title) — that pins down the cft schema additions, the zone-delimiter format, the resolution order for `include-sources`, the editorial-stage contract, and the migration path for existing templates. +2. A prompt in [`context-v/plans/`](../plans/) — chunked implementation steps for the spec. +3. The `market-map-profile` template updated to use the new keys, treated as the reference implementation. +4. A v2 follow-up exploration in this folder once we've felt how Option A actually behaves with one or two real market maps generated through it. + +## Related + +- [[market-map-profile]] — the lean v1 template this exploration extends +- [[Using-Files-as-Prompt-Outlines]] — the original spec that motivated directory templates +- [[Partials-And-Preambles-For-Perplexed-Templates]] — the issue that established the splice primitive `include-sources` builds on +- [[Getting-Claude-to-Respond-With-Research]] — prior work on Claude as a research-aware agent +- [[search-lossless-corpus]] — the Claude-Code-side skill that already encodes the four-collection RAG discipline; the in-plugin RAG flow is the Obsidian analog +- [`lossless-monorepo/CLAUDE.md`](../../../../CLAUDE.md) — the canonical description of the four Chroma collections this exploration's RAG stage would draw on diff --git a/context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md b/context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md new file mode 100644 index 0000000..b43340f --- /dev/null +++ b/context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md @@ -0,0 +1,119 @@ +--- +title: Wall-clock timeout cuts off long deep-research streams +lede: "The directory-template runtime caps every stream by total wall-clock duration, but the legacy modal flow already moved to per-chunk idle-timeout discipline two iterations ago — and the discrepancy is now actively truncating analyst-grade market-map drafts mid-sentence." +date_created: 2026-05-26 +date_modified: 2026-05-26 +authors: + - Michael Staton +augmented_with: + - Claude Opus 4.7 (1M context) +semantic_version: 0.0.0.1 +type: issue +status: open +target_repo: perplexed +tags: + - Issue-Resolution + - Perplexed + - Streaming-Timeouts + - Deep-Research + - Directory-Templates +related: + - "[[market-map-profile]]" + - "[[Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG]]" + - "[[Partials-And-Preambles-For-Perplexed-Templates]]" +--- + +# Wall-clock timeout cuts off long deep-research streams + +## Symptom + +Running `market-map-profile` on `lost-in-public/market-maps/Humanoid Robots and their Input Industries.md` produced a ~7,500-word draft that terminated mid-sentence inside the *Frontier and Open Questions* section: + +``` +Will the evolution of robot-as-a-service (R +``` + +The trailing parenthesis is the last byte written. The *Adjacent Concepts and Maps* section — the final heading in the template skeleton — never appeared. The frontmatter stamps (`cf_last_run`, `cf_last_run_model`) landed correctly, so the run did complete its `processFrontMatter` post-step; what was lost was the streamed body content that hadn't yet arrived when the `AbortController` fired. + +The Humanoid Robots run is the first observable instance of this specific cut-off shape, but the pattern is structural — it will reproduce on any sufficiently long deep-research generation run through the directory-template flow. + +## Diagnosis + +Two different streaming primitives live in this codebase, with two different timeout disciplines: + +**Directory-template flow** ([`src/services/directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts) `streamPerplexityToFile`, the function `market-map-profile` and the four other shipped templates run through): + +```ts +const controller = new AbortController(); +const timer = activeWindow.setTimeout(() => controller.abort(), timeoutMs); +``` + +A **single wall-clock `setTimeout`** is armed at the moment of fetch. After `timeoutMs` elapses — regardless of whether the stream is actively producing bytes — `controller.abort()` fires and the in-flight `reader.read()` throws. The catch sets `truncated = true` and falls through to a final flush of whatever streamed so far. The plugin-level default was `600_000` ms (10 min) before today's fix; analyst-grade deep-research runs routinely run 15-25 min, so the cut-off was inevitable on long templates. + +**Legacy modal flow** ([`src/services/perplexityService.ts:659`](../../src/services/perplexityService.ts), `PerplexityModal`): + +```ts +const STREAM_IDLE_TIMEOUT_MS = isDeepResearch ? 270_000 : 90_000; +const readWithIdleTimeout = (): Promise<...> => { + let timer: number | undefined; + const timeout = new Promise((_, reject) => { + timer = window.setTimeout(() => { + reject(new Error(`stream went idle for ${...}s ...`)); + }, STREAM_IDLE_TIMEOUT_MS); + }); + return Promise.race([reader.read(), timeout]).finally(() => { + if (timer !== undefined) window.clearTimeout(timer); + }); +}; +``` + +**Per-chunk idle timeout.** A fresh `setTimeout` is armed and racing each `reader.read()` call. As long as bytes keep arriving, the timer keeps getting cleared and re-armed. The stream is only killed if it goes *quiet* for 270 s (deep-research) or 90 s (normal). Total wall-clock duration is unbounded. + +The legacy modal moved to this pattern because the same problem hit users there first — but the directory-template flow was forked from an earlier iteration of the streaming code and never received the idle-timeout backport. The legacy `PerplexityModal` and `streamPerplexityToFile` now disagree on how to time-bound a Perplexity stream, and the directory-template flow has the strictly weaker discipline. + +## What we shipped today (partial fix) + +Not a structural fix — a pressure-relief valve. Two changes: + +1. **Bumped the plugin-level default** ([`main.ts:333`](../../main.ts)) from `600_000` ms (10 min) to `1_800_000` ms (30 min). The settings-pane description was updated to call out the override and the cost framing (`$10-$50 of analyst time per good output is worth waiting for`). +2. **Added a per-template override** — `request-timeout-ms:` in the cft block. Resolution code lives in [`directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts) just before the `streamPerplexityToFile` call; accepts number or numeric-string, silently falls back on non-positive / non-numeric values. +3. **`market-map-profile.md`** declares `request-timeout-ms: 2400000` (40 min) with an inline comment explaining the budget; the other four shipped templates inherit the plugin-level 30-min default. +4. **Docs** — [`docs/directory-templates.md`](../../docs/directory-templates.md) gained a *Per-template timeout override* section with override semantics and a "when to bump" checklist; [`src/docs/templates/README.md`](../../src/docs/templates/README.md) calls out the key in the cft-block list. + +This buys headroom. It does not fix the structural problem: any sufficiently long deep-research run will still hit the wall eventually. The 40-min cap is a *guess* about how long the longest reasonable market-map should take, not a property derived from the stream's actual behavior. + +## Why this is only a partial fix + +The wall-clock timeout fails in two distinct shapes that the idle-timeout pattern handles correctly: + +**Shape 1 — slow but healthy stream.** Deep-research generations on long templates may sustain a slow trickle of tokens for 30-45 minutes. The wall-clock cap kills them at the ceiling regardless of whether they're still producing. The idle-timeout pattern lets them complete naturally as long as some byte arrives every N seconds. + +**Shape 2 — silently stalled stream.** Conversely, a stream may go quiet at minute 3 (Perplexity rate-limit, socket close, upstream stall) and the wall-clock cap won't notice until minute 30. The user stares at an empty file, watching the spinner spin, for 27 unnecessary minutes. The idle-timeout pattern surfaces the failure within `STREAM_IDLE_TIMEOUT_MS` seconds — fast feedback when something is genuinely wrong. + +Both shapes are real. The wall-clock pattern punishes the healthy-but-slow case while tolerating the stalled-but-silent case. The idle-timeout pattern inverts both — slow-but-healthy completes; stalled-but-silent fails fast. + +## Proposed structural fix + +Port the idle-timeout pattern from [`perplexityService.ts:659-668`](../../src/services/perplexityService.ts) into `streamPerplexityToFile`. Concretely: + +1. Replace the single wall-clock `setTimeout(controller.abort, timeoutMs)` with a per-chunk `readWithIdleTimeout()` that wraps each `reader.read()` in a `Promise.race` against a fresh timeout. +2. Choose idle-timeout values consistent with the existing modal flow: `270_000` ms (4.5 min) for deep-research models, `90_000` ms (1.5 min) for normal models. Detect deep-research from the resolved model name string (`/deep-research/i`), the same way `perplexityService.ts` does. +3. Retain the cft-block override key, but rename it to `stream-idle-timeout-ms:` for accuracy. Templates that have been declaring `request-timeout-ms:` need a compatibility alias for one release; document the rename in the changelog entry. +4. Optionally retain a generous absolute wall-clock ceiling (60 min or 120 min) as a sanity backstop — but that's belt-and-suspenders, not load-bearing. The idle timeout is doing the actual safety work. + +The diff is moderate — `streamPerplexityToFile` is ~140 lines today; the refactor touches roughly the first 40 of those (the timer setup and the `reader.read()` call inside the loop). The post-stream cleanup pipeline (`wrapThinkBlocks`, `processContentWithImages`, `buildSourcesFooter`) is unaffected. + +## Open items before this becomes a spec + +- [ ] Decide whether the cft-block key stays named `request-timeout-ms:` (which becomes a misnomer once idle-timeout is what's actually applied) or migrates to `stream-idle-timeout-ms:`. Compatibility-alias plan if the latter. +- [ ] Decide whether to retain a wall-clock absolute ceiling alongside the idle timeout, and if so what the value is (60 min? 120 min? infinite with a Notice that surfaces "this run has been going for X min" past a threshold?). +- [ ] Audit other long-stream callsites in the codebase that may have the same wall-clock pathology (Gemini service, LM Studio service, Claude service streaming flows) — the idle-timeout discipline should be the house style across all of them. +- [ ] Confirm the model-name-based deep-research detection (`/deep-research/i` regex on the resolved model) is robust against future Perplexity model name changes; consider exposing the idle-timeout values as plugin settings so they can be tuned without a code change. + +## Related + +- [[market-map-profile]] — the template that surfaced the bug +- [[Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG]] — the broader exploration this issue feeds findings back into; the idle-timeout refactor is listed there as an open item +- [[Partials-And-Preambles-For-Perplexed-Templates]] — the architecture-review structure of that issue is the precedent this one follows +- [`src/services/perplexityService.ts`](../../src/services/perplexityService.ts) lines 659-668 — the idle-timeout implementation in the legacy modal flow that we're proposing to port +- [`src/services/directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts) `streamPerplexityToFile` lines 499-635 — the wall-clock implementation in the directory-template flow that the port replaces diff --git a/docs/directory-templates.md b/docs/directory-templates.md index 557f19d..d233131 100644 --- a/docs/directory-templates.md +++ b/docs/directory-templates.md @@ -79,7 +79,7 @@ # User Notes ### The three zones 1. **Frontmatter** (between `---` lines at the top) — carries `title`, `applies-to-paths` (array of globs), and an optional `description`. The runtime uses `applies-to-paths` to match a template to a target file. -2. **`cft` block** (a code fence with language `cft`) — YAML carrying `provider`, `model`, optional `search-recency`, `return-citations`, `return-images`, and a multi-line `system:` prompt. Everything **above** the `cft` block is dropped from the request. +2. **`cft` block** (a code fence with language `cft`) — YAML carrying `provider`, `model`, optional `search-recency`, `return-citations`, `return-images`, optional `request-timeout-ms:` (per-template wall-clock override — see *Per-template timeout override* below), and a multi-line `system:` prompt. Everything **above** the `cft` block is dropped from the request. 3. **Heading skeleton** — the markdown structure between the `cft` block's closing fence and the first `***` divider. This becomes the user prompt. Bullets under each heading are *instructions to the model*, not literal output. Everything **below** the first `***` is excluded from the request. ### Interpolation tokens @@ -140,7 +140,7 @@ ### Frontmatter stamps ## Shipped templates -Four templates ship inlined into `main.js` (via esbuild's `.md` text loader) and are seeded into the user's vault on first plugin load. +Five templates ship inlined into `main.js` (via esbuild's `.md` text loader) and are seeded into the user's vault on first plugin load. | File | Targets | Model | Notes | |---|---|---|---| @@ -148,6 +148,7 @@ ## Shipped templates | `vocabulary-profile.md` | `Vocabulary/**` | `sonar-pro` | Term definitions with disambiguation through an innovation-consulting lens. | | `source-profile.md` | `Sources/**` | `sonar-pro` | Profiles of trusted sources — books, people, channels, publications, journals, reports, events. Type-aware: the system prompt enumerates seven canonical types and the model picks one from frontmatter signals (`youtube_channel_url` → channel, `aliases` → likely book, etc.). Each section has per-type bullet shapes. | | `toolkit-profile.md` | `Tooling/**` | `sonar-pro` | Profiles of tools, products, platforms, frameworks. | +| `market-map-profile.md` | `lost-in-public/market-maps/**`, `market-maps/**` | `sonar-deep-research` | Analyst-grade market-map drafts. Dual flavor: Known Category (Quantum Computing, Humanoid Robots) or Thesis-Driven (e.g., Neural Network Hardware as Brains for Robotics) traversing adjacent categories. Lean v1, single-stage. Skips image return by design — deep-research's image metadata is unreliable and market-map imagery is generated separately (Ideogram → frontmatter `banner_image` / `portrait_image` / `square_image`). Multi-stage v2 (RAG pre-flight to inject canonical Lossless tools/concepts as context + Claude editorial pass to emit `[[wikilink]]`s) is deferred — see the User Notes zone of the template for the roadmap. | `source-profile` is the trickiest because `Sources/` is genuinely heterogeneous. The solution is one template, type-conditional content. Books also trigger Google Books URL handling: frontmatter `google_books_url` is used if present, otherwise the model finds it; either way the URL is harvested into frontmatter post-generation via regex, so subsequent runs skip the search. @@ -199,6 +200,34 @@ ## Commands --- +## Per-template timeout override + +The runtime applies a wall-clock timeout to every Perplexity stream via an `AbortController` (see [`directoryTemplateService.ts`](../src/services/directoryTemplateService.ts) `streamPerplexityToFile`). The plugin-level default (*Plugin settings → Directory templates → Request timeout (ms)*) is **30 min** (1,800,000 ms) — generous because most templates are well under that ceiling, and the few that aren't (the analyst-grade deep-research templates) shouldn't be cut off in the middle of a draft worth $10-$50 of analyst time. + +Templates can override the plugin-level default by declaring `request-timeout-ms:` inside their `cft` block: + +```cft +provider: perplexity +model: sonar-deep-research +request-timeout-ms: 2400000 # 40 min +system: | + ... +``` + +Override semantics: + +- A numeric value or a numeric string both work; non-positive / non-numeric values are silently ignored and the plugin-level default is used. +- The override applies only to that template's runs. Other templates continue to use the plugin-level value. +- The runtime tolerates mid-stream truncation gracefully: a stream cut off by timeout leaves a partial draft on disk with `truncated: true` flagged in the result and a Notice telling the user to re-run. + +When to bump above the plugin-level default: + +- `sonar-deep-research` on a multi-section analyst template (market maps, in-depth research reports, sector overviews). These routinely emit 6–8K-word bodies and the tail sections (Frontier, Adjacent Concepts, Open Questions) are the first to be cut. +- Templates with deeply nested skeleton structure and many `[IMAGE N: …]` markers — Perplexity's image-result population time stretches the wall clock. +- Templates that declare a large `include-sources:` block (per the [multi-stage exploration](../context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md)) when that feature ships — RAG context inflation adds prompt-processing time. + +Market-map-profile ships with `request-timeout-ms: 2400000` (40 min) for exactly the reason above. Concept, vocabulary, source, and toolkit profiles do not declare the field and inherit the plugin-level 30-min default — they all finish comfortably under that. + ## Settings `Plugin settings → Directory templates`: diff --git a/main.ts b/main.ts index 920453c..01b66ca 100644 --- a/main.ts +++ b/main.ts @@ -330,7 +330,7 @@ Structure the article as follows: { name: 'image-placement', when: 'return-images' }, ], directoryTemplatesFrontmatterWhitelist: ['title', 'og_description', 'tags', 'og_image'], - directoryTemplatesRequestTimeoutMs: 600000, + directoryTemplatesRequestTimeoutMs: 1800000, // Find images for selection findImagesMaxImages: 3 @@ -2040,9 +2040,9 @@ class PerplexedSettingTab extends PluginSettingTab { new Setting(containerEl) .setName('Request timeout (ms)') - .setDesc('Maximum time to wait for the perplexity deep research response. Default 300000 (5 min).') + .setDesc('Maximum wall-clock time to wait for a Perplexity response. Default 1800000 (30 min) — generous because deep-research runs on long analyst-grade templates routinely take 15-25 min and the $10-$50 of value per good output is worth waiting for. Individual templates may override this per-template via `request-timeout-ms:` in their cft block.') .addText(text => text - .setPlaceholder('300000') + .setPlaceholder('1800000') .setValue(String(this.plugin.settings.directoryTemplatesRequestTimeoutMs)) .onChange(async (value: string) => { const n = parseInt(value, 10); diff --git a/src/docs/templates/README.md b/src/docs/templates/README.md index fc599d5..f9e79c9 100644 --- a/src/docs/templates/README.md +++ b/src/docs/templates/README.md @@ -17,6 +17,7 @@ ## Shipped templates | `vocabulary-profile.md` | `Vocabulary/**` | Definitions of terms with disambiguation through an innovation-consulting lens. | | `source-profile.md` | `Sources/**` | Profiles of trusted sources — books, people, channels, publications, journals, reports, events. Adapts emphasis to the source's type. | | `toolkit-profile.md` | `Tooling/**` | Profiles of tools, products, platforms, frameworks. | +| `market-map-profile.md` | `lost-in-public/market-maps/**`, `market-maps/**` | Analyst-grade market-map drafts — both Known Category (e.g., Humanoid Robots) and Thesis-Driven (e.g., Neural Network Hardware as Brains for Robotics). Runs on `sonar-deep-research`. Single-stage v1; multi-stage RAG + Claude-edit pass is planned. | ## How a template works @@ -59,7 +60,7 @@ # User Notes ### The three zones 1. **Frontmatter** (top, between `---` lines) — carries `title`, `applies-to-paths` (array of glob patterns), and an optional `description`. The plugin uses `applies-to-paths` to match a template to a target file. -2. **`cft` block** (a code fence with language `cft`) — YAML config: `provider`, `model`, `search-recency`, `return-citations`, `return-images`, plus a multi-line `system:` prompt. Anything above the `cft` block is treated as documentation and dropped from the request. +2. **`cft` block** (a code fence with language `cft`) — YAML config: `provider`, `model`, `search-recency`, `return-citations`, `return-images`, optional `request-timeout-ms:` (overrides the plugin-level *Request timeout (ms)* setting for this template — useful for long deep-research runs that need 20-40 min), plus a multi-line `system:` prompt. Anything above the `cft` block is treated as documentation and dropped from the request. 3. **Heading skeleton** (everything between the `cft` block's closing fence and the first `***`) — the user prompt. This is the markdown structure the model fills in. Bullets under each heading are *instructions to the model*, not literal output. The `***` divider terminates the user prompt. Anything below it (the User Notes zone) is for your own scratch work and never reaches the model. diff --git a/src/docs/templates/market-map-profile.md b/src/docs/templates/market-map-profile.md new file mode 100644 index 0000000..422c334 --- /dev/null +++ b/src/docs/templates/market-map-profile.md @@ -0,0 +1,233 @@ +--- +title: Market Map (Analyst Draft) +applies-to-paths: + - "lost-in-public/market-maps/**" + - "market-maps/**" +description: Generates an analyst-grade draft of a market map — either a Known Category (e.g., Humanoid Robots, Light-based Computing) or a Thesis-driven map (e.g., Neural Network Hardware as `Brains` for Robotics) that traverses known categories. +date_created: 2026-05-26 +date_modified: 2026-05-26 +--- + +# About this template + +Use this for files under `lost-in-public/market-maps/` (or any top-level `market-maps/`) whose body is empty or whose curated lead-in (Topics, Lighthouse Examples) has been authored but the analytical body is missing. + +A **market map** is the draft a well-paid analyst would hand to a partner: not an encyclopedia entry, not marketing copy. It explains who is doing what in a category, why now, who funded them, how segments differ, and what the open questions are. Two flavors are supported: + +1. **Known Category** — Humanoid Robots, Light-based Computing, Quantum Computing. The taxonomy is roughly settled; the work is enumerating innovators within established sub-segments and explaining the current frontier. +2. **Thesis-Driven** — "Neural Network Hardware as `Brains` for Robotics." The thesis traverses multiple known categories under a hypothesis. The work is making the thesis legible, then enumerating innovators from each adjacent category that the thesis pulls in. + +The heading skeleton works for both flavors. The model picks up the flavor from the file's `title`, `tags`, and any thesis paragraph the user has pre-authored above the body. + +This template runs on `sonar-deep-research` and skips image embedding by design — deep research returns better citation density and analytical length but unreliable image metadata. Banner / portrait / square imagery for market maps is generated separately (Ideogram) and lives in frontmatter. + +```cft +provider: perplexity +model: sonar-deep-research +return-citations: true +return-images: false +# 40-minute wall-clock budget for this template specifically. Market-map deep +# research runs routinely produce 6-8K-word drafts with 20-40 named innovators +# across 4-8 sub-segments, and the tail of that generation (Frontier and +# Adjacent Concepts sections) is the most cuttable under aggressive timeouts. +# Overrides the plugin-level default. A complete run is worth $10-$50 of +# analyst time; let it cook. +request-timeout-ms: 2400000 +system: | + You are writing the analyst-grade draft of a MARKET MAP titled "{{basename}}". + + A market map is the draft a well-paid analyst hands to a partner. It is not + an encyclopedia entry, not a marketing post, not a listicle. It explains: + + - WHO is doing what in this market — named companies, named founders, named + research labs, with funding stage and primary URL. + - HOW the market segments — what natural sub-buckets exist, and which + innovators sit in which sub-bucket. + - WHY NOW — what changed (technology unlock, regulatory shift, capital + flow, customer behavior) that made this market legible. + - WHAT IS DISPUTED — where credible operators disagree about category + boundaries, winner archetypes, or thesis viability. + + Determine the FLAVOR of map from the title and frontmatter: + + - KNOWN CATEGORY (e.g., "Quantum Computing is Confusing", "Humanoid Robots", + "Agentic AI in Fintech") — the category name is settled. Your job is to + enumerate sub-segments and innovators within each, and explain the frontier. + - THESIS-DRIVEN (e.g., "Neural Network Hardware as Brains for Robotics", + "Blockchain and Web3 Institutional Invasions") — a hypothesis traverses + known categories. Your job is to make the thesis legible, then enumerate + innovators from each adjacent category the thesis pulls in. + + Frontmatter for "{{basename}}": + {{frontmatter}} + + RESEARCH DISCIPLINE: + + - Use Perplexity's web search aggressively. For a market map, breadth of + named entities matters more than depth on any single one. + - For every factual claim — funding round, founding year, customer name, + market sizing, product capability — append an inline numeric citation + marker [1], [2], etc. corresponding to the search-result order. + - Quote phrasing from primary sources where useful (founder interviews, + investor blog posts, earnings notes, academic abstracts). + - Prefer primary surfaces: company homepage, founder Twitter/X, technical + blog posts, conference talks, investor announcements. Aggregator pages + (Crunchbase, PitchBook summaries) are fallbacks for funding stage only. + - Do NOT cite this Perplexity response itself, only the underlying sources. + + EDITORIAL STANCE — attribute innovation correctly: + + Markets are pioneered by startups, academics, research labs, and indie + practitioners — NOT by tech giants. Training data over-represents + incumbents. Counteract this systematically: + + - Treat big tech (Microsoft, Google, Amazon, Apple, Meta, Oracle, Salesforce, + IBM post-1990s, Nvidia post-2020) as ADOPTERS or POPULARIZERS in this + market unless the entry documents an originating research-lab paper or + a heyday-era origination story (Bell Labs, Xerox PARC, DeepMind, OpenAI's + early years). + - In every Lighthouse Examples sub-bucket, cap big-tech entries at 1 of + 5–10. Prefer Series A-C startups, seed-stage frontier bets, + open-source projects, indie practitioners, research labs. + - In Market Dynamics, name the named operators driving the curve, not + the incumbents profiting from it. + - Where an incumbent IS the originator, say so explicitly with the + research paper or product release that documents the origination. + + INNOVATOR CARD SHAPE: + + Each named innovator under "Innovator Profiles" follows this shape exactly: + + #### [Innovator Name](https://homepage.url) + **Offering**: one-sentence description of what they do that is specific + enough that a partner could repeat it back. Use product names, customer + names, and category boundaries. Cite. [N] + **Funding**: stage and round size if disclosed (e.g., "$30M Series A, + DN Capital, 2024"). "Undisclosed" or "Bootstrapped" if no public data. + **Why they matter**: one sentence on what makes them distinctive — the + technical bet, the GTM angle, the team's background. Not marketing + adjectives; a specific differentiator. Cite. [N] + **Coverage**: 1-2 references in trade press, founder podcasts, or + analyst notes if available. Format: `[Outlet, Title](url)`. Cite. [N] + + Aim for 3-7 innovators per sub-bucket. If a sub-bucket has fewer than 3 + credible named entities, MERGE it into an adjacent sub-bucket rather than + padding with weak entries. + + LINKS AND WIKILINKS: + + - For innovator names and source links, use `[Name](https://url)` form. + - Do NOT invent `[[wikilink]]` syntax. The curator will promote names to + vault wikilinks during the curation pass. If you happen to know a + canonical concept this market touches (e.g., "Compliance AI", "Agentic + Workspaces"), surface the concept name as plain text in the Adjacent + Concepts section so the curator can wikilink it later. + + CALIBRATION ON LENGTH: + + This is a deep-research run. Lean long, not short. A complete market map + is roughly 4,000-8,000 words of body, with 20-40 named innovators across + 4-8 sub-buckets, a summary table per sub-bucket where useful, and explicit + funding-trend / adoption-pattern data in Market Dynamics. Better to over- + enumerate and let the curator prune than to under-enumerate. +``` + +# Market Snapshot + +- One-paragraph italicized lede (max 2 sentences) that captures the punchy thesis of this market. Voice: the analyst opening their memo. Use markdown italics: `_..._`. +- Then the headline stat — one quoted statistic from a credible source that signals scale or velocity, with inline citation. Format the quote as a blockquote (`> "..."`). +- Then 2-3 sentences orienting the reader: what is this market, what is the timeframe, why is it worth a map right now. + +# The Question this Map Answers + +- One paragraph (3-5 sentences) stating the question this map clarifies for an operator or investor reading it. +- If the file is a KNOWN CATEGORY map, frame the question as "what shape has this category settled into, and where is the frontier." +- If the file is a THESIS-DRIVEN map, state the thesis explicitly in one sentence, then explain which adjacent categories the thesis traverses and why the traversal is non-obvious. + +# Why Now + +- 3-5 bullets, each one a specific unlock that explains why this market is mappable in the current quarter and would have been premature 18 months ago. +- Unlock types to consider: a technical capability crossing a threshold (cost, latency, accuracy), a regulatory or standards shift, a capital-formation pattern (a fund vintage, an exit precedent), a customer-behavior shift, an open-source release that lowered the floor. +- Cite each unlock. Where possible, quote a founder, researcher, or operator who named the unlock. + +# Map of the Market — Sub-Segments + +- Identify 4-8 natural sub-segments that partition this market. For a thesis-driven map, the sub-segments are the adjacent categories the thesis traverses. +- Give each sub-segment a 1-2 sentence definition: what falls inside, what falls outside, and what distinguishes it from its neighbors. +- This section is the TABLE OF CONTENTS for the Innovator Profiles section below. Sub-segment names here must match section headings below. + +# Lighthouse Examples + +- For each sub-segment, list 5-10 lighthouse innovators in `[Name](url) — one-line description` form. These are the names a partner would expect to hear in this category — recognized leaders, well-funded operators, frontier bets the analyst would brief on. +- Group under `## ` subheadings matching the Map of the Market above. +- This is a flat reference list. The deeper analysis lives in Innovator Profiles below. +- Use the editorial-stance cap: at most 1 of 5-10 in any sub-bucket may be big tech. + +# Innovator Profiles + +For each sub-segment from the Map of the Market, produce a `## ` heading and under it, 3-7 innovator cards in the format defined in the system prompt: + + #### [Innovator Name](https://homepage.url) + **Offering**: ... + **Funding**: ... + **Why they matter**: ... + **Coverage**: ... + +- Order within each sub-segment from most-established to most-frontier (seed-stage / stealth at the end). +- After the innovator cards in each sub-segment, render a single summary table with columns: `Innovator | Stage | Differentiator | Primary Customer`. The table lets a partner skim the sub-segment without reading every card. + +# Media, Voices, and Coverage + +- 6-12 bullets covering the publications, podcasts, YouTubers, analysts, and individual operator-thinkers who shape the conversation about this market. +- Format: `**Name** — Platform — one-line note on their angle / why they are worth following`. Include a primary URL link. +- Sub-group with `## Publications`, `## Podcasts & YouTube`, `## Analysts & Operator-Thinkers` if the list is long enough to warrant it. +- Prefer specialized trade press over generalist business press. Prefer named operator-bloggers over corporate marketing surfaces. + +# Market Dynamics + +## Sizing and Growth + +- 2-4 cited bullets covering: TAM / current market size, projected CAGR, the report or analyst behind each number. +- Be skeptical of single-source sizing claims; where two credible sources disagree, surface the disagreement. + +## Adoption Patterns and Barriers + +- 2-4 cited bullets covering: what percentage of the addressable buyer base has adopted, what the canonical barriers are (procurement cycle, regulatory uncertainty, technical readiness, talent shortage), and where the adoption curve is bending. + +## Capital Flow + +- 2-4 cited bullets covering: where the funding has concentrated by sub-segment, who the active funds are (named partners where public), and any recent exit, acquisition, or IPO that reset valuation expectations in the category. + +# Frontier and Open Questions + +- 4-7 bullets, each one a specific open question that a partner reading this map would want the analyst to think about next. +- Frame each as a question, not a statement: "Will agent-to-agent micropayments require a separate settlement rail, or will existing card networks absorb the use case?" not "Settlement rails are evolving." +- Pair each question with a one-sentence note on which innovators or research streams are likely to produce the answer. + +# Adjacent Concepts and Maps + +- 4-8 plain-text concept names (no wikilink syntax — the curator wikilinks during curation) that an operator working in this market would want to explore next. +- Mix of: adjacent market maps (other categories this one borders), foundational concepts (the mental models this market sits on), and vocabulary terms (the specific jargon the curator may want to define in `Vocabulary/`). +- Format: `- — one-line on why it adjoins this map`. + +*** + +# User Notes + +Anything below the `***` line is excluded from the request. Use this zone for: + +- The thesis paragraph (for thesis-driven maps) you want the model to fold into the system context. To do that, move the paragraph ABOVE the `***` divider — into the Market Snapshot or Question section — before running the template. +- Hand-curated `:::tool-showcase` blocks pointing to vault tools you want to feature. +- Tuning notes, prior model outputs, and iteration history while you refine the template for your domain. + +## Multi-stage roadmap (deferred) + +This v1 template is intentionally single-stage: one Perplexity Deep Research run produces the full draft, the curator promotes Lighthouse Examples to `:::tool-showcase` blocks and `[Name](url)` references to `[[wikilink]]` references during the curation pass. + +The deferred multi-stage version will run: + +1. **RAG pre-flight** — pull canonical Lossless sources for this market (tools under `Tooling/`, concepts under `concepts/`, prior market maps that overlap) and feed them as context to the research stage. This eliminates the wikilink-invention problem and lets the model name actual vault entries. +2. **Perplexity research stage** — deep research with the RAG context as a primer, producing the draft this v1 template produces. +3. **Claude editing stage** — an editorial pass that enforces the analyst voice, prunes the over-enumeration, restructures sub-segment boundaries where the data demands, and emits the final `[[wikilink]]` form. + +The plumbing for stage 3 (the Claude orchestrator) lives in `claudeService.ts`. The plumbing for stage 1 (RAG over the Lossless corpus) is partially built via the `chroma` MCP server — the missing piece is a per-template `rag-context:` block in the cft fence that names which collections to query and how many results to inject. Spec lives in `context-v/specs/` — open it before starting the multi-stage build. diff --git a/src/services/directoryTemplateService.ts b/src/services/directoryTemplateService.ts index d8613bf..4fc975a 100644 --- a/src/services/directoryTemplateService.ts +++ b/src/services/directoryTemplateService.ts @@ -775,12 +775,27 @@ export async function applyTemplate( // Set initial state before streaming begins. await app.vault.modify(target, initialContent); + // Template-level `request-timeout-ms:` in the cft block overrides the + // plugin-level default. Deep-research templates routinely need 20-40min + // to fully render a long analyst draft and shouldn't be capped by a + // setting tuned for short concept-profile runs. Accept number or + // numeric string. Ignore non-positive / non-numeric values silently. + const cftTimeoutRaw = template.cftConfig['request-timeout-ms']; + const cftTimeoutMs = typeof cftTimeoutRaw === 'number' + ? cftTimeoutRaw + : typeof cftTimeoutRaw === 'string' + ? parseInt(cftTimeoutRaw, 10) + : NaN; + const effectiveTimeoutMs = Number.isFinite(cftTimeoutMs) && cftTimeoutMs > 0 + ? cftTimeoutMs + : settings.requestTimeoutMs; + const { streamed, sources, images, truncated } = await streamPerplexityToFile( app, settings.perplexityApiKey, settings.perplexityEndpoint, payload, - settings.requestTimeoutMs, + effectiveTimeoutMs, target, initialContent, isCancelled, diff --git a/src/services/templateSeederService.ts b/src/services/templateSeederService.ts index feebe34..8272f14 100644 --- a/src/services/templateSeederService.ts +++ b/src/services/templateSeederService.ts @@ -6,6 +6,7 @@ import conceptProfile from '../docs/templates/concept-profile.md'; import vocabularyProfile from '../docs/templates/vocabulary-profile.md'; import sourceProfile from '../docs/templates/source-profile.md'; import toolkitProfile from '../docs/templates/toolkit-profile.md'; +import marketMapProfile from '../docs/templates/market-map-profile.md'; import partialsReadme from '../docs/partials/README.md'; import mermaidDisciplinePartial from '../docs/partials/mermaid-discipline.md'; @@ -30,6 +31,7 @@ const TEMPLATE_FILES: SeedFile[] = [ { name: 'vocabulary-profile.md', content: vocabularyProfile }, { name: 'source-profile.md', content: sourceProfile }, { name: 'toolkit-profile.md', content: toolkitProfile }, + { name: 'market-map-profile.md', content: marketMapProfile }, ]; const PARTIALS_README: SeedFile = { name: 'README.md', content: partialsReadme };