mirror of
https://github.com/lossless-group/perplexed-plugin.git
synced 2026-07-22 06:49:50 +00:00
feat(directory-templates): market-map analyst profile + per-template request-timeout-ms cft override
A new shipped template aimed at analyst-grade market maps (Known Category and Thesis-Driven flavors), and the timeout refactor that came out of running it against a real 7-8K-word draft and watching the wall-clock AbortController truncate the tail mid-sentence. The two ship together because the template surfaced the bug — running market-map-profile on lost-in-public/market-maps/Humanoid Robots and their Input Industries.md was the first observable instance of the wall-clock pathology — and they're both groundwork for the multi-stage Claude+Perplexity+RAG pipeline the exploration doc now sketches.
Impact: a fifth shipped template auto-seeded into Content-Dev/Templates/ on first plugin load (and re-seedable from settings), running on sonar-deep-research with return-images off by design; templates can override the plugin-level wall-clock timeout per-template via a new cft-block key; the plugin-level default jumped from 10 min to 30 min because the old default was tuned for short concept-profile runs and was cutting off any deep-research template that took its time. Existing four templates are unchanged in behavior — they don't declare the override and inherit the new 30-min default, which they all finish comfortably under.
Market-map template (src/docs/templates/market-map-profile.md):
- Heading skeleton: Market Snapshot, The Question this Map Answers, Why Now, Map of the Market, Lighthouse Examples, Innovator Profiles (Link/Offering/Funding/Why/Coverage cards + per-sub-segment summary table), Media-Voices-Coverage, Market Dynamics (Sizing, Adoption, Capital Flow), Frontier and Open Questions, Adjacent Concepts.
- Dual-flavor: the system prompt has the model pick between Known Category (Humanoid Robots, Quantum Computing) and Thesis-Driven (e.g. Neural Network Hardware as Brains for Robotics) based on title and tags.
- Anti-incumbent editorial stance copied from the concept template family — cap big tech at 1 of 5-10 in every sub-bucket, attribute innovation to startups/labs/founders.
- Declares request-timeout-ms: 2400000 (40 min) with an inline comment explaining the budget — the only shipped template that overrides the plugin default today.
- Registered in templateSeederService.ts so first-load seeding and the Re-seed button both pick it up; surfaced in src/docs/templates/README.md and docs/directory-templates.md.
Timeout refactor (src/services/directoryTemplateService.ts, main.ts):
- Per-template cft-block key request-timeout-ms: accepts number or numeric string, silently falls back on non-positive / non-numeric values, otherwise wins over the plugin-level setting for that template's runs. Resolution code sits just before the streamPerplexityToFile call so the rest of the streaming primitive is untouched.
- Plugin default directoryTemplatesRequestTimeoutMs bumped 600000 → 1800000 (10 min → 30 min). Settings-pane description rewritten to call out the override and the cost framing ("$10-$50 of analyst time per good output is worth waiting for").
- Structural fix (idle-timeout discipline ported from perplexityService.ts:659-668, where the legacy modal flow already does it correctly) is filed as its own open issue, not in scope for this commit.
Context-v writing:
- context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md — three architectures weighed (zoned single-file appends, multi-doc folder per market map, per-section cft-section blocks), an include-sources YAML schema proposed covering vault paths/globs and Chroma queries, and the "Perplexity and Claude do not overwrite each other until editing" heuristic translated into three implementation strictness levels. Recommends Option A (zoned appends) for v1.
- context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md — the Humanoid Robots truncation as motivating symptom, side-by-side diagnosis of the two timeout disciplines in the codebase (wall-clock here, per-chunk idle in the legacy modal), the partial fix shipped today, why it's only partial, the structural-fix proposal, and four pre-spec open items. Cross-linked both ways to the exploration so the issue is reachable from either entry point.
Files changed:
- src/docs/templates/market-map-profile.md (new)
- src/services/templateSeederService.ts
- src/services/directoryTemplateService.ts
- main.ts
- src/docs/templates/README.md
- docs/directory-templates.md
- context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md (new)
- context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md (new)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
mpstaton
parent
279c954143
commit
96df70595f
9 changed files with 954 additions and 7 deletions
202
changelog/2026-05-26_01.md
Normal file
202
changelog/2026-05-26_01.md
Normal file
|
|
@ -0,0 +1,202 @@
|
|||
---
|
||||
title: "Market-map analyst template + the per-template timeout override the long deep-research it needs surfaced as load-bearing"
|
||||
lede: "A fifth shipped directory template aimed at analyst-grade market maps (Known Category and Thesis-Driven flavors) — the kind of document a partner pays a $150K analyst three days to produce."
|
||||
date_work_started: 2026-05-26
|
||||
date_work_completed: 2026-05-26
|
||||
date_created: 2026-05-26
|
||||
date_modified: 2026-05-26
|
||||
publish: true
|
||||
category: Changelog
|
||||
tags:
|
||||
- Directory-Templates
|
||||
- Market-Maps
|
||||
- Deep-Research
|
||||
- Streaming-Timeouts
|
||||
- cft-Block
|
||||
- Perplexity-sonar-deep-research
|
||||
- Multi-Stage-Generation
|
||||
authors:
|
||||
- Michael Staton
|
||||
augmented_with:
|
||||
- Claude Code on Claude Opus 4.7 (1M context)
|
||||
files_changed:
|
||||
- src/docs/templates/market-map-profile.md
|
||||
- src/services/templateSeederService.ts
|
||||
- src/services/directoryTemplateService.ts
|
||||
- src/docs/templates/README.md
|
||||
- docs/directory-templates.md
|
||||
- main.ts
|
||||
- context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md
|
||||
- context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md
|
||||
---
|
||||
|
||||
# A template for analyst-grade market maps, and the timeout the work it produces actually needs
|
||||
|
||||
## Why care?
|
||||
|
||||
If you write market maps — the kind of document where you enumerate every credible operator in a category, segment them, name the funders, surface the open questions, and hand it to a partner — Perplexed now ships a directory template that produces a respectable analyst draft in one pass. Drop a file into `lost-in-public/market-maps/` (or any top-level `market-maps/`), give it a title and a one-line lede, run *Apply directory template to current file*, and you get back a 6-8K-word draft on `sonar-deep-research` with named innovators, per-segment funding tables, media coverage, market dynamics, and a frontier-questions section. Then you spend your time on the curation pass — promoting `[Name](url)` references to `[[wikilink]]`s back into your vault, layering in your `:::tool-showcase` blocks, tightening the analyst voice — instead of on the enumeration grind. The framing is straightforward: an analyst-day at $150K-a-year rates is $600 of someone's time, the API run costs a few dollars, and the rest is your judgment doing what your judgment does best.
|
||||
|
||||
If you author *any* directory template that runs on a deep-research model, the timeout refactor is the more load-bearing change. The plugin's wall-clock cap before today was 10 minutes — perfectly reasonable for `concept-profile` and `vocabulary-profile`, fatal for anything that legitimately needs 20-30 minutes to produce its long analytical body. Today's market-map run on *Humanoid Robots and their Input Industries* was the first observable instance: ~7,500 words of body produced, then a hard stop mid-sentence in the *Frontier and Open Questions* section as the `AbortController` fired at the ceiling. The fix: per-template `request-timeout-ms:` cft-block override, plugin-level default bumped to 30 min, and an open issue filed for the structural fix (port the idle-timeout discipline the legacy modal flow already uses).
|
||||
|
||||
## What's new?
|
||||
|
||||
Three shipped behaviors:
|
||||
|
||||
```
|
||||
zz-cf-lib/
|
||||
├── templates/
|
||||
│ ├── concept-profile.md
|
||||
│ ├── vocabulary-profile.md
|
||||
│ ├── source-profile.md
|
||||
│ ├── toolkit-profile.md
|
||||
│ └── market-map-profile.md (new — runs on sonar-deep-research)
|
||||
├── partials/ (existing — referenced via {{include: name}})
|
||||
└── preambles/ (existing — auto-attached to every request)
|
||||
```
|
||||
|
||||
- **`market-map-profile.md`** is auto-seeded into `Content-Dev/Templates/` on first plugin load and re-seedable from settings via the *Re-seed templates* button. It matches `lost-in-public/market-maps/**` and `market-maps/**` so both your `lost-in-public/`-organized vault and a top-level `market-maps/` folder work.
|
||||
- **`request-timeout-ms:` cft-block override** lets a template declare its own wall-clock budget that wins over the plugin-level setting for that template's runs. `market-map-profile.md` declares `2400000` (40 min); the other four don't declare it and inherit the new default.
|
||||
- **Plugin-level default bumped 600000 → 1800000** (10 min → 30 min). Settings-pane description rewritten to call out the override and the cost framing ("$10-$50 of analyst time per good output is worth waiting for").
|
||||
|
||||
Two context-v artifacts captured at the same time, because the market-map work raised both as the next-natural moves:
|
||||
|
||||
- **`context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md`** — opens the explorations folder for this submodule. Weighs three architectures for evolving the lean v1 template into a cooperative pipeline (zoned single-file appends, multi-doc folder per market map, per-section `cft-section` blocks); proposes an `include-sources:` YAML schema covering vault paths/globs and Chroma queries; translates the "Perplexity and Claude do not overwrite each other until editing" framing into three implementation strictness levels. Recommends Option A (zoned appends) for v1.
|
||||
- **`context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md`** — the Humanoid Robots truncation as motivating symptom, side-by-side diagnosis of the two timeout disciplines in this codebase (wall-clock here, per-chunk idle in the legacy modal), the partial fix shipped today, why it's only partial, and the structural-fix proposal.
|
||||
|
||||
## How the market-map template thinks
|
||||
|
||||
A market map is the draft a well-paid analyst hands to a partner. It is not an encyclopedia entry, not a marketing post, not a listicle. The template's system prompt commits to this framing explicitly and asks the model to determine the *flavor* of the map from the title and frontmatter:
|
||||
|
||||
- **Known Category** — Humanoid Robots, Quantum Computing, Agentic AI in Fintech. The taxonomy is roughly settled; the work is enumerating innovators within established sub-segments and explaining the frontier.
|
||||
- **Thesis-Driven** — Neural Network Hardware as Brains for Robotics, Blockchain and Web3 Institutional Invasions. A hypothesis traverses multiple known categories. The work is making the thesis legible, then enumerating innovators from each adjacent category the thesis pulls in.
|
||||
|
||||
The skeleton works for both flavors. Sections (in order):
|
||||
|
||||
1. **Market Snapshot** — italicized lede + headline-stat blockquote + orienting paragraph.
|
||||
2. **The Question this Map Answers** — what this map clarifies for an operator or investor reading it.
|
||||
3. **Why Now** — 3-5 cited unlocks (technical thresholds, regulatory shifts, capital patterns, customer-behavior shifts) that make the market mappable in the current quarter.
|
||||
4. **Map of the Market — Sub-Segments** — 4-8 sub-segments with 1-2 sentence definitions. Doubles as the TOC for the next section.
|
||||
5. **Lighthouse Examples** — 5-10 named operators per sub-segment in `[Name](url) — one-line` form, flat reference list.
|
||||
6. **Innovator Profiles** — 3-7 deeper cards per sub-segment in a strict Link / Offering / Funding / Why-they-matter / Coverage shape, followed by a per-sub-segment summary table.
|
||||
7. **Media, Voices, and Coverage** — publications, podcasts, YouTubers, analysts, operator-thinkers.
|
||||
8. **Market Dynamics** — Sizing & Growth, Adoption Patterns & Barriers, Capital Flow.
|
||||
9. **Frontier and Open Questions** — 4-7 specific questions phrased as questions.
|
||||
10. **Adjacent Concepts and Maps** — plain-text concept names (no wikilinks invented; the curator wikilinks during curation).
|
||||
|
||||
The anti-incumbent editorial stance from the concept-profile family carries over: big tech is treated as adopters/popularizers, not innovators, unless a research-lab paper or heyday-era origination supports otherwise. Cap big tech at 1 of 5-10 in every sub-bucket. The market-map prompt phrases this in terms specific to category mapping — "name the named operators driving the curve, not the incumbents profiting from it."
|
||||
|
||||
Image-return is off by design for this template. `sonar-deep-research` returns better citation density and analytical length than `sonar-pro` but its image metadata is unreliable; market-map imagery (banner, portrait, square) is generated separately via Ideogram and lives in frontmatter, so the per-section image markers the other templates use aren't load-bearing here.
|
||||
|
||||
## How the timeout override works
|
||||
|
||||
The directory-template flow has always armed a single wall-clock `setTimeout` at fetch time, racing the entire stream against `settings.requestTimeoutMs`:
|
||||
|
||||
```ts
|
||||
const controller = new AbortController();
|
||||
const timer = activeWindow.setTimeout(() => controller.abort(), timeoutMs);
|
||||
```
|
||||
|
||||
This worked fine when the entire suite of shipped templates ran on `sonar-pro` and finished in 2-3 minutes. It doesn't work when a single template legitimately needs 25-30 minutes. The naïve fix would have been to bump `settings.requestTimeoutMs` to something generous for everyone, but that punishes the short-template case (a 90-second concept run that stalls silently shouldn't take 30 minutes to surface). So we put the dial where the cft already lives — inside the template itself:
|
||||
|
||||
```yaml
|
||||
provider: perplexity
|
||||
model: sonar-deep-research
|
||||
return-citations: true
|
||||
return-images: false
|
||||
request-timeout-ms: 2400000 # 40 minutes for this template specifically
|
||||
system: |
|
||||
...
|
||||
```
|
||||
|
||||
Resolution sits in `directoryTemplateService.ts` just before the `streamPerplexityToFile` call:
|
||||
|
||||
```ts
|
||||
const cftTimeoutRaw = template.cftConfig['request-timeout-ms'];
|
||||
const cftTimeoutMs = typeof cftTimeoutRaw === 'number'
|
||||
? cftTimeoutRaw
|
||||
: typeof cftTimeoutRaw === 'string'
|
||||
? parseInt(cftTimeoutRaw, 10)
|
||||
: NaN;
|
||||
const effectiveTimeoutMs = Number.isFinite(cftTimeoutMs) && cftTimeoutMs > 0
|
||||
? cftTimeoutMs
|
||||
: settings.requestTimeoutMs;
|
||||
```
|
||||
|
||||
Number or numeric-string both work. Non-positive / non-numeric values are silently ignored and the plugin-level default takes over — the template author can't accidentally lock themselves out of the run with a typo.
|
||||
|
||||
Plugin-level default bumped because the old 10-minute ceiling was clearly tuned for the original four templates and not for any subsequent deep-research template anyone might write. 30 minutes is the conservative new default: comfortably above what any of the existing four templates need, generous enough that most new deep-research templates won't need to override, and visible-enough in the settings pane that a user who hits a longer-running case can find the dial.
|
||||
|
||||
## The bug that surfaced this — Humanoid Robots, line 400
|
||||
|
||||
The first real-world run of `market-map-profile` was `lost-in-public/market-maps/Humanoid Robots and their Input Industries.md`. The model produced a draft that looked respectable through nine of the ten skeleton sections, then terminated mid-sentence inside the tenth:
|
||||
|
||||
```
|
||||
Will the evolution of robot-as-a-service (R
|
||||
```
|
||||
|
||||
That trailing `(R` is the last byte written to the file. The *Adjacent Concepts and Maps* heading never appeared. Frontmatter stamps (`cf_last_run`, `cf_last_run_model`) landed correctly — the run did make it to its `processFrontMatter` post-step — so what was lost was streamed body content that hadn't arrived when the `AbortController` fired. Classic wall-clock cut-off shape.
|
||||
|
||||
The directory-template flow uses a single wall-clock timeout. The *legacy* modal flow (`perplexityService.ts:659-668`) has used a per-chunk idle-timeout for several iterations:
|
||||
|
||||
```ts
|
||||
const STREAM_IDLE_TIMEOUT_MS = isDeepResearch ? 270_000 : 90_000;
|
||||
const readWithIdleTimeout = (): Promise<...> => {
|
||||
let timer: number | undefined;
|
||||
const timeout = new Promise<never>((_, reject) => {
|
||||
timer = window.setTimeout(() => {
|
||||
reject(new Error(`stream went idle for ${...}s ...`));
|
||||
}, STREAM_IDLE_TIMEOUT_MS);
|
||||
});
|
||||
return Promise.race([reader.read(), timeout]).finally(() => {
|
||||
if (timer !== undefined) window.clearTimeout(timer);
|
||||
});
|
||||
};
|
||||
```
|
||||
|
||||
The idle-timeout pattern is structurally correct. Per `reader.read()`, a fresh timeout is armed; as long as bytes keep arriving, the timer keeps getting cleared and re-armed. The stream is only killed if it goes *quiet* for N seconds. Slow-but-healthy completes; stalled-but-silent fails fast.
|
||||
|
||||
We didn't port the idle-timeout pattern into the directory-template flow today, because the structural refactor warrants its own design and its own commit, not a side-quest in a template-shipping changelog. Today's fix is the pressure-relief valve: a bigger ceiling, plus a per-template dial. The structural fix is filed at `context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md` with a 4-step concrete plan, a key-rename proposal (`stream-idle-timeout-ms:`), and a backstop-or-no-backstop open question.
|
||||
|
||||
## Under the hood — the things that earned their own design decisions
|
||||
|
||||
**Both flavors with one template, not two.** The Known Category vs Thesis-Driven distinction is real, but it lives in the *content* of the map, not its *structure*. A Known Category map and a Thesis-Driven map both want sub-segments, both want innovator profiles, both want the same frontier-questions framing. The system prompt names the flavor difference once and asks the model to pick from `{{title}}` and `{{frontmatter}}` signals. Two templates would have produced more drift than value.
|
||||
|
||||
**No invented wikilinks.** The model doesn't know what's in your vault. If it tries to emit `[[Tooling/AI-Toolkit/Agentic AI/Crew AI]]` from training data, it'll get the path wrong half the time and the basename wrong the rest of the time. So the prompt is explicit: use `[Name](url)` form for every entity, leave wikilink promotion to the curator. The future multi-stage pipeline (per the exploration doc) will inject canonical vault content via an `include-sources:` block at request time and authorize the model to emit `[[wikilink]]`s for the entries it's been shown — but that's v2.
|
||||
|
||||
**`return-images: false` because deep-research's image metadata isn't reliable enough.** This is a documented limit on `sonar-deep-research` we've eaten before in the concept-profile changelog (2026-05-12). Market-map imagery doesn't lean on inline images anyway — `banner_image`, `portrait_image`, `square_image` in frontmatter is the canonical surface, generated separately via Ideogram.
|
||||
|
||||
**The override key is named `request-timeout-ms:`, not `stream-idle-timeout-ms:`.** That'll likely change when the structural fix lands and the underlying primitive moves from wall-clock to idle-timeout. We chose the misnomer-tolerant present-day name on purpose: the cft schema can rename keys with a compatibility alias in one release, and naming the new key for what it actually does *today* keeps the surface honest until the underlying mechanism changes.
|
||||
|
||||
**Length not trimmed.** The market-map template's system prompt is over 1,000 words of guidance. The skeleton with bullet instructions is another 1,000+. A first instinct was to trim — most prompts of that length are over-engineered — but every section of the template's prompt is doing real work: the editorial stance, the innovator-card shape, the dual-flavor detection, the wikilink discipline, the calibration on length ("lean long, not short"). We left it long. The cost is a longer first-time-readable system prompt for vault users; the benefit is fewer regenerations and a draft that's closer to a partner-ready document.
|
||||
|
||||
## Files touched
|
||||
|
||||
```
|
||||
plugin-modules/perplexed/
|
||||
├── src/docs/templates/
|
||||
│ ├── market-map-profile.md (new)
|
||||
│ └── README.md (cft-block-keys list now mentions request-timeout-ms)
|
||||
├── src/services/
|
||||
│ ├── directoryTemplateService.ts (request-timeout-ms override resolution)
|
||||
│ └── templateSeederService.ts (market-map-profile registered as 5th seeded template)
|
||||
├── docs/
|
||||
│ └── directory-templates.md (shipped-templates table now has market-map row;
|
||||
│ new "Per-template timeout override" section)
|
||||
├── context-v/
|
||||
│ ├── explorations/ (new folder; opens with the multi-stage doc)
|
||||
│ │ └── Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md
|
||||
│ └── issues/
|
||||
│ └── Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md
|
||||
└── main.ts
|
||||
├── directoryTemplatesRequestTimeoutMs default: 600000 → 1800000
|
||||
└── settings-pane description rewritten with the override + cost framing
|
||||
```
|
||||
|
||||
Pointer-bumped at the parent: `content-farm` commit `e31b982` folds in the perplexed pointer (`a44f23b`) and a 3-line drag-along addition to `context-v/plans/Create-a-Study-of-the-Best-Obsidian-Plugins.md` (`neural-composer` by oscampo flagged as a study-candidate plugin).
|
||||
|
||||
## What's next
|
||||
|
||||
- **Run the rebuilt plugin against Humanoid Robots in append mode** to extend the truncated tail. With the 40-minute ceiling on the market-map template, the *Frontier and Open Questions* section should finish and *Adjacent Concepts and Maps* should land for the first time.
|
||||
- **The structural idle-timeout fix.** The wall-clock primitive in `directoryTemplateService.ts:511` has a known better replacement living in the same codebase (`perplexityService.ts:659-668`). The plan and the open design questions live at `context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md`. Worth doing before the multi-stage spec lands, because stage 3 (Claude editorial on a 7-8K-word draft) will hit the same pathology as stage 2 (Perplexity research).
|
||||
- **The multi-stage Claude+Perplexity+RAG spec.** The exploration doc commits to a tentative direction (Option A — zoned single-file appends with `include-sources:` and `editor:` keys added to the cft schema). The next move is a spec in `context-v/specs/` pinning down the schema additions, the zone-delimiter format, the resolution order for `include-sources`, and the migration path for the existing four templates. Then a chunked prompt in `context-v/plans/`. Then `market-map-profile` becomes the reference implementation.
|
||||
- **The anti-incumbent editorial stance is now duplicated across four templates.** `concept-profile`, `vocabulary-profile`, `source-profile`, and now `market-map-profile` all carry a variation. Per the partials/preambles work that landed 2026-05-19, this is the next obvious extraction — `partials/editorial-stance-anti-incumbent.md` referenced via `{{include: editorial-stance-anti-incumbent}}`. Not done today; flagged.
|
||||
|
|
@ -0,0 +1,346 @@
|
|||
---
|
||||
title: "Multi-Stage Cooperative Claude + Perplexity with RAG"
|
||||
lede: "What if a market-map draft were the output of three agents — a RAG pre-flight that names what to include, a Perplexity research run that finds the rest, and a Claude editorial pass that never steps on either — instead of one single-shot prompt?"
|
||||
date_created: 2026-05-26
|
||||
date_modified: 2026-05-26
|
||||
authors:
|
||||
- Michael Staton
|
||||
augmented_with:
|
||||
- Claude Opus 4.7 (1M context)
|
||||
semantic_version: 0.0.0.1
|
||||
tags:
|
||||
- Exploration
|
||||
- Perplexed
|
||||
- Multi-Stage-Generation
|
||||
- Agentic-Orchestration
|
||||
- RAG
|
||||
- Chroma
|
||||
- Market-Maps
|
||||
- Claude
|
||||
- Perplexity-Deep-Research
|
||||
status: Open
|
||||
related:
|
||||
- "[[market-map-profile]]"
|
||||
- "[[Using-Files-as-Prompt-Outlines]]"
|
||||
- "[[Partials-And-Preambles-For-Perplexed-Templates]]"
|
||||
- "[[Getting-Claude-to-Respond-With-Research]]"
|
||||
- "[[Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams]]"
|
||||
---
|
||||
|
||||
# Multi-Stage Cooperative Claude + Perplexity with RAG
|
||||
|
||||
A market map is the kind of document where one-shot generation hits its limit. The lean v1 `market-map-profile` template ([`src/docs/templates/market-map-profile.md`](../../src/docs/templates/market-map-profile.md)) produces a respectable analyst draft in a single Perplexity deep-research run, but every analyst who has read one of our existing market maps notices the same gaps: the wikilinks back to our own `Tooling/` and `concepts/` aren't there, the segmentation occasionally misses a sub-bucket we cover heavily in the vault, and the editorial voice drifts from "well-paid analyst" toward "encyclopedia summary" without a second pass. The fix is multi-stage, but multi-stage has more design surface than it looks: who runs first, how they hand off, where the RAG context lives, and — the question that motivated this doc — **how do we keep Claude and Perplexity from overwriting each other's work?**
|
||||
|
||||
## The question
|
||||
|
||||
We want to evolve `market-map-profile` (and, by precedent, any other research-heavy template) from a one-shot Perplexity run into a cooperative pipeline that includes:
|
||||
|
||||
1. A **RAG pre-flight** that pulls canonical Lossless source material (vault tools, concepts, prior market maps; Chroma-indexed corpus chunks) and bakes it into the prompt context as named, citeable inputs.
|
||||
2. A **Perplexity research stage** that does what `sonar-deep-research` does today, but primed with the RAG context so it cites our own canonical entries by name rather than generating shadow versions of them.
|
||||
3. A **Claude editorial stage** that enforces analyst voice, prunes over-enumeration, restructures sub-segments where the data demands, emits `[[wikilink]]`s for entries that exist in the vault, and surfaces "the question this map answers" framing.
|
||||
|
||||
The specific design questions:
|
||||
|
||||
- **How do RAG-provided sources get expressed in a template?** The user articulated this as an `include_sources:` argument — a list of sources to include, distinct from `search-domains:` (which constrains where Perplexity searches the open web). What's the surface shape — a list of file paths, a Chroma query spec, both?
|
||||
- **How do the three stages share a target file without clobbering each other?** Perplexity streams text into the file body. Claude's editorial pass needs to *edit* that text. If Claude runs while Perplexity is still streaming, or if Claude edits the same region Perplexity wrote, we get a race or a confused output.
|
||||
- **Single file or multi-document folder?** A market map could remain one large markdown file with sectioned output, or it could become a folder where each sub-segment is its own document with a manifest at the root. The latter is more agentic-friendly (each section is independently re-runnable) but breaks the single-file mental model that every other directory template uses.
|
||||
- **Where does the RAG index live, and how does it stay fresh?** The Chroma collections in [`lossless-monorepo/CLAUDE.md`](../../../../CLAUDE.md) (`context-vigilance-corpus`, `lossless-changelog`, `claude-code-sessions`, `claude-code-tool-traces`) are reachable via the `chroma` MCP server inside Claude Code, but the perplexed plugin runs inside Obsidian — it can't reach an MCP server. The plugin would need its own client to the same Chroma instance, OR the RAG pre-flight runs *outside* the plugin and writes its output into the target file before the plugin's Perplexity stage triggers.
|
||||
|
||||
## Why we don't already know
|
||||
|
||||
Three reasons.
|
||||
|
||||
**One — the existing `cft` template surface is intentionally minimal.** The `ParsedTemplate` interface ([`src/services/directoryTemplateService.ts:31`](../../src/services/directoryTemplateService.ts)) gives a template four things: a config block, a system prompt, a user skeleton, and the four shipped templates ride that surface comfortably because each is *one* provider call. The moment we add a second provider — Claude after Perplexity, or a RAG client before either — we need to decide whether the template is still one file (with multiple stages declared inside) or whether it's a *pipeline* with a per-stage manifest. We have no precedent in this codebase for the latter.
|
||||
|
||||
**Two — `include_sources` isn't a Perplexity API field, and we haven't decided what it means.** `search_domain_filter` constrains where Perplexity searches; it doesn't inject our content into the prompt. What the user is gesturing at — "a list of sources to include" — is closer to **retrieval-augmented generation**: pull canonical text from named documents (or a Chroma vector query), splice it into the prompt as named, citeable inputs, then let the model lean on it. We already have the splice primitive (`{{include: name}}` partials, [`directoryTemplateService.ts:83`](../../src/services/directoryTemplateService.ts)) — but partials are pure snippets, content-agnostic. RAG sources need different handling: they want to be named, attributed, and bounded so the model knows what it's quoting.
|
||||
|
||||
**Three — we don't yet know whether the editing stage should run as an *edit* or as an *additive pass*.** The user's framing — *"Perplexity and Claude do not overwrite each other until editing"* — implies a model where Perplexity writes its draft in one zone of the file, Claude writes in another, and only at the explicit edit-phase do they consolidate. That's a real design choice with consequences for the file shape, the cft block schema, and how the user knows where they are in the pipeline.
|
||||
|
||||
## Options
|
||||
|
||||
Three architectures to weigh. They are not mutually exclusive — Option C is closer to a generalization of A and B than a distinct third path.
|
||||
|
||||
### Option A — Sequential single-file appends with named zones
|
||||
|
||||
The template stays one markdown file in `zz-cf-lib/templates/`. The cft block grows new keys but the file shape stays familiar.
|
||||
|
||||
Pipeline:
|
||||
|
||||
1. **RAG pre-flight (in-plugin).** Plugin reads `include_sources:` from the cft block, resolves it (vault paths and/or Chroma queries), splices the retrieved text into the system prompt as a named context block. No file writes happen at this stage — RAG context flows straight into the prompt.
|
||||
2. **Perplexity stage.** Runs as today: streams body content into the target file, stamps `cf_last_run` / `cf_last_run_model`, appends `# Sources` footer. Output lands in a zone delimited by `<!-- perplexity:start -->` and `<!-- perplexity:end -->` HTML comments.
|
||||
3. **Claude editorial stage.** Reads the entire file, runs Claude with a specific edit prompt (`edit-system:` from the cft block, or a separate `claude` block), writes the edited version into a NEW zone delimited by `<!-- claude:start -->` and `<!-- claude:end -->`. The Perplexity zone is preserved untouched as provenance; the Claude zone is what the published doc draws from.
|
||||
|
||||
cft block grows:
|
||||
|
||||
```yaml
|
||||
provider: perplexity
|
||||
model: sonar-deep-research
|
||||
include-sources:
|
||||
- vault: "Tooling/AI-Toolkit/Agentic AI/**"
|
||||
- vault: "concepts/Explainers for AI/Agentic Workspaces.md"
|
||||
- chroma:
|
||||
collection: context-vigilance-corpus
|
||||
query: "{{title}} market map prior art"
|
||||
n_results: 5
|
||||
system: |
|
||||
...
|
||||
|
||||
editor:
|
||||
provider: anthropic
|
||||
model: claude-opus-4-7
|
||||
pass: editorial-restructure
|
||||
system: |
|
||||
You are editing the Perplexity-drafted market map below into final form...
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
|
||||
- One file per market map; consistent with every other directory template.
|
||||
- The zoned-append discipline (Perplexity zone, Claude zone) makes provenance auditable — you can read exactly what each agent produced.
|
||||
- Both stages re-runnable independently. Re-run Perplexity to refresh the source data; re-run Claude to refresh the edit without re-burning a deep-research credit.
|
||||
- `include-sources` becomes a structured field that handles both vault-path and Chroma-query forms, with the resolution logic centralized in the plugin.
|
||||
|
||||
**Cons:**
|
||||
|
||||
- One file gets long fast. Market maps are already 4-8K words; doubling the body to keep both zones means the on-disk file is hefty. Obsidian handles it, but readability degrades.
|
||||
- The Chroma client has to run inside the Obsidian plugin. That means embedding a Chroma HTTP client in `main.js`, configuring the Chroma URL in plugin settings, and handling the failure mode where Chroma is unreachable.
|
||||
- The HTML-comment zone markers are fragile — a user editing the file in another pane could break them, and at that point the re-run logic doesn't know where to write.
|
||||
- Claude's editorial output sitting in a sibling zone means downstream rendering (the public site) needs to know to draw from the Claude zone, not the Perplexity zone. That's a coupling that didn't exist before.
|
||||
|
||||
### Option B — Multi-document folder, section per file, with a manifest
|
||||
|
||||
Each market map becomes a folder, not a file. `lost-in-public/market-maps/Quantum Computing is Confusing/` rather than `Quantum Computing is Confusing.md`. Inside the folder:
|
||||
|
||||
```
|
||||
Quantum Computing is Confusing/
|
||||
├── _manifest.md # frontmatter only — title, lede, banner_image, etc.
|
||||
├── 00-snapshot.md # Market Snapshot section
|
||||
├── 01-question.md # The Question this Map Answers
|
||||
├── 02-why-now.md # Why Now
|
||||
├── 03-map.md # Map of the Market — Sub-Segments
|
||||
├── 04-lighthouse.md # Lighthouse Examples
|
||||
├── 05-profiles-superconducting.md # one file per sub-segment
|
||||
├── 05-profiles-photonic.md
|
||||
├── 05-profiles-trapped-ion.md
|
||||
├── 06-media.md
|
||||
├── 07-dynamics.md
|
||||
├── 08-frontier.md
|
||||
├── 09-adjacent.md
|
||||
└── _sources.md # consolidated sources footer
|
||||
```
|
||||
|
||||
Each `NN-*.md` file is independently re-runnable. The template becomes a *pipeline manifest* that names which agent runs on which file and in what order. The cft format extends to declare per-file stages.
|
||||
|
||||
Pipeline:
|
||||
|
||||
1. **Scaffold.** A new command — `Initialize market map folder` — creates the folder structure from a template manifest, populating each section file's frontmatter with its position in the pipeline.
|
||||
2. **Per-section RAG + Perplexity.** Each section file declares its own `include-sources:` and runs its own targeted Perplexity query. Sub-segment files (`05-profiles-*.md`) get the most aggressive RAG injection (vault tools matching the sub-segment).
|
||||
3. **Section-level Claude pass.** Claude edits each section file independently against a section-specific edit prompt.
|
||||
4. **Roll-up.** A final command stitches the section files into one rendered markdown file for publication, or the public site renders the folder directly via a folder-aware renderer.
|
||||
|
||||
**Pros:**
|
||||
|
||||
- Each section is independently re-runnable. The biggest pain in a one-shot 6K-word generation is "the trapped-ion sub-segment is thin, I'd like to re-run just that one." Folder form makes that natural.
|
||||
- RAG context can be tightly scoped per section — the Lighthouse Examples section gets a different vault query than the Media section. With one mega-prompt, RAG context is one-size-fits-all.
|
||||
- Per-section editorial passes are cheaper. Editing a 600-word section is faster and tighter than editing a 6K-word document.
|
||||
- The folder maps cleanly to how an actual analyst works — they don't write the whole memo in one sitting; they research the sub-segments first, then sequence the framing prose.
|
||||
- Agentic orchestration becomes natural: a future orchestrator-agent can iterate over the section files, decide which need refresh, and dispatch jobs in parallel.
|
||||
|
||||
**Cons:**
|
||||
|
||||
- Breaks every existing convention in the perplexed plugin and the vault. The four shipped templates target one file each; making market maps target a folder is a tier shift.
|
||||
- The existing `applies-to-paths` glob ([`directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts)) matches a file against a template; we'd need parallel `applies-to-folders` matching, or a new command entirely.
|
||||
- Cross-section consistency is harder. The Innovator Profiles section needs to reference the same sub-segment names declared in the Map of the Market section. With separate files, drift across sections becomes likely.
|
||||
- The published-on-the-web rendering needs a folder-aware renderer. Astro Knots can do this (it already renders nested content), but every existing market map is a single file and migrating them is its own project.
|
||||
- Users edit market maps as one document in their head. A folder breaks that mental model.
|
||||
|
||||
### Option C — Per-section refresh blocks inside a single file
|
||||
|
||||
Take the multi-`cft` idea floated in the directory-templates doc's *Known limits and open items* and run with it. The template file stays one markdown file. The skeleton declares per-section refresh prompts via dedicated `cft` blocks inline under each H1.
|
||||
|
||||
```markdown
|
||||
# Market Snapshot
|
||||
|
||||
```cft-section
|
||||
provider: perplexity
|
||||
model: sonar-deep-research
|
||||
include-sources: [...]
|
||||
prompt: |
|
||||
Write the Market Snapshot section per the instructions below.
|
||||
```
|
||||
|
||||
- Bullet instructions for this section...
|
||||
|
||||
# Innovator Profiles
|
||||
|
||||
```cft-section
|
||||
provider: perplexity
|
||||
model: sonar-deep-research
|
||||
include-sources:
|
||||
- vault: "Tooling/AI-Toolkit/Agentic AI/**"
|
||||
prompt: |
|
||||
Write the Innovator Profiles section. Use RAG content to name actual
|
||||
vault tools where possible.
|
||||
```
|
||||
|
||||
- Bullet instructions...
|
||||
```
|
||||
|
||||
A header-level `cft` block stays at the top of the template declaring the editorial stage (Claude pass) that runs across the whole file at the end.
|
||||
|
||||
Pipeline:
|
||||
|
||||
1. **Section-by-section Perplexity runs.** The plugin walks each `cft-section` block and runs its own targeted query. Section content lands under that H1.
|
||||
2. **File-level Claude editorial pass.** After all sections are filled, Claude reads the assembled file and writes the edited version per the file-level `editor:` block.
|
||||
|
||||
**Pros:**
|
||||
|
||||
- One file per market map (keeps Option A's familiarity).
|
||||
- Per-section RAG and Perplexity targeting (gets Option B's tight scoping).
|
||||
- Re-runnable per section — invoke `Refresh section` on the active section's H1 to re-run just that `cft-section`.
|
||||
- Sub-segment-aware RAG: each section's `include-sources` can be tuned to what that section needs.
|
||||
|
||||
**Cons:**
|
||||
|
||||
- The `cft-section` schema is novel; we'd have to design the multi-block parser, the per-section command surface, and the interaction with the existing one-`cft` flow.
|
||||
- The file is longer at-rest because it carries every section's prompt inline. This is a feature (visible provenance) and a bug (clutter) at the same time.
|
||||
- "Append vs. fill" semantics become per-section. The mode logic in `applyTemplate` gets more states.
|
||||
|
||||
## The `include_sources` sub-exploration
|
||||
|
||||
This is the design call most likely to set the trajectory of everything else. The user named it as a key argument, and it sits at the intersection of three primitives we already have:
|
||||
|
||||
| Primitive | What it does today | What's missing for `include_sources` |
|
||||
|---|---|---|
|
||||
| `search-domains:` ([`directoryTemplateService.ts:486`](../../src/services/directoryTemplateService.ts)) | Constrains Perplexity's open-web search to a domain allowlist + a job-board denylist | Doesn't inject content into the prompt; only filters where the model searches |
|
||||
| `{{include: name}}` partials ([`directoryTemplateService.ts:83`](../../src/services/directoryTemplateService.ts)) | Splices a named partial's body into the prompt, content-agnostic | No naming/attribution wrapper for the spliced content; no Chroma-query form |
|
||||
| Chroma collections (`context-vigilance-corpus` et al, [`lossless-monorepo/CLAUDE.md`](../../../../CLAUDE.md)) | Section-chunked vault content indexed and queryable via the `chroma` MCP server | Only reachable from inside Claude Code today, not from inside Obsidian plugin runtime |
|
||||
|
||||
Proposed shape for `include-sources:` in the cft block:
|
||||
|
||||
```yaml
|
||||
include-sources:
|
||||
# Vault path — read the file, splice its body into the prompt as a named,
|
||||
# attributed context block. Glob form is allowed; matched files are
|
||||
# concatenated with their basenames as the attribution label.
|
||||
- vault: "concepts/Explainers for AI/Agentic Workspaces.md"
|
||||
- vault: "Tooling/AI-Toolkit/Agentic AI/**"
|
||||
max_files: 10 # cap for glob expansion
|
||||
body_only: true # strip each file's frontmatter (default true)
|
||||
|
||||
# Chroma query — run a semantic search against a named collection,
|
||||
# splice the top N results into the prompt with their source_path
|
||||
# and source_repo_slug as attribution.
|
||||
- chroma:
|
||||
collection: context-vigilance-corpus
|
||||
query: "{{title}} market segmentation"
|
||||
n_results: 5
|
||||
where: # optional metadata filter
|
||||
source_repo_slug: "lossless"
|
||||
|
||||
# External URL — fetch and inline. Lowest priority because we can't
|
||||
# cache, but useful for one-off canonical references the user wants
|
||||
# baked in. Requires explicit opt-in in settings.
|
||||
- url: "https://www.lossless.group/projects/gallery/agentic-workspaces"
|
||||
max_chars: 8000
|
||||
```
|
||||
|
||||
How the spliced content lands in the prompt:
|
||||
|
||||
```
|
||||
## Canonical sources (use these as primary attributions where applicable)
|
||||
|
||||
### Source: Tooling/AI-Toolkit/Agentic AI/Crew AI.md
|
||||
<body of that vault file>
|
||||
|
||||
### Source: concepts/Explainers for AI/Agentic Workspaces.md
|
||||
<body of that vault file>
|
||||
|
||||
### Source: Chroma (context-vigilance-corpus) — n=3
|
||||
<top chunks, each labeled with source_path>
|
||||
|
||||
## Search the open web for additional sources beyond the canonical set above.
|
||||
```
|
||||
|
||||
This shape solves the "named, attributed" problem (the model knows what it's quoting and can cite by vault path), keeps glob expansion bounded (`max_files`), and degrades gracefully when Chroma is unreachable (skip the chroma entries with a warning notice; vault entries still resolve).
|
||||
|
||||
Open sub-questions:
|
||||
|
||||
- **Does `include-sources` belong at the template level or the section level?** Template-level is simpler. Section-level is more powerful but only matters if we go Option B or C.
|
||||
- **How big can the canonical-sources block grow before we OOM the prompt?** `sonar-deep-research` has generous context but not infinite. A vault glob matching 100 files is too much. The `max_files` cap matters; defaults should be conservative (5-10).
|
||||
- **Is the Chroma client an in-plugin dependency or an out-of-plugin pre-flight?** If we ship a Chroma client in `main.js`, plugin size grows and we need to surface Chroma URL configuration in settings. If we keep RAG pre-flight outside the plugin (a CLI that writes a `_rag-context.md` partial the plugin then includes), the plugin stays lean but the workflow gains an extra step.
|
||||
|
||||
## "Perplexity and Claude do not overwrite each other until editing"
|
||||
|
||||
This is the design heuristic that distinguishes Option A's zoned-append from a naive sequential pipeline. Three implementations of the heuristic, in increasing strictness:
|
||||
|
||||
1. **Soft separation (zones).** Perplexity writes into one zone, Claude writes into another. Both zones live in the same file. Provenance is preserved; the published doc draws from the Claude zone but the Perplexity zone is readable.
|
||||
2. **Hard separation (files).** Perplexity writes its draft to `<map>.perplexity.md`. Claude reads that file and writes its edit to `<map>.md` (or vice versa, depending on which is canonical). The original is preserved untouched.
|
||||
3. **Strict additive (no edits, only appendages).** Both agents append; neither edits. The final document is a stitched sequence of their outputs with explicit attribution. Useful if we want to preserve "the analyst's draft" and "the editor's reorganization" as separate readable artifacts.
|
||||
|
||||
The strict additive form (3) has appeal — it makes the multi-agent collaboration audit-trail-perfect — but it's hostile to the actual editorial goal, which is *to merge the two voices into one coherent document*. Soft separation (1) hits the right tradeoff for a v1: provenance is preserved, the final published form is one voice, and re-running either stage doesn't destroy the other's work.
|
||||
|
||||
The "until editing" clause is where the user's framing gets sharp. **The editing stage is the only point at which the two agents' outputs are consolidated.** Before that, they're parallel artifacts. This is why Option A's zoned append is the most faithful interpretation: Perplexity and Claude do their work in parallel zones; the editing stage (which Claude itself performs) is when the consolidation happens, producing the canonical Claude zone.
|
||||
|
||||
## Live example surfacing the timeout issue
|
||||
|
||||
The first real-world run of the v1 `market-map-profile` template — `lost-in-public/market-maps/Humanoid Robots and their Input Industries.md`, executed 2026-05-26 — produced a ~7,500-word draft that terminated mid-sentence inside *Frontier and Open Questions* because the directory-template runtime's wall-clock `AbortController` fired before the stream completed. That truncation is the motivating use case for the **per-template `request-timeout-ms:` cft-block override** we shipped on the same day as this exploration (default bumped to 30 min, market-map-profile overridden to 40 min). The pressure-relief valve is in place; the structural fix — porting the idle-timeout discipline from `perplexityService.ts:659-668` into `streamPerplexityToFile` — is captured as its own issue at [[Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams]] and feeds back into this exploration's *Open items* below.
|
||||
|
||||
This is the shape of validation we want for the multi-stage pipeline overall: ship the lean v1, run it on a real document, surface the structural gaps as their own context-v issues (where they get the attention they deserve), and let the issues feed into the eventual spec.
|
||||
|
||||
## Findings so far
|
||||
|
||||
From this exploration session, the [partials-and-preambles issue](../issues/Partials-And-Preambles-For-Perplexed-Templates.md) the architecture builds on, and the [wall-clock-timeout issue](../issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md) surfaced by the first real market-map run:
|
||||
|
||||
- **The `{{include: name}}` partial primitive is already structurally close to what `include-sources` needs.** What's missing is (a) the per-source attribution wrapper, (b) the Chroma-query form, and (c) the multi-source named-block formatting in the prompt. The partial loader's depth and cycle guards transfer directly.
|
||||
- **The cft config is already a structured YAML surface.** Adding `include-sources:` (a list) and `editor:` (an object) doesn't break the existing parser; the four shipped templates remain valid because they don't declare either key.
|
||||
- **`search-domains:` and `include-sources:` are complementary, not redundant.** Domains constrain where Perplexity searches the open web; include-sources injects canonical content into the prompt directly. A future market-map run would likely use both: include-sources to seed our canonical entries, search-domains to keep open-web search restricted to credible analyst sources (Bloomberg, FT, sector trade press, founder blogs).
|
||||
- **Claude can already stream to a file** ([`claudeService.ts:48`](../../src/services/claudeService.ts) `queryClaude`). The editorial stage doesn't need a new transport — it needs a different prompt construction path and a different write-zone strategy.
|
||||
- **The `editor:` block can be opt-in.** Templates that don't declare it stay single-stage and behave exactly as they do today. The four shipped templates would stay single-stage; only `market-map-profile` and any future research-heavy templates would declare an `editor:`.
|
||||
- **Chroma reachability from inside Obsidian is the long pole.** The MCP server only exists in Claude Code; the Obsidian plugin runs in Electron and needs its own HTTP client to a Chroma instance, OR we keep RAG pre-flight as an out-of-band step that writes a context partial the plugin then includes.
|
||||
|
||||
## Tentative direction
|
||||
|
||||
Lean toward **Option A (zoned single-file appends) with the `include-sources:` and `editor:` keys added to the cft schema**, and keep RAG pre-flight as an *in-plugin* feature with Chroma queries optional (vault-path includes always work; Chroma includes degrade to a Notice when unreachable).
|
||||
|
||||
Reasoning:
|
||||
|
||||
- Option A is the smallest delta from the existing template surface. The four shipped templates remain unchanged; market-map-profile gains two new keys.
|
||||
- Vault-path includes get us most of the RAG win without the Chroma-reachability problem. The dominant value of "include our canonical tools and concepts" is satisfied by globbed vault includes; Chroma adds discoverability over the wider corpus but isn't load-bearing for the first iteration.
|
||||
- Zoned appends preserve the "Perplexity and Claude do not overwrite each other until editing" heuristic literally: each agent writes its zone, the editing stage consolidates. If we want hard separation later, we can promote the zones into separate files without redesigning the schema.
|
||||
- Option B (multi-doc folder) is the right end-state for sufficiently complex maps but is a tier shift that should be earned, not pre-emptively designed.
|
||||
- Option C (per-section cft blocks) is the right *generalization* but is also the right way to over-design. Defer until we've shipped Option A and felt the actual pain.
|
||||
|
||||
Defer for a follow-up spec:
|
||||
|
||||
- The Chroma in-plugin client work. Ship vault-path includes first; revisit Chroma after we know whether the in-plugin client size cost is worth the discoverability gain.
|
||||
- The multi-doc folder form (Option B). Track as a possible v3 once we have 10+ market maps and can feel the cross-section consistency pain.
|
||||
- The per-section `cft-section` block (Option C). Track as a possible v2 alternative if Option A's zoned appends turn out to be insufficient for sub-segment scoping.
|
||||
|
||||
## Open items before we promote to a spec
|
||||
|
||||
- [ ] Confirm zone delimiter shape. HTML comments are tolerant in Obsidian and Astro Knots renderers; named-anchor headings (`<!-- region: perplexity -->`) work too. Pick one and document.
|
||||
- [ ] Decide whether `editor:` is a block or a list. A list (`editor: [{ provider: anthropic, ... }, { provider: anthropic, pass: copyedit, ... }]`) would let us chain multiple Claude passes; a block keeps the single-pass v1 simpler.
|
||||
- [ ] Spec the `include-sources` resolution order and de-duplication policy. If a vault glob and a Chroma query both surface the same source file, do we splice it twice?
|
||||
- [ ] Decide attribution-block format. Does each included source get a top-level H2 in the prompt context, or a fenced block, or YAML-front-matter-wrapped? Trade off model-comprehension against prompt-token economy.
|
||||
- [ ] Decide what `cf_last_run` stamps look like for a multi-stage run. One timestamp per stage (`cf_last_run_perplexity`, `cf_last_run_editor`)? A composite? This matters for the "is this map stale?" query later.
|
||||
- [ ] Verify Chroma collections are queryable from a non-MCP HTTP client. The [chroma-local skill](../../../../context-v/skills/chroma-local/SKILL.md) covers `ChromaClient`/`HttpClient` setup; should be a 30-minute spike.
|
||||
- [ ] Resolve [[Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams]] before the multi-stage spec lands — the current wall-clock primitive cuts off long stage-2 (Perplexity) runs and will cut off long stage-3 (Claude editorial) runs too, since the editor's per-section reasoning on a 7-8K-word draft is itself a long generation. The structural fix (idle-timeout discipline) belongs upstream of multi-stage, not as part of it.
|
||||
|
||||
## Outcome
|
||||
|
||||
Open. When this resolves, the expected artifacts are:
|
||||
|
||||
1. A spec in [`context-v/specs/`](../specs/) — `Multi-Stage-Templates-with-Include-Sources-and-Editor.md` (working title) — that pins down the cft schema additions, the zone-delimiter format, the resolution order for `include-sources`, the editorial-stage contract, and the migration path for existing templates.
|
||||
2. A prompt in [`context-v/plans/`](../plans/) — chunked implementation steps for the spec.
|
||||
3. The `market-map-profile` template updated to use the new keys, treated as the reference implementation.
|
||||
4. A v2 follow-up exploration in this folder once we've felt how Option A actually behaves with one or two real market maps generated through it.
|
||||
|
||||
## Related
|
||||
|
||||
- [[market-map-profile]] — the lean v1 template this exploration extends
|
||||
- [[Using-Files-as-Prompt-Outlines]] — the original spec that motivated directory templates
|
||||
- [[Partials-And-Preambles-For-Perplexed-Templates]] — the issue that established the splice primitive `include-sources` builds on
|
||||
- [[Getting-Claude-to-Respond-With-Research]] — prior work on Claude as a research-aware agent
|
||||
- [[search-lossless-corpus]] — the Claude-Code-side skill that already encodes the four-collection RAG discipline; the in-plugin RAG flow is the Obsidian analog
|
||||
- [`lossless-monorepo/CLAUDE.md`](../../../../CLAUDE.md) — the canonical description of the four Chroma collections this exploration's RAG stage would draw on
|
||||
|
|
@ -0,0 +1,119 @@
|
|||
---
|
||||
title: Wall-clock timeout cuts off long deep-research streams
|
||||
lede: "The directory-template runtime caps every stream by total wall-clock duration, but the legacy modal flow already moved to per-chunk idle-timeout discipline two iterations ago — and the discrepancy is now actively truncating analyst-grade market-map drafts mid-sentence."
|
||||
date_created: 2026-05-26
|
||||
date_modified: 2026-05-26
|
||||
authors:
|
||||
- Michael Staton
|
||||
augmented_with:
|
||||
- Claude Opus 4.7 (1M context)
|
||||
semantic_version: 0.0.0.1
|
||||
type: issue
|
||||
status: open
|
||||
target_repo: perplexed
|
||||
tags:
|
||||
- Issue-Resolution
|
||||
- Perplexed
|
||||
- Streaming-Timeouts
|
||||
- Deep-Research
|
||||
- Directory-Templates
|
||||
related:
|
||||
- "[[market-map-profile]]"
|
||||
- "[[Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG]]"
|
||||
- "[[Partials-And-Preambles-For-Perplexed-Templates]]"
|
||||
---
|
||||
|
||||
# Wall-clock timeout cuts off long deep-research streams
|
||||
|
||||
## Symptom
|
||||
|
||||
Running `market-map-profile` on `lost-in-public/market-maps/Humanoid Robots and their Input Industries.md` produced a ~7,500-word draft that terminated mid-sentence inside the *Frontier and Open Questions* section:
|
||||
|
||||
```
|
||||
Will the evolution of robot-as-a-service (R
|
||||
```
|
||||
|
||||
The trailing parenthesis is the last byte written. The *Adjacent Concepts and Maps* section — the final heading in the template skeleton — never appeared. The frontmatter stamps (`cf_last_run`, `cf_last_run_model`) landed correctly, so the run did complete its `processFrontMatter` post-step; what was lost was the streamed body content that hadn't yet arrived when the `AbortController` fired.
|
||||
|
||||
The Humanoid Robots run is the first observable instance of this specific cut-off shape, but the pattern is structural — it will reproduce on any sufficiently long deep-research generation run through the directory-template flow.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
Two different streaming primitives live in this codebase, with two different timeout disciplines:
|
||||
|
||||
**Directory-template flow** ([`src/services/directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts) `streamPerplexityToFile`, the function `market-map-profile` and the four other shipped templates run through):
|
||||
|
||||
```ts
|
||||
const controller = new AbortController();
|
||||
const timer = activeWindow.setTimeout(() => controller.abort(), timeoutMs);
|
||||
```
|
||||
|
||||
A **single wall-clock `setTimeout`** is armed at the moment of fetch. After `timeoutMs` elapses — regardless of whether the stream is actively producing bytes — `controller.abort()` fires and the in-flight `reader.read()` throws. The catch sets `truncated = true` and falls through to a final flush of whatever streamed so far. The plugin-level default was `600_000` ms (10 min) before today's fix; analyst-grade deep-research runs routinely run 15-25 min, so the cut-off was inevitable on long templates.
|
||||
|
||||
**Legacy modal flow** ([`src/services/perplexityService.ts:659`](../../src/services/perplexityService.ts), `PerplexityModal`):
|
||||
|
||||
```ts
|
||||
const STREAM_IDLE_TIMEOUT_MS = isDeepResearch ? 270_000 : 90_000;
|
||||
const readWithIdleTimeout = (): Promise<...> => {
|
||||
let timer: number | undefined;
|
||||
const timeout = new Promise<never>((_, reject) => {
|
||||
timer = window.setTimeout(() => {
|
||||
reject(new Error(`stream went idle for ${...}s ...`));
|
||||
}, STREAM_IDLE_TIMEOUT_MS);
|
||||
});
|
||||
return Promise.race([reader.read(), timeout]).finally(() => {
|
||||
if (timer !== undefined) window.clearTimeout(timer);
|
||||
});
|
||||
};
|
||||
```
|
||||
|
||||
**Per-chunk idle timeout.** A fresh `setTimeout` is armed and racing each `reader.read()` call. As long as bytes keep arriving, the timer keeps getting cleared and re-armed. The stream is only killed if it goes *quiet* for 270 s (deep-research) or 90 s (normal). Total wall-clock duration is unbounded.
|
||||
|
||||
The legacy modal moved to this pattern because the same problem hit users there first — but the directory-template flow was forked from an earlier iteration of the streaming code and never received the idle-timeout backport. The legacy `PerplexityModal` and `streamPerplexityToFile` now disagree on how to time-bound a Perplexity stream, and the directory-template flow has the strictly weaker discipline.
|
||||
|
||||
## What we shipped today (partial fix)
|
||||
|
||||
Not a structural fix — a pressure-relief valve. Two changes:
|
||||
|
||||
1. **Bumped the plugin-level default** ([`main.ts:333`](../../main.ts)) from `600_000` ms (10 min) to `1_800_000` ms (30 min). The settings-pane description was updated to call out the override and the cost framing (`$10-$50 of analyst time per good output is worth waiting for`).
|
||||
2. **Added a per-template override** — `request-timeout-ms:` in the cft block. Resolution code lives in [`directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts) just before the `streamPerplexityToFile` call; accepts number or numeric-string, silently falls back on non-positive / non-numeric values.
|
||||
3. **`market-map-profile.md`** declares `request-timeout-ms: 2400000` (40 min) with an inline comment explaining the budget; the other four shipped templates inherit the plugin-level 30-min default.
|
||||
4. **Docs** — [`docs/directory-templates.md`](../../docs/directory-templates.md) gained a *Per-template timeout override* section with override semantics and a "when to bump" checklist; [`src/docs/templates/README.md`](../../src/docs/templates/README.md) calls out the key in the cft-block list.
|
||||
|
||||
This buys headroom. It does not fix the structural problem: any sufficiently long deep-research run will still hit the wall eventually. The 40-min cap is a *guess* about how long the longest reasonable market-map should take, not a property derived from the stream's actual behavior.
|
||||
|
||||
## Why this is only a partial fix
|
||||
|
||||
The wall-clock timeout fails in two distinct shapes that the idle-timeout pattern handles correctly:
|
||||
|
||||
**Shape 1 — slow but healthy stream.** Deep-research generations on long templates may sustain a slow trickle of tokens for 30-45 minutes. The wall-clock cap kills them at the ceiling regardless of whether they're still producing. The idle-timeout pattern lets them complete naturally as long as some byte arrives every N seconds.
|
||||
|
||||
**Shape 2 — silently stalled stream.** Conversely, a stream may go quiet at minute 3 (Perplexity rate-limit, socket close, upstream stall) and the wall-clock cap won't notice until minute 30. The user stares at an empty file, watching the spinner spin, for 27 unnecessary minutes. The idle-timeout pattern surfaces the failure within `STREAM_IDLE_TIMEOUT_MS` seconds — fast feedback when something is genuinely wrong.
|
||||
|
||||
Both shapes are real. The wall-clock pattern punishes the healthy-but-slow case while tolerating the stalled-but-silent case. The idle-timeout pattern inverts both — slow-but-healthy completes; stalled-but-silent fails fast.
|
||||
|
||||
## Proposed structural fix
|
||||
|
||||
Port the idle-timeout pattern from [`perplexityService.ts:659-668`](../../src/services/perplexityService.ts) into `streamPerplexityToFile`. Concretely:
|
||||
|
||||
1. Replace the single wall-clock `setTimeout(controller.abort, timeoutMs)` with a per-chunk `readWithIdleTimeout()` that wraps each `reader.read()` in a `Promise.race` against a fresh timeout.
|
||||
2. Choose idle-timeout values consistent with the existing modal flow: `270_000` ms (4.5 min) for deep-research models, `90_000` ms (1.5 min) for normal models. Detect deep-research from the resolved model name string (`/deep-research/i`), the same way `perplexityService.ts` does.
|
||||
3. Retain the cft-block override key, but rename it to `stream-idle-timeout-ms:` for accuracy. Templates that have been declaring `request-timeout-ms:` need a compatibility alias for one release; document the rename in the changelog entry.
|
||||
4. Optionally retain a generous absolute wall-clock ceiling (60 min or 120 min) as a sanity backstop — but that's belt-and-suspenders, not load-bearing. The idle timeout is doing the actual safety work.
|
||||
|
||||
The diff is moderate — `streamPerplexityToFile` is ~140 lines today; the refactor touches roughly the first 40 of those (the timer setup and the `reader.read()` call inside the loop). The post-stream cleanup pipeline (`wrapThinkBlocks`, `processContentWithImages`, `buildSourcesFooter`) is unaffected.
|
||||
|
||||
## Open items before this becomes a spec
|
||||
|
||||
- [ ] Decide whether the cft-block key stays named `request-timeout-ms:` (which becomes a misnomer once idle-timeout is what's actually applied) or migrates to `stream-idle-timeout-ms:`. Compatibility-alias plan if the latter.
|
||||
- [ ] Decide whether to retain a wall-clock absolute ceiling alongside the idle timeout, and if so what the value is (60 min? 120 min? infinite with a Notice that surfaces "this run has been going for X min" past a threshold?).
|
||||
- [ ] Audit other long-stream callsites in the codebase that may have the same wall-clock pathology (Gemini service, LM Studio service, Claude service streaming flows) — the idle-timeout discipline should be the house style across all of them.
|
||||
- [ ] Confirm the model-name-based deep-research detection (`/deep-research/i` regex on the resolved model) is robust against future Perplexity model name changes; consider exposing the idle-timeout values as plugin settings so they can be tuned without a code change.
|
||||
|
||||
## Related
|
||||
|
||||
- [[market-map-profile]] — the template that surfaced the bug
|
||||
- [[Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG]] — the broader exploration this issue feeds findings back into; the idle-timeout refactor is listed there as an open item
|
||||
- [[Partials-And-Preambles-For-Perplexed-Templates]] — the architecture-review structure of that issue is the precedent this one follows
|
||||
- [`src/services/perplexityService.ts`](../../src/services/perplexityService.ts) lines 659-668 — the idle-timeout implementation in the legacy modal flow that we're proposing to port
|
||||
- [`src/services/directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts) `streamPerplexityToFile` lines 499-635 — the wall-clock implementation in the directory-template flow that the port replaces
|
||||
|
|
@ -79,7 +79,7 @@ # User Notes
|
|||
### The three zones
|
||||
|
||||
1. **Frontmatter** (between `---` lines at the top) — carries `title`, `applies-to-paths` (array of globs), and an optional `description`. The runtime uses `applies-to-paths` to match a template to a target file.
|
||||
2. **`cft` block** (a code fence with language `cft`) — YAML carrying `provider`, `model`, optional `search-recency`, `return-citations`, `return-images`, and a multi-line `system:` prompt. Everything **above** the `cft` block is dropped from the request.
|
||||
2. **`cft` block** (a code fence with language `cft`) — YAML carrying `provider`, `model`, optional `search-recency`, `return-citations`, `return-images`, optional `request-timeout-ms:` (per-template wall-clock override — see *Per-template timeout override* below), and a multi-line `system:` prompt. Everything **above** the `cft` block is dropped from the request.
|
||||
3. **Heading skeleton** — the markdown structure between the `cft` block's closing fence and the first `***` divider. This becomes the user prompt. Bullets under each heading are *instructions to the model*, not literal output. Everything **below** the first `***` is excluded from the request.
|
||||
|
||||
### Interpolation tokens
|
||||
|
|
@ -140,7 +140,7 @@ ### Frontmatter stamps
|
|||
|
||||
## Shipped templates
|
||||
|
||||
Four templates ship inlined into `main.js` (via esbuild's `.md` text loader) and are seeded into the user's vault on first plugin load.
|
||||
Five templates ship inlined into `main.js` (via esbuild's `.md` text loader) and are seeded into the user's vault on first plugin load.
|
||||
|
||||
| File | Targets | Model | Notes |
|
||||
|---|---|---|---|
|
||||
|
|
@ -148,6 +148,7 @@ ## Shipped templates
|
|||
| `vocabulary-profile.md` | `Vocabulary/**` | `sonar-pro` | Term definitions with disambiguation through an innovation-consulting lens. |
|
||||
| `source-profile.md` | `Sources/**` | `sonar-pro` | Profiles of trusted sources — books, people, channels, publications, journals, reports, events. Type-aware: the system prompt enumerates seven canonical types and the model picks one from frontmatter signals (`youtube_channel_url` → channel, `aliases` → likely book, etc.). Each section has per-type bullet shapes. |
|
||||
| `toolkit-profile.md` | `Tooling/**` | `sonar-pro` | Profiles of tools, products, platforms, frameworks. |
|
||||
| `market-map-profile.md` | `lost-in-public/market-maps/**`, `market-maps/**` | `sonar-deep-research` | Analyst-grade market-map drafts. Dual flavor: Known Category (Quantum Computing, Humanoid Robots) or Thesis-Driven (e.g., Neural Network Hardware as Brains for Robotics) traversing adjacent categories. Lean v1, single-stage. Skips image return by design — deep-research's image metadata is unreliable and market-map imagery is generated separately (Ideogram → frontmatter `banner_image` / `portrait_image` / `square_image`). Multi-stage v2 (RAG pre-flight to inject canonical Lossless tools/concepts as context + Claude editorial pass to emit `[[wikilink]]`s) is deferred — see the User Notes zone of the template for the roadmap. |
|
||||
|
||||
`source-profile` is the trickiest because `Sources/` is genuinely heterogeneous. The solution is one template, type-conditional content. Books also trigger Google Books URL handling: frontmatter `google_books_url` is used if present, otherwise the model finds it; either way the URL is harvested into frontmatter post-generation via regex, so subsequent runs skip the search.
|
||||
|
||||
|
|
@ -199,6 +200,34 @@ ## Commands
|
|||
|
||||
---
|
||||
|
||||
## Per-template timeout override
|
||||
|
||||
The runtime applies a wall-clock timeout to every Perplexity stream via an `AbortController` (see [`directoryTemplateService.ts`](../src/services/directoryTemplateService.ts) `streamPerplexityToFile`). The plugin-level default (*Plugin settings → Directory templates → Request timeout (ms)*) is **30 min** (1,800,000 ms) — generous because most templates are well under that ceiling, and the few that aren't (the analyst-grade deep-research templates) shouldn't be cut off in the middle of a draft worth $10-$50 of analyst time.
|
||||
|
||||
Templates can override the plugin-level default by declaring `request-timeout-ms:` inside their `cft` block:
|
||||
|
||||
```cft
|
||||
provider: perplexity
|
||||
model: sonar-deep-research
|
||||
request-timeout-ms: 2400000 # 40 min
|
||||
system: |
|
||||
...
|
||||
```
|
||||
|
||||
Override semantics:
|
||||
|
||||
- A numeric value or a numeric string both work; non-positive / non-numeric values are silently ignored and the plugin-level default is used.
|
||||
- The override applies only to that template's runs. Other templates continue to use the plugin-level value.
|
||||
- The runtime tolerates mid-stream truncation gracefully: a stream cut off by timeout leaves a partial draft on disk with `truncated: true` flagged in the result and a Notice telling the user to re-run.
|
||||
|
||||
When to bump above the plugin-level default:
|
||||
|
||||
- `sonar-deep-research` on a multi-section analyst template (market maps, in-depth research reports, sector overviews). These routinely emit 6–8K-word bodies and the tail sections (Frontier, Adjacent Concepts, Open Questions) are the first to be cut.
|
||||
- Templates with deeply nested skeleton structure and many `[IMAGE N: …]` markers — Perplexity's image-result population time stretches the wall clock.
|
||||
- Templates that declare a large `include-sources:` block (per the [multi-stage exploration](../context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md)) when that feature ships — RAG context inflation adds prompt-processing time.
|
||||
|
||||
Market-map-profile ships with `request-timeout-ms: 2400000` (40 min) for exactly the reason above. Concept, vocabulary, source, and toolkit profiles do not declare the field and inherit the plugin-level 30-min default — they all finish comfortably under that.
|
||||
|
||||
## Settings
|
||||
|
||||
`Plugin settings → Directory templates`:
|
||||
|
|
|
|||
6
main.ts
6
main.ts
|
|
@ -330,7 +330,7 @@ Structure the article as follows:
|
|||
{ name: 'image-placement', when: 'return-images' },
|
||||
],
|
||||
directoryTemplatesFrontmatterWhitelist: ['title', 'og_description', 'tags', 'og_image'],
|
||||
directoryTemplatesRequestTimeoutMs: 600000,
|
||||
directoryTemplatesRequestTimeoutMs: 1800000,
|
||||
|
||||
// Find images for selection
|
||||
findImagesMaxImages: 3
|
||||
|
|
@ -2040,9 +2040,9 @@ class PerplexedSettingTab extends PluginSettingTab {
|
|||
|
||||
new Setting(containerEl)
|
||||
.setName('Request timeout (ms)')
|
||||
.setDesc('Maximum time to wait for the perplexity deep research response. Default 300000 (5 min).')
|
||||
.setDesc('Maximum wall-clock time to wait for a Perplexity response. Default 1800000 (30 min) — generous because deep-research runs on long analyst-grade templates routinely take 15-25 min and the $10-$50 of value per good output is worth waiting for. Individual templates may override this per-template via `request-timeout-ms:` in their cft block.')
|
||||
.addText(text => text
|
||||
.setPlaceholder('300000')
|
||||
.setPlaceholder('1800000')
|
||||
.setValue(String(this.plugin.settings.directoryTemplatesRequestTimeoutMs))
|
||||
.onChange(async (value: string) => {
|
||||
const n = parseInt(value, 10);
|
||||
|
|
|
|||
3
src/docs/templates/README.md
vendored
3
src/docs/templates/README.md
vendored
|
|
@ -17,6 +17,7 @@ ## Shipped templates
|
|||
| `vocabulary-profile.md` | `Vocabulary/**` | Definitions of terms with disambiguation through an innovation-consulting lens. |
|
||||
| `source-profile.md` | `Sources/**` | Profiles of trusted sources — books, people, channels, publications, journals, reports, events. Adapts emphasis to the source's type. |
|
||||
| `toolkit-profile.md` | `Tooling/**` | Profiles of tools, products, platforms, frameworks. |
|
||||
| `market-map-profile.md` | `lost-in-public/market-maps/**`, `market-maps/**` | Analyst-grade market-map drafts — both Known Category (e.g., Humanoid Robots) and Thesis-Driven (e.g., Neural Network Hardware as Brains for Robotics). Runs on `sonar-deep-research`. Single-stage v1; multi-stage RAG + Claude-edit pass is planned. |
|
||||
|
||||
## How a template works
|
||||
|
||||
|
|
@ -59,7 +60,7 @@ # User Notes
|
|||
### The three zones
|
||||
|
||||
1. **Frontmatter** (top, between `---` lines) — carries `title`, `applies-to-paths` (array of glob patterns), and an optional `description`. The plugin uses `applies-to-paths` to match a template to a target file.
|
||||
2. **`cft` block** (a code fence with language `cft`) — YAML config: `provider`, `model`, `search-recency`, `return-citations`, `return-images`, plus a multi-line `system:` prompt. Anything above the `cft` block is treated as documentation and dropped from the request.
|
||||
2. **`cft` block** (a code fence with language `cft`) — YAML config: `provider`, `model`, `search-recency`, `return-citations`, `return-images`, optional `request-timeout-ms:` (overrides the plugin-level *Request timeout (ms)* setting for this template — useful for long deep-research runs that need 20-40 min), plus a multi-line `system:` prompt. Anything above the `cft` block is treated as documentation and dropped from the request.
|
||||
3. **Heading skeleton** (everything between the `cft` block's closing fence and the first `***`) — the user prompt. This is the markdown structure the model fills in. Bullets under each heading are *instructions to the model*, not literal output.
|
||||
|
||||
The `***` divider terminates the user prompt. Anything below it (the User Notes zone) is for your own scratch work and never reaches the model.
|
||||
|
|
|
|||
233
src/docs/templates/market-map-profile.md
vendored
Normal file
233
src/docs/templates/market-map-profile.md
vendored
Normal file
|
|
@ -0,0 +1,233 @@
|
|||
---
|
||||
title: Market Map (Analyst Draft)
|
||||
applies-to-paths:
|
||||
- "lost-in-public/market-maps/**"
|
||||
- "market-maps/**"
|
||||
description: Generates an analyst-grade draft of a market map — either a Known Category (e.g., Humanoid Robots, Light-based Computing) or a Thesis-driven map (e.g., Neural Network Hardware as `Brains` for Robotics) that traverses known categories.
|
||||
date_created: 2026-05-26
|
||||
date_modified: 2026-05-26
|
||||
---
|
||||
|
||||
# About this template
|
||||
|
||||
Use this for files under `lost-in-public/market-maps/` (or any top-level `market-maps/`) whose body is empty or whose curated lead-in (Topics, Lighthouse Examples) has been authored but the analytical body is missing.
|
||||
|
||||
A **market map** is the draft a well-paid analyst would hand to a partner: not an encyclopedia entry, not marketing copy. It explains who is doing what in a category, why now, who funded them, how segments differ, and what the open questions are. Two flavors are supported:
|
||||
|
||||
1. **Known Category** — Humanoid Robots, Light-based Computing, Quantum Computing. The taxonomy is roughly settled; the work is enumerating innovators within established sub-segments and explaining the current frontier.
|
||||
2. **Thesis-Driven** — "Neural Network Hardware as `Brains` for Robotics." The thesis traverses multiple known categories under a hypothesis. The work is making the thesis legible, then enumerating innovators from each adjacent category that the thesis pulls in.
|
||||
|
||||
The heading skeleton works for both flavors. The model picks up the flavor from the file's `title`, `tags`, and any thesis paragraph the user has pre-authored above the body.
|
||||
|
||||
This template runs on `sonar-deep-research` and skips image embedding by design — deep research returns better citation density and analytical length but unreliable image metadata. Banner / portrait / square imagery for market maps is generated separately (Ideogram) and lives in frontmatter.
|
||||
|
||||
```cft
|
||||
provider: perplexity
|
||||
model: sonar-deep-research
|
||||
return-citations: true
|
||||
return-images: false
|
||||
# 40-minute wall-clock budget for this template specifically. Market-map deep
|
||||
# research runs routinely produce 6-8K-word drafts with 20-40 named innovators
|
||||
# across 4-8 sub-segments, and the tail of that generation (Frontier and
|
||||
# Adjacent Concepts sections) is the most cuttable under aggressive timeouts.
|
||||
# Overrides the plugin-level default. A complete run is worth $10-$50 of
|
||||
# analyst time; let it cook.
|
||||
request-timeout-ms: 2400000
|
||||
system: |
|
||||
You are writing the analyst-grade draft of a MARKET MAP titled "{{basename}}".
|
||||
|
||||
A market map is the draft a well-paid analyst hands to a partner. It is not
|
||||
an encyclopedia entry, not a marketing post, not a listicle. It explains:
|
||||
|
||||
- WHO is doing what in this market — named companies, named founders, named
|
||||
research labs, with funding stage and primary URL.
|
||||
- HOW the market segments — what natural sub-buckets exist, and which
|
||||
innovators sit in which sub-bucket.
|
||||
- WHY NOW — what changed (technology unlock, regulatory shift, capital
|
||||
flow, customer behavior) that made this market legible.
|
||||
- WHAT IS DISPUTED — where credible operators disagree about category
|
||||
boundaries, winner archetypes, or thesis viability.
|
||||
|
||||
Determine the FLAVOR of map from the title and frontmatter:
|
||||
|
||||
- KNOWN CATEGORY (e.g., "Quantum Computing is Confusing", "Humanoid Robots",
|
||||
"Agentic AI in Fintech") — the category name is settled. Your job is to
|
||||
enumerate sub-segments and innovators within each, and explain the frontier.
|
||||
- THESIS-DRIVEN (e.g., "Neural Network Hardware as Brains for Robotics",
|
||||
"Blockchain and Web3 Institutional Invasions") — a hypothesis traverses
|
||||
known categories. Your job is to make the thesis legible, then enumerate
|
||||
innovators from each adjacent category the thesis pulls in.
|
||||
|
||||
Frontmatter for "{{basename}}":
|
||||
{{frontmatter}}
|
||||
|
||||
RESEARCH DISCIPLINE:
|
||||
|
||||
- Use Perplexity's web search aggressively. For a market map, breadth of
|
||||
named entities matters more than depth on any single one.
|
||||
- For every factual claim — funding round, founding year, customer name,
|
||||
market sizing, product capability — append an inline numeric citation
|
||||
marker [1], [2], etc. corresponding to the search-result order.
|
||||
- Quote phrasing from primary sources where useful (founder interviews,
|
||||
investor blog posts, earnings notes, academic abstracts).
|
||||
- Prefer primary surfaces: company homepage, founder Twitter/X, technical
|
||||
blog posts, conference talks, investor announcements. Aggregator pages
|
||||
(Crunchbase, PitchBook summaries) are fallbacks for funding stage only.
|
||||
- Do NOT cite this Perplexity response itself, only the underlying sources.
|
||||
|
||||
EDITORIAL STANCE — attribute innovation correctly:
|
||||
|
||||
Markets are pioneered by startups, academics, research labs, and indie
|
||||
practitioners — NOT by tech giants. Training data over-represents
|
||||
incumbents. Counteract this systematically:
|
||||
|
||||
- Treat big tech (Microsoft, Google, Amazon, Apple, Meta, Oracle, Salesforce,
|
||||
IBM post-1990s, Nvidia post-2020) as ADOPTERS or POPULARIZERS in this
|
||||
market unless the entry documents an originating research-lab paper or
|
||||
a heyday-era origination story (Bell Labs, Xerox PARC, DeepMind, OpenAI's
|
||||
early years).
|
||||
- In every Lighthouse Examples sub-bucket, cap big-tech entries at 1 of
|
||||
5–10. Prefer Series A-C startups, seed-stage frontier bets,
|
||||
open-source projects, indie practitioners, research labs.
|
||||
- In Market Dynamics, name the named operators driving the curve, not
|
||||
the incumbents profiting from it.
|
||||
- Where an incumbent IS the originator, say so explicitly with the
|
||||
research paper or product release that documents the origination.
|
||||
|
||||
INNOVATOR CARD SHAPE:
|
||||
|
||||
Each named innovator under "Innovator Profiles" follows this shape exactly:
|
||||
|
||||
#### [Innovator Name](https://homepage.url)
|
||||
**Offering**: one-sentence description of what they do that is specific
|
||||
enough that a partner could repeat it back. Use product names, customer
|
||||
names, and category boundaries. Cite. [N]
|
||||
**Funding**: stage and round size if disclosed (e.g., "$30M Series A,
|
||||
DN Capital, 2024"). "Undisclosed" or "Bootstrapped" if no public data.
|
||||
**Why they matter**: one sentence on what makes them distinctive — the
|
||||
technical bet, the GTM angle, the team's background. Not marketing
|
||||
adjectives; a specific differentiator. Cite. [N]
|
||||
**Coverage**: 1-2 references in trade press, founder podcasts, or
|
||||
analyst notes if available. Format: `[Outlet, Title](url)`. Cite. [N]
|
||||
|
||||
Aim for 3-7 innovators per sub-bucket. If a sub-bucket has fewer than 3
|
||||
credible named entities, MERGE it into an adjacent sub-bucket rather than
|
||||
padding with weak entries.
|
||||
|
||||
LINKS AND WIKILINKS:
|
||||
|
||||
- For innovator names and source links, use `[Name](https://url)` form.
|
||||
- Do NOT invent `[[wikilink]]` syntax. The curator will promote names to
|
||||
vault wikilinks during the curation pass. If you happen to know a
|
||||
canonical concept this market touches (e.g., "Compliance AI", "Agentic
|
||||
Workspaces"), surface the concept name as plain text in the Adjacent
|
||||
Concepts section so the curator can wikilink it later.
|
||||
|
||||
CALIBRATION ON LENGTH:
|
||||
|
||||
This is a deep-research run. Lean long, not short. A complete market map
|
||||
is roughly 4,000-8,000 words of body, with 20-40 named innovators across
|
||||
4-8 sub-buckets, a summary table per sub-bucket where useful, and explicit
|
||||
funding-trend / adoption-pattern data in Market Dynamics. Better to over-
|
||||
enumerate and let the curator prune than to under-enumerate.
|
||||
```
|
||||
|
||||
# Market Snapshot
|
||||
|
||||
- One-paragraph italicized lede (max 2 sentences) that captures the punchy thesis of this market. Voice: the analyst opening their memo. Use markdown italics: `_..._`.
|
||||
- Then the headline stat — one quoted statistic from a credible source that signals scale or velocity, with inline citation. Format the quote as a blockquote (`> "..."`).
|
||||
- Then 2-3 sentences orienting the reader: what is this market, what is the timeframe, why is it worth a map right now.
|
||||
|
||||
# The Question this Map Answers
|
||||
|
||||
- One paragraph (3-5 sentences) stating the question this map clarifies for an operator or investor reading it.
|
||||
- If the file is a KNOWN CATEGORY map, frame the question as "what shape has this category settled into, and where is the frontier."
|
||||
- If the file is a THESIS-DRIVEN map, state the thesis explicitly in one sentence, then explain which adjacent categories the thesis traverses and why the traversal is non-obvious.
|
||||
|
||||
# Why Now
|
||||
|
||||
- 3-5 bullets, each one a specific unlock that explains why this market is mappable in the current quarter and would have been premature 18 months ago.
|
||||
- Unlock types to consider: a technical capability crossing a threshold (cost, latency, accuracy), a regulatory or standards shift, a capital-formation pattern (a fund vintage, an exit precedent), a customer-behavior shift, an open-source release that lowered the floor.
|
||||
- Cite each unlock. Where possible, quote a founder, researcher, or operator who named the unlock.
|
||||
|
||||
# Map of the Market — Sub-Segments
|
||||
|
||||
- Identify 4-8 natural sub-segments that partition this market. For a thesis-driven map, the sub-segments are the adjacent categories the thesis traverses.
|
||||
- Give each sub-segment a 1-2 sentence definition: what falls inside, what falls outside, and what distinguishes it from its neighbors.
|
||||
- This section is the TABLE OF CONTENTS for the Innovator Profiles section below. Sub-segment names here must match section headings below.
|
||||
|
||||
# Lighthouse Examples
|
||||
|
||||
- For each sub-segment, list 5-10 lighthouse innovators in `[Name](url) — one-line description` form. These are the names a partner would expect to hear in this category — recognized leaders, well-funded operators, frontier bets the analyst would brief on.
|
||||
- Group under `## <Sub-segment name>` subheadings matching the Map of the Market above.
|
||||
- This is a flat reference list. The deeper analysis lives in Innovator Profiles below.
|
||||
- Use the editorial-stance cap: at most 1 of 5-10 in any sub-bucket may be big tech.
|
||||
|
||||
# Innovator Profiles
|
||||
|
||||
For each sub-segment from the Map of the Market, produce a `## <Sub-segment name>` heading and under it, 3-7 innovator cards in the format defined in the system prompt:
|
||||
|
||||
#### [Innovator Name](https://homepage.url)
|
||||
**Offering**: ...
|
||||
**Funding**: ...
|
||||
**Why they matter**: ...
|
||||
**Coverage**: ...
|
||||
|
||||
- Order within each sub-segment from most-established to most-frontier (seed-stage / stealth at the end).
|
||||
- After the innovator cards in each sub-segment, render a single summary table with columns: `Innovator | Stage | Differentiator | Primary Customer`. The table lets a partner skim the sub-segment without reading every card.
|
||||
|
||||
# Media, Voices, and Coverage
|
||||
|
||||
- 6-12 bullets covering the publications, podcasts, YouTubers, analysts, and individual operator-thinkers who shape the conversation about this market.
|
||||
- Format: `**Name** — Platform — one-line note on their angle / why they are worth following`. Include a primary URL link.
|
||||
- Sub-group with `## Publications`, `## Podcasts & YouTube`, `## Analysts & Operator-Thinkers` if the list is long enough to warrant it.
|
||||
- Prefer specialized trade press over generalist business press. Prefer named operator-bloggers over corporate marketing surfaces.
|
||||
|
||||
# Market Dynamics
|
||||
|
||||
## Sizing and Growth
|
||||
|
||||
- 2-4 cited bullets covering: TAM / current market size, projected CAGR, the report or analyst behind each number.
|
||||
- Be skeptical of single-source sizing claims; where two credible sources disagree, surface the disagreement.
|
||||
|
||||
## Adoption Patterns and Barriers
|
||||
|
||||
- 2-4 cited bullets covering: what percentage of the addressable buyer base has adopted, what the canonical barriers are (procurement cycle, regulatory uncertainty, technical readiness, talent shortage), and where the adoption curve is bending.
|
||||
|
||||
## Capital Flow
|
||||
|
||||
- 2-4 cited bullets covering: where the funding has concentrated by sub-segment, who the active funds are (named partners where public), and any recent exit, acquisition, or IPO that reset valuation expectations in the category.
|
||||
|
||||
# Frontier and Open Questions
|
||||
|
||||
- 4-7 bullets, each one a specific open question that a partner reading this map would want the analyst to think about next.
|
||||
- Frame each as a question, not a statement: "Will agent-to-agent micropayments require a separate settlement rail, or will existing card networks absorb the use case?" not "Settlement rails are evolving."
|
||||
- Pair each question with a one-sentence note on which innovators or research streams are likely to produce the answer.
|
||||
|
||||
# Adjacent Concepts and Maps
|
||||
|
||||
- 4-8 plain-text concept names (no wikilink syntax — the curator wikilinks during curation) that an operator working in this market would want to explore next.
|
||||
- Mix of: adjacent market maps (other categories this one borders), foundational concepts (the mental models this market sits on), and vocabulary terms (the specific jargon the curator may want to define in `Vocabulary/`).
|
||||
- Format: `- <Concept Name> — one-line on why it adjoins this map`.
|
||||
|
||||
***
|
||||
|
||||
# User Notes
|
||||
|
||||
Anything below the `***` line is excluded from the request. Use this zone for:
|
||||
|
||||
- The thesis paragraph (for thesis-driven maps) you want the model to fold into the system context. To do that, move the paragraph ABOVE the `***` divider — into the Market Snapshot or Question section — before running the template.
|
||||
- Hand-curated `:::tool-showcase` blocks pointing to vault tools you want to feature.
|
||||
- Tuning notes, prior model outputs, and iteration history while you refine the template for your domain.
|
||||
|
||||
## Multi-stage roadmap (deferred)
|
||||
|
||||
This v1 template is intentionally single-stage: one Perplexity Deep Research run produces the full draft, the curator promotes Lighthouse Examples to `:::tool-showcase` blocks and `[Name](url)` references to `[[wikilink]]` references during the curation pass.
|
||||
|
||||
The deferred multi-stage version will run:
|
||||
|
||||
1. **RAG pre-flight** — pull canonical Lossless sources for this market (tools under `Tooling/`, concepts under `concepts/`, prior market maps that overlap) and feed them as context to the research stage. This eliminates the wikilink-invention problem and lets the model name actual vault entries.
|
||||
2. **Perplexity research stage** — deep research with the RAG context as a primer, producing the draft this v1 template produces.
|
||||
3. **Claude editing stage** — an editorial pass that enforces the analyst voice, prunes the over-enumeration, restructures sub-segment boundaries where the data demands, and emits the final `[[wikilink]]` form.
|
||||
|
||||
The plumbing for stage 3 (the Claude orchestrator) lives in `claudeService.ts`. The plumbing for stage 1 (RAG over the Lossless corpus) is partially built via the `chroma` MCP server — the missing piece is a per-template `rag-context:` block in the cft fence that names which collections to query and how many results to inject. Spec lives in `context-v/specs/` — open it before starting the multi-stage build.
|
||||
|
|
@ -775,12 +775,27 @@ export async function applyTemplate(
|
|||
// Set initial state before streaming begins.
|
||||
await app.vault.modify(target, initialContent);
|
||||
|
||||
// Template-level `request-timeout-ms:` in the cft block overrides the
|
||||
// plugin-level default. Deep-research templates routinely need 20-40min
|
||||
// to fully render a long analyst draft and shouldn't be capped by a
|
||||
// setting tuned for short concept-profile runs. Accept number or
|
||||
// numeric string. Ignore non-positive / non-numeric values silently.
|
||||
const cftTimeoutRaw = template.cftConfig['request-timeout-ms'];
|
||||
const cftTimeoutMs = typeof cftTimeoutRaw === 'number'
|
||||
? cftTimeoutRaw
|
||||
: typeof cftTimeoutRaw === 'string'
|
||||
? parseInt(cftTimeoutRaw, 10)
|
||||
: NaN;
|
||||
const effectiveTimeoutMs = Number.isFinite(cftTimeoutMs) && cftTimeoutMs > 0
|
||||
? cftTimeoutMs
|
||||
: settings.requestTimeoutMs;
|
||||
|
||||
const { streamed, sources, images, truncated } = await streamPerplexityToFile(
|
||||
app,
|
||||
settings.perplexityApiKey,
|
||||
settings.perplexityEndpoint,
|
||||
payload,
|
||||
settings.requestTimeoutMs,
|
||||
effectiveTimeoutMs,
|
||||
target,
|
||||
initialContent,
|
||||
isCancelled,
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import conceptProfile from '../docs/templates/concept-profile.md';
|
|||
import vocabularyProfile from '../docs/templates/vocabulary-profile.md';
|
||||
import sourceProfile from '../docs/templates/source-profile.md';
|
||||
import toolkitProfile from '../docs/templates/toolkit-profile.md';
|
||||
import marketMapProfile from '../docs/templates/market-map-profile.md';
|
||||
|
||||
import partialsReadme from '../docs/partials/README.md';
|
||||
import mermaidDisciplinePartial from '../docs/partials/mermaid-discipline.md';
|
||||
|
|
@ -30,6 +31,7 @@ const TEMPLATE_FILES: SeedFile[] = [
|
|||
{ name: 'vocabulary-profile.md', content: vocabularyProfile },
|
||||
{ name: 'source-profile.md', content: sourceProfile },
|
||||
{ name: 'toolkit-profile.md', content: toolkitProfile },
|
||||
{ name: 'market-map-profile.md', content: marketMapProfile },
|
||||
];
|
||||
|
||||
const PARTIALS_README: SeedFile = { name: 'README.md', content: partialsReadme };
|
||||
|
|
|
|||
Loading…
Reference in a new issue