diff --git a/README.md b/README.md index 0390977..8eb0643 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,20 @@ # Perplexed: AI Content Generation for Obsidian **Perplexed** is an Obsidian plugin that enables AI-powered content generation with source citations using [Perplexity](https://www.perplexity.ai/), [Anthropic Claude](https://www.anthropic.com/), [Google Gemini](https://ai.google.dev/) (with Google Search grounding), and [Perplexica / Vane](https://github.com/ItzCrazyKns/Vane) (self-hosted). This plugin brings research-grade AI capabilities directly into your Obsidian workspace, allowing you to generate well-cited content for your notes. +## πŸ’Ό For Venture Capital, Private Equity, and Equities-Trading Workflows + +Perplexed ships a set of **analyst-grade directory templates** aimed at the research deliverables a VC analyst, PE associate, equity-research analyst, or trading-desk strategist produces daily. Drop an empty file into the matching folder, run *Apply directory template to current file*, and Perplexity Deep Research returns a 6-9K-word cited analyst draft you can curate into a memo for a partner, an IC, or a portfolio review. + +| Workflow | Template | What it produces | +|---|---|---| +| **Naming the players in a category** (incumbents vs challengers vs innovators by financial stage) | `market-category-profile.md` β†’ `concepts/Market-Categories/` | Three-tier company landscape with explicit definitions: **Incumbents** (public / late-stage private / PE-owned), **Challengers** (Series C+ scale-ups, recently public), **Innovators** (Pre-Seed through Series B). Plus Why Now / What's Happening sections covering CAGR + category-creation momentum, and an Industry Coverage section sub-grouped into Market Reports (Gartner, IDC, Forrester, ABI) / Industry Articles / Financial News (Bloomberg, FT, Pitchbook). | +| **Authoring a market map** (Known Category or Thesis-Driven) | `market-map-profile.md` β†’ `lost-in-public/market-maps/` | Analyst-grade memo with 4-8 sub-segments, 20-40 named innovator cards (Offering / Funding / Why-they-matter / Coverage), Market Dynamics (Sizing / Adoption / Capital Flow), Frontier and Open Questions. Anti-incumbent editorial stance prevents big-tech over-representation. | +| **Profiling an open spec or standard** an investment thesis depends on | `standards-and-specs-profile.md` β†’ `Sources/Standards-and-Specs/` | Five-way authority typing (de-jure / consortium / vendor-led-open / community / de-facto), three-tier adoption framing with notable holdouts, named editors, stewardship-transition stories, named public critics with their arguments. | +| **Catalogue an authoritative source** (book, person, channel, report, conference) | `source-profile.md` β†’ `Sources/` | Type-aware emphasis (author / publisher / cadence / methodology), Google Books URL harvesting for books, signature-work catalog. | +| **Encyclopedia entry on a concept, pattern, or mental model** the desk repeatedly invokes | `concept-profile.md` β†’ `concepts/` | Definition, usage, history, examples, case studies. Mermaid + LaTeX rendering discipline baked in for diagrams and formulas. | + +Every analyst-grade template runs on `sonar-deep-research`, ships with idle-only timeout safety (`request-timeout-ms: 0`, per-chunk idle timer at 270s) and a 24,000-token output budget (`max-tokens: 24000`) β€” enough for the longest analyst drafts to land without silent mid-document truncation. See [Directory Templates](#directory-templates) below for the full set and the cft-block grammar for tuning your own. + ## 🎯 Key Features ![Perplexed UI Modal interface](https://i.imgur.com/jaZ4UfS.png) - **Source-Cited AI Responses**: Get AI-generated content with proper citations and references @@ -50,6 +64,7 @@ ## Network use and accounts ## πŸ“‹ Table of Contents +- [For Venture Capital, Private Equity, and Equities-Trading Workflows](#-for-venture-capital-private-equity-and-equities-trading-workflows) - [User Onboarding](#user-onboarding) - [Installation](#installation) - [Initial Setup](#initial-setup) @@ -435,9 +450,10 @@ ### Partials and preambles β€” shared guidance across templates ``` Content-Dev/ -β”œβ”€β”€ templates/ (your four profile templates) -β”œβ”€β”€ partials/ (reusable snippets: mermaid-discipline, etc.) -β”‚ └── mermaid-discipline.md +β”œβ”€β”€ templates/ (your seven profile templates) +β”œβ”€β”€ partials/ (reusable snippets included via {{include: name}}) +β”‚ β”œβ”€β”€ mermaid-discipline.md (paired BAD/GOOD examples + 6-item self-check) +β”‚ └── latex-discipline.md (Obsidian MathJax delimiters + $ escaping) └── preambles/ (auto-attached to every request as system / user messages) β”œβ”€β”€ inline-citation.md β”œβ”€β”€ image-placement.md @@ -452,16 +468,19 @@ ### Partials and preambles β€” shared guidance across templates ### Shipped templates -Four templates ship inlined into the plugin and seed into your vault on first plugin load: +Seven templates ship inlined into the plugin and seed into your vault on first plugin load: | File | Targets | Use for | |---|---|---| -| `concept-profile.md` | `concepts/**` | Encyclopedia-style entries on ideas, patterns, mental models. Anti-incumbent editorial stance baked in (tech giants treated as adopters/popularizers, not innovators, unless documented heyday-era origination supports otherwise). | +| `concept-profile.md` | `concepts/**` | Encyclopedia-style entries on ideas, patterns, mental models. Anti-incumbent editorial stance baked in (tech giants treated as adopters/popularizers, not innovators, unless documented heyday-era origination supports otherwise). Includes both `mermaid-discipline` and `latex-discipline` partials. | | `vocabulary-profile.md` | `Vocabulary/**` | Term definitions with disambiguation through an innovation-consulting lens. | | `source-profile.md` | `Sources/**` | Profiles of books, people, channels, publications, journals, reports, events β€” type-aware, with Google Books URL harvesting for books. | | `toolkit-profile.md` | `Tooling/**` | Profiles of tools, products, platforms, frameworks. | +| `market-map-profile.md` | `lost-in-public/market-maps/**`, `market-maps/**` | Analyst-grade market-map drafts β€” both Known Category (e.g., Humanoid Robots) and Thesis-Driven (e.g., Neural Network Hardware as Brains for Robotics). Runs on `sonar-deep-research` with idle-only timeout, `max-tokens: 24000`, and a 40-min absolute wall-clock ceiling. | +| `standards-and-specs-profile.md` | `Sources/Standards-and-Specs/**`, `Standards-and-Specs/**` | Analyst-grade profiles of open specs and standards. Five-way authority typing (de-jure / consortium / vendor-led-open / community / de-facto). Three-tier structural adoption framing (incumbents / challengers / innovators) plus notable holdouts. Named editors, stewardship transitions, named critics. | +| `market-category-profile.md` | `concepts/Market-Categories/**`, `Market-Categories/**` | Concept-folder reference card for a named market category. Three-tier company landscape with explicit FINANCIAL-STAGE definitions: Incumbents (public / late-stage private / PE-owned) β†’ Challengers (Series C+ scale-ups, recently public) β†’ Innovators (Pre-Seed through Series B). Separate Why Now / What's Happening sections covering CAGR + category-creation momentum. Industry Coverage sub-grouped into Market Reports / Industry Articles / Financial News. | -All four use Perplexity's `sonar-pro` (deep-research is unreliable for image return). +The first four templates use Perplexity's `sonar-pro`. The three deep-research templates (`market-map-profile`, `standards-and-specs-profile`, `market-category-profile`) use `sonar-deep-research` and declare per-template cft-block overrides for the wall-clock ceiling (`request-timeout-ms`), the per-chunk idle timer (`stream-idle-timeout-ms`), and the Perplexity output-token budget (`max-tokens: 24000`). See [`docs/directory-templates.md`](docs/directory-templates.md) for the full cft-block grammar and the diagnostic table for distinguishing wall-clock-timeout truncation from max_tokens truncation. ### Auto-seed behavior diff --git a/changelog/2026-05-26_02.md b/changelog/2026-05-26_02.md new file mode 100644 index 0000000..09597d2 --- /dev/null +++ b/changelog/2026-05-26_02.md @@ -0,0 +1,120 @@ +--- +title: "Long deep-research streams no longer get cut off at the ceiling β€” the timer now resets every time bytes arrive" +lede: "The per-chunk idle-timeout discipline the legacy modal flow has used for two iterations now governs the directory-template flow too. A healthy slow stream completes naturally; a silently-stalled one fails in seconds instead of minutes." +date_work_started: 2026-05-26 +date_work_completed: 2026-05-26 +date_created: 2026-05-26 +date_modified: 2026-05-26 +publish: true +category: Changelog +tags: + - Directory-Templates + - Streaming-Timeouts + - Deep-Research + - Idle-Timeout + - cft-Block + - Perplexity-sonar-deep-research + - Structural-Fix +authors: + - Michael Staton +augmented_with: + - Claude Code on Claude Opus 4.7 (1M context) +files_changed: + - src/services/directoryTemplateService.ts + - docs/directory-templates.md + - src/docs/templates/README.md + - src/docs/templates/market-map-profile.md + - context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md +--- + +# The structural fix the morning's market-map run earned + +## Why care? + +If you run any directory template on Perplexity β€” especially the long deep-research templates like `market-map-profile` β€” your timeouts now behave the way you'd intuit they should. **A stream that is healthily producing bytes runs as long as it needs to run.** A stream that goes silent (Perplexity rate-limit, socket close, upstream stall) surfaces the failure inside seconds, not at the end of a 30-minute ceiling. The change is invisible when everything goes well and only noticeable when something would have gone wrong β€” which is when you most want it noticeable. + +Earlier today we shipped a market-map template, watched it produce a beautiful 7,500-word draft on Humanoid Robots, and then watched the last sentence cut off mid-word because the wall-clock timer fired before the stream finished. We patched the immediate pain by raising the ceiling. **This entry is the structural fix underneath it.** + +## What's new? + +A single concept: the timer that decides "is this stream still alive?" is now armed *per chunk*, not once per request. + +- **Old behavior.** One `setTimeout(controller.abort, timeoutMs)` at fetch time. If the stream takes longer than `timeoutMs`, even by one byte, abort. If the stream goes silent for 27 of the 30 allotted minutes, wait the full 30 anyway before noticing. +- **New behavior.** Each `reader.read()` is raced against a fresh timer. As long as bytes keep arriving, the timer is cleared and re-armed. The stream is only killed if it goes **quiet** for `idleMs` β€” by default 270 seconds for deep-research models, 90 seconds for everything else. + +Two cft-block keys now govern timeouts (both optional): + +```cft +provider: perplexity +model: sonar-deep-research +stream-idle-timeout-ms: 270000 # per-chunk idle (default already 270s for deep-research) +request-timeout-ms: 2400000 # absolute wall-clock ceiling (40 min) +system: | + ... +``` + +`stream-idle-timeout-ms:` is the new primary safety. `request-timeout-ms:` is the legacy key, now repurposed as an opt-in absolute ceiling β€” set to `0` to disable and rely on idle-only. + +## How it works + +Two different streaming primitives have lived in this codebase since the directory-template flow forked from the legacy `PerplexityModal` flow. The legacy modal moved to per-chunk idle-timeout discipline two iterations ago after the same problem hit users there first. The directory-template flow never received the backport β€” until today. + +The pattern, ported from `perplexityService.ts:659-668` into `streamPerplexityToFile`: + +```ts +const readWithIdleTimeout = (): Promise> => { + let timer: number | undefined; + const timeout = new Promise((_, reject) => { + timer = activeWindow.setTimeout(() => { + reject(new Error(`stream went idle for ${idleMs / 1000}s (likely API stall, rate limit, or socket close)`)); + }, idleMs); + }); + return Promise.race([reader.read(), timeout]).finally(() => { + if (timer !== undefined) activeWindow.clearTimeout(timer); + }); +}; + +// inside the read loop: +({ value, done } = await readWithIdleTimeout()); +``` + +`Promise.race([reader.read(), timeout])` resolves with whichever wins. If a byte arrives, the `.finally()` clears the timer before it can fire and the loop continues. If the timer fires first, it rejects, the surrounding catch sets `truncated = true`, the existing cleanup pipeline flushes whatever already arrived, and the run returns with a partial-but-cited draft on disk and a Notice telling the user to re-run. + +The `AbortController` stays β€” both as the cancel mechanism for the user-initiated *Cancel* command, and as the abort target for the optional wall-clock ceiling. The two timers are complementary: idle handles "is this stream alive right now?" and ceiling handles "regardless, do not let this run forever." + +### The two pathologies the old design handled poorly + +**Shape 1 β€” slow but healthy stream.** Deep-research generations on long templates sustain a slow trickle of tokens for tens of minutes. The old wall-clock cap killed them at the ceiling regardless of whether they were still producing. The idle timer lets them complete as long as bytes keep arriving inside the idle window. (The ceiling is still there if you want a hard cap β€” it's just not what's making the moment-to-moment safety decision anymore.) + +**Shape 2 β€” silently stalled stream.** Conversely, a stream may go quiet at minute 3 (Perplexity rate-limit, socket close, upstream stall) and the old wall-clock cap wouldn't notice until minute 30. The user stared at an empty file for 27 unnecessary minutes. The idle timer surfaces the failure within `idleMs` seconds β€” fast feedback when something is genuinely wrong. + +Both shapes are real. The wall-clock pattern punished healthy-but-slow while tolerating stalled-but-silent. The idle-timeout pattern inverts both β€” slow-but-healthy completes; stalled-but-silent fails fast. + +## Migration notes + +**Nothing breaks for existing templates.** The `request-timeout-ms:` key is still honored where declared; it just means "absolute wall-clock ceiling" now instead of "the only timer at all." Templates that didn't declare it still inherit `settings.requestTimeoutMs` as their ceiling. `market-map-profile.md` keeps its `2400000` (40 min) value β€” under the new semantics it's the ceiling on top of the 270s idle timer, which is exactly what an analyst-grade deep-research budget should look like. + +To get **truly unbounded healthy streams** (idle-only safety, no ceiling), set `request-timeout-ms: 0` in the cft block, or set the plugin-level *Request timeout (ms)* setting to `0`. The idle timer will catch silent stalls fast either way. + +To override **the idle timer itself**, declare `stream-idle-timeout-ms: ` in the cft block. The defaults (270s deep, 90s normal) match the legacy modal flow and have been load-bearing there for two iterations, so most templates won't need to touch this. + +## What's deferred + +The [open issue](../context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md) listed four follow-ups. This entry closes items 1 and 2 (the port itself, and the dual-key naming decision). Two remain explicitly deferred: + +- **Cross-service audit.** The same wall-clock pathology may live in the Gemini service, the LM Studio service, and the Claude streaming flows. The idle-timeout discipline should be the house style across all of them; the audit is its own change. +- **Settings-pane exposure of idle defaults.** Today the 270s/90s values are hardcoded in `streamPerplexityToFile` (matching `perplexityService.ts`). Exposing them as plugin settings is a follow-up if we ever need to tune them without a code change. + +The deep-research detection is by model-name regex (`/deep-research/i`) β€” the same shape `perplexityService.ts` uses. Robust enough for today's Perplexity model lineup; revisit if Perplexity ever ships a long-running model under a different naming convention. + +## Why this matters for the multi-stage exploration + +The [Multi-Stage Cooperative Claude + Perplexity with RAG exploration](../context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md) explicitly listed this fix as upstream of the multi-stage spec. Reason: the eventual Claude editorial pass on a 7-8K-word draft is itself a long generation that would hit the same wall-clock cliff. With per-chunk idle-timeout discipline now load-bearing in the directory-template flow, the editorial pass can reuse the same `readWithIdleTimeout` primitive when it lands. The multi-stage spec is now unblocked on this dimension. + +## Files touched + +- `src/services/directoryTemplateService.ts` β€” `streamPerplexityToFile` signature changed from `timeoutMs: number` to `timeouts: { idleMs: number; ceilingMs: number }`. `readWithIdleTimeout()` helper added. Callsite in `applyTemplate` resolves both keys from cft config with sensible fallbacks (idle defaults from deep-research detection, ceiling defaults from `settings.requestTimeoutMs`, explicit `0` disables ceiling). +- `docs/directory-templates.md` β€” *Per-template timeout override* section rewritten as *Per-template timeout overrides* (plural), now documents both keys, the two pathologies the idle timer fixes, and the migration semantics. +- `src/docs/templates/README.md` β€” cft-key list updated to mention both keys. +- `src/docs/templates/market-map-profile.md` β€” inline comment on the `request-timeout-ms` declaration rewritten to explain it as the absolute ceiling on top of the 270s idle timer. +- `context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md` β€” referenced as the canonical issue this closes. diff --git a/changelog/2026-05-27_01.md b/changelog/2026-05-27_01.md new file mode 100644 index 0000000..176278d --- /dev/null +++ b/changelog/2026-05-27_01.md @@ -0,0 +1,216 @@ +--- +title: "Two analyst-grade templates land β€” spec profiles and market-category profiles β€” plus two quiet bugs the testing pass caught" +lede: "Perplexed now ships templates for profiling open specs (with five-way authority typing and three-tier adoption) and named market categories (with explicit financial-stage tiers for incumbents, challengers, and innovators). The third shipped thing was a max_tokens override we built after diagnosing why the first real spec profile stopped halfway through with no warning at all. The fourth was a rendering-discipline upgrade β€” strengthened mermaid rules plus new LaTeX rules β€” after diagnosing that the vault's concept-profile template had drifted from the bundled source and was generating diagrams with no rendering rules in scope at all." +date_work_started: 2026-05-27 +date_work_completed: 2026-05-27 +date_created: 2026-05-27 +date_modified: 2026-05-27 +at_semantic_version: 0.3.0 +publish: true +category: Changelog +tags: + - Directory-Templates + - Standards-and-Specs + - Market-Categories + - Three-Tier-Framing + - Deep-Research + - cft-Block + - Perplexity-sonar-deep-research + - Max-Tokens + - Silent-Truncation + - Mermaid-Rendering + - LaTeX-Rendering + - Partials + - Vault-Drift +authors: + - Michael Staton +augmented_with: + - Claude Code on Claude Opus 4.7 (1M context) +files_changed: + - src/docs/templates/standards-and-specs-profile.md + - src/docs/templates/market-category-profile.md + - src/docs/templates/concept-profile.md + - src/docs/templates/README.md + - src/docs/templates/market-map-profile.md + - src/docs/partials/mermaid-discipline.md + - src/docs/partials/latex-discipline.md + - src/services/templateSeederService.ts + - src/services/directoryTemplateService.ts + - docs/directory-templates.md +--- + +# Two new templates, two quiet bug fixes, and the rendering-discipline upgrade we needed to catch them + +## Why care? + +If you spend any time figuring out which open specs your team should care about β€” MCP versus A2A versus the latest agent-to-agent protocol someone is pitching at a conference β€” Perplexed now produces an analyst-grade profile of any one of them in a single run. Same workflow you already use for concepts and market maps: drop a file into `Sources/Standards-and-Specs/`, give it the spec's name, run *Apply directory template to current file*, and you get back a 6-9K-word draft with the spec's full authorship and stewardship history, three tiers of named implementations (incumbents, challengers, innovators), the publicly named critics, the political fault lines, and the open frontier. + +If your day involves naming the companies inside a market category β€” "who are the actual players in agentic workspaces, sorted by stage" β€” there's now a separate template for that too, in `concepts/Market-Categories/`. It uses the same three-tier framing the spec-profile template uses, but tuned for FINANCIAL stage rather than adoption tier: Incumbents (the legacy public-and-PE-owned crowd you can't ignore), Challengers (the well-funded scale-ups eating share), Innovators (Pre-Seed through Series B). And unlike the market-map template β€” which caps big-tech mentions to counteract training-data bias β€” the market-category template *wants* the big tech named in Incumbents, because for a category profile, knowing the incumbents is the point. + +If you author any directory template that runs on `sonar-deep-research`, the third shipped thing matters more than either template. We found and fixed a silent-truncation pathology where Perplexity's default `max_tokens` cap (~8192 tokens, ~6K words) was ending the stream **cleanly** mid-template β€” full sources footer, no Notice, no truncation flag β€” making it look like a healthy completion that just happened to be missing the back half. The new `max-tokens:` cft override lets any template that legitimately wants 6-9K words of body bump the budget; both shipped deep-research templates now declare it at 24,000. + +If you've been frustrated that Perplexity-authored Mermaid diagrams come back broken roughly 4-of-5 times β€” bare `&` characters inside labels, unquoted multi-word labels with parentheses, `\n` instead of `
` for line breaks β€” the fourth shipped thing is for you. The visible fix was strengthening the `mermaid-discipline` partial with paired BAD/GOOD examples and adding a new `latex-discipline` partial for Obsidian's MathJax delimiters. The deeper fix was diagnostic: the vault's `concept-profile.md` had drifted from the bundled source and was missing the `{{include: mermaid-discipline}}` directive entirely β€” meaning every concept-profile run had been generating Mermaid with no rendering rules in scope at all. Strengthening the partial wouldn't have changed a thing until the include was wired up. + +## What's new? + +Four shipped behaviors: + +``` +zz-cf-lib/ +β”œβ”€β”€ templates/ +β”‚ β”œβ”€β”€ concept-profile.md +β”‚ β”œβ”€β”€ vocabulary-profile.md +β”‚ β”œβ”€β”€ source-profile.md +β”‚ β”œβ”€β”€ toolkit-profile.md +β”‚ β”œβ”€β”€ market-map-profile.md +β”‚ β”œβ”€β”€ standards-and-specs-profile.md (new β€” open specs profiler) +β”‚ └── market-category-profile.md (new β€” financial-stage tiers) +└── ... +``` + +- **`standards-and-specs-profile.md`** β€” auto-seeded into `Content-Dev/Templates/` and re-seedable. Matches `Sources/Standards-and-Specs/**` and `Standards-and-Specs/**`. Runs on `sonar-deep-research`. +- **`market-category-profile.md`** β€” auto-seeded too. Matches `concepts/Market-Categories/**` and `Market-Categories/**`. Runs on `sonar-deep-research`. +- **`max-tokens:` cft-block override** β€” new per-template knob that overrides Perplexity's default output-token cap. Number or numeric string, non-positive ignored. Both the new templates ship with `max-tokens: 24000`; the existing `market-map-profile.md` was also bumped to 24000 (it was masked from this issue by the wall-clock cap firing first; with the recent `request-timeout-ms: 0` change, it would have started hitting the same silent-truncation cliff on the next thorough run). +- **Rendering-discipline partials β€” strengthened and expanded.** The `mermaid-discipline.md` partial doubled in length (250 tokens β†’ 440) with paired BAD/GOOD examples, the `&` escape rule, the explicit `\n` vs `
` ban with example, a six-item self-check before emit, and a simplify-rather-than-break ethos. A new `latex-discipline.md` partial (~165 tokens) covers the three Obsidian MathJax pitfalls (`$$...$$` not `\[...\]`, `$...$` not `\(...\)`, escape literal `\$` in prose). Both wired into `concept-profile.md` (the bundled source AND the vault copy, which had drifted) immediately after the "render a mermaid codefence here" instruction so the rules are in scope at the moment of generation. + +The `directory-templates.md` doc gained a *Per-template `max-tokens:` override* section with the diagnostic table operators need to tell wall-clock truncation apart from max-tokens truncation β€” a vocabulary that didn't exist before today and that we needed in the moment to debug our first real spec profile. + +## How the standards-and-specs profile thinks + +A spec profile is the document an innovation consultant hands to a partner who has just asked "should we care about this spec?" It is not implementer documentation, not a tutorial, not marketing for the spec. The template's system prompt is explicit about this β€” conformance matrices, MUST/SHOULD/MAY discipline, and wire-format details are out of scope; stewardship, adoption tiers, political fault lines, and named critics are in scope. + +The template handles five authority types and tunes its treatment of every downstream section based on which one the spec is: + +| Type | Examples | What changes | +|---|---|---| +| **De-jure** | W3C, IETF/RFC, ISO, IEEE, ECMA, NIST | Editors lifted from spec cover; working-group archives cited | +| **Industry consortium** | Khronos, Linux Foundation, OpenJS, OASIS, CNCF | Member companies named with their voting weight | +| **Vendor-led-open** | MCP (Anthropic), OpenAPI's early Wordnik years | Both originating team AND cross-vendor contributors named | +| **Community** | llms.txt (Jeremy Howard), AGENTS.md (post-OpenAI handoff) | Originator's identity is the political center | +| **De-facto** | README convention, package.json shape, the curl interface | Dominant implementation defines the spec; stewardship shifts implicitly | + +The skeleton has twelve sections (Snapshot β†’ The Question this Spec Answers β†’ Identity & Status β†’ Why It Matters β†’ Position in the Ecosystem Stack β†’ Lineage β†’ Governance & Stewardship β†’ Adoption by Tier β†’ Critique & Open Disputes β†’ Frontier & Open Questions β†’ Media, Voices, and Coverage β†’ Adjacent Specs and Standards). Two pieces of structural discipline are load-bearing: + +**One β€” created-by and maintained-by are first-class identity.** The Snapshot lede ends with an explicit one-line callout: `**Created by** {name(s)} ({year}) Β· **Maintained by** {name(s)} Β· **Type:** {authority-type}`. For specs where the creator and current steward differ, the Governance & Stewardship section gets a dedicated paragraph telling the transition story β€” when it happened, what triggered it, what changed in governance pace afterward. AGENTS.md (OpenAI β†’ community via Sourcegraph), OpenAPI (Wordnik β†’ SmartBear β†’ Linux Foundation), HTTP (Tim Berners-Lee at CERN β†’ IETF/W3C), JSON (Crockford β†’ IETF + Ecma, two parallel stewards) β€” all named explicitly in the system prompt as the kind of transition the template wants surfaced with care. + +**Two β€” the three-tier adoption framing is STRUCTURAL.** The Adoption section partitions named implementations into Incumbents (the canonical/reference implementations and the dominant deployed OSS, plus the commercial implementations from market leaders β€” big tech belongs here, do NOT suppress), Challengers (production-grade alternatives from mid-sized companies and well-funded startups), and Innovators (early-stage / experimental / research implementations exploring the spec's edges). Each tier gets 4-8 named entries plus 2-3 deeper implementation cards for the most strategically significant. After the three tiers, Notable Holdouts surfaces the orgs that explicitly declined, forked, or are running incompatible alternatives β€” often as informative as the adopters about where the spec made enemies. + +## How the market-category profile differs + +Same three-tier shape, completely different definitions. The market-category template is for a concept-folder entry (`concepts/Market-Categories/`), not a published memo. The reader is looking for a reference card on a named market β€” Humanoid Robots, Agentic Workspaces, Specialized Computing for Edge Robotics β€” and wants to know who plays in it. + +The three tiers are sorted by **financial stage**, not by adoption: + +- **Incumbents** β€” large public companies, tech giants, late-stage private (post-Series E or $1B+ valuation with 10+ years of operation), PE-owned behemoths. Legacy footprint with huge market presence. The companies an enterprise buyer already has a contract with, even if not in this category yet. +- **Challengers** β€” well-funded scale-ups (Series C through pre-IPO, or recently public via SPAC/IPO with under 7 years of operation). Rapidly growing, hype-driven, capital position to credibly threaten incumbent share. +- **Innovators** β€” Pre-Seed through Series B funded startups. Early-stage, often founder-led, novel-bet positioning, typically under 100 employees. + +The editorial stance flips relative to the market-map template. Market maps cap big tech at 1 of 5-10 to counteract training-data bias β€” for an analyst memo, the named operators driving the curve matter more than the incumbents profiting from it. Market-CATEGORY profiles invert this: the goal is the *full* financial landscape of a named market, and naming the incumbents IS the goal. Both editorial stances are right for their respective documents; the templates make the disagreement explicit so neither prompt has to apologize for it. + +The template also splits **Why Now** and **What's Happening** into separate sections because they answer different questions: + +- **Why Now** = what FORCES aligned to make this category coherent right now? (Enabling conditions: technological unlocks, regulatory shifts, capital-formation patterns, customer-behavior shifts.) +- **What's Happening** = what is the current MOMENTUM? (CAGR figures with named reports, category-creation events like a defining IPO or acquisition, where capital is concentrating by tier.) + +The reader leaves Why Now understanding the conditions and What's Happening understanding the velocity. And the Industry Coverage and Market Data section is sub-grouped explicitly: Market Reports (Gartner, IDC, Forrester, ABI Research, McKinsey, Frost & Sullivan…) β†’ Industry Articles (specialized trade press, operator-bloggers) β†’ Financial News (Bloomberg, FT, WSJ, Pitchbook, Crunchbase News, dealroom.co). + +## The max-tokens story β€” diagnosing a silent failure + +Our first real run of the new standards-and-specs template was a profile of ARM (the architecture, not the company; the model classified it correctly as VENDOR-LED-OPEN). The output looked beautifully clean: full think-output at the top, Snapshot through the start of the Adoption section, all 50 cited sources rendered correctly in the footer. No truncation Notice. No "stream went idle" warning. Just… missing six of the fourteen template sections (Challengers, Innovators, Notable Holdouts, Critique, Frontier, Media, Adjacent). + +This was hostile in a way the wall-clock issue from last week never was. The Humanoid Robots truncation that motivated yesterday's [`request-timeout-ms:` override and idle-timeout port](./2026-05-26_01.md) ended mid-sentence at "R" β€” obviously a stream abort. The ARM run ended at a clean H2 section break with the full sources footer rendered. Every diagnostic signal pointed to "healthy completion" β€” except eight of fourteen sections were missing. + +The cause was Perplexity's default `max_tokens` cap (~8,192 tokens, ~6,000 words for `sonar-deep-research`). The stream completed cleanly via `finish_reason: "length"` β€” the model self-rationed its output budget across the skeleton, ran out, and wrapped up with a clean tail. The plugin's existing `streamPerplexityToFile` cleanup pipeline saw a clean stream end, rendered the sources footer, and stamped frontmatter β€” exactly as designed for a healthy completion, because from the stream's perspective it WAS a healthy completion. + +The fix: + +```ts +// In buildPayload (directoryTemplateService.ts:451): +const maxTokensRaw = cfg['max-tokens']; +const maxTokens = typeof maxTokensRaw === 'number' + ? maxTokensRaw + : typeof maxTokensRaw === 'string' + ? parseInt(maxTokensRaw, 10) + : NaN; +if (Number.isFinite(maxTokens) && maxTokens > 0) { + payload.max_tokens = maxTokens; +} +``` + +Templates can now declare `max-tokens: 24000` (or any positive value) in their cft block to lift the cap. Both the new shipped templates declare 24,000 β€” generous enough that a thorough 6-9K-word body lands with headroom for the three-tier adoption section, deeper implementation cards, named critics, and the frontier. The market-map template was bumped to 24,000 too, because it was previously masked from this issue by the wall-clock cap firing first; with the recent `request-timeout-ms: 0` change disabling that, it would have hit this silent-truncation cliff on the next thorough run. + +The new doc section in `directory-templates.md` includes the diagnostic table operators need to distinguish the two pathologies: + +| Symptom | Cause | +|---|---| +| Mid-sentence cutoff (literally ends with a partial word or trailing punctuation); no sources footer; user sees "stream went idle" or no Notice at all | Wall-clock timer (`request-timeout-ms:`) or idle timer (`stream-idle-timeout-ms:`) fired during streaming | +| Clean section-end cutoff (last byte is a paragraph break or full sentence); **sources footer renders correctly with all citations**; no Notice | Perplexity `max_tokens` cap; stream completed via `finish_reason: "length"` | + +The wall-clock and idle timers are streaming-time controls; `max-tokens` is an output-budget control. They're independent. A template can hit either, neither, or both. Set all three knobs generously for thorough deep-research templates and the safety mechanisms compose without interfering. + +## The rendering-discipline fix β€” and the vault-drift bug it surfaced + +The other silent failure we caught today was different in kind from the max_tokens one but identical in flavor: a problem that LOOKS like a model output issue but is actually a configuration issue one layer down. Mermaid charts in concept-profile runs were coming back broken roughly 4 out of 5 times. Common failure modes: unquoted multi-word labels with parentheses (`A[Raw inputs (text, audio)]`), bare `&` characters inside labels, `\n` instead of `
` for line breaks, subgraphs with spaces in titles but no quoted display title. Each broken diagram meant another round-trip to Claude to repair before the entry could ship. + +The instinctive fix was to strengthen the rendering rules in the prompt. The Claude Code shell drafted an excellent BAD/GOOD-paired set of rules β€” about 350 tokens, more concrete than the existing abstract-rule version, with a five-item self-check before emit and an explicit "simplify the labels rather than emit a broken diagram" ethos. Drafting models follow rules better when each rule comes with a BAD β†’ GOOD pair than when stated abstractly; that part of the diagnosis was sound. + +But the deeper diagnosis was that the rules weren't the bottleneck at all: + +**The vault's `concept-profile.md` had drifted from the bundled source and was missing the `{{include: mermaid-discipline}}` directive entirely.** The model was generating mermaid with no rendering rules in scope at all. Strengthening the partial wouldn't have changed a thing until the include was wired up in the running prompt. + +This is the kind of bug a long-running plugin generates when the seeded templates are user-editable and re-seed only fills in MISSING files: as the bundled template improves over time, the vault copy stays frozen at whatever version was originally seeded. The improvements never reach the running prompt. We saw the same shape earlier in this conversation β€” the `request-timeout-ms` and `max-tokens` knobs both required vault-copy updates separate from the bundled-source updates, because the vault templates had user edits that the seeder won't overwrite. + +The combined fix: + +- **`mermaid-discipline.md` partial β€” strengthened.** Original ~250 tokens, now ~440. Kept the broader special-character list and shape rules from the existing version (which were more comprehensive than what the shell drafted β€” `=`, `+`, backtick, brackets, braces, angle brackets, pipe, hash, LaTeX), added the BAD/GOOD pair format throughout (which is what actually shifts model behavior), added the `&` escape rule, the explicit `\n` vs `
` ban with example, the six-item self-check before emit, and the simplify-rather-than-break guidance. Per the *no lazy outs* discipline: no escape hatch like "describe the diagram in prose if you're unsure" β€” the simplify-the-labels guidance forces a working diagram with shortened text rather than a broken one with the labels the model wanted. +- **`latex-discipline.md` partial β€” new.** ~165 tokens. Covers the three Obsidian MathJax pitfalls: `$$...$$` for block math (not `\[...\]`), `$...$` for inline (not `\(...\)`), and escape literal `\$` in prose to avoid accidental inline-math spans where two unescaped dollar amounts in the same paragraph get parsed as a math delimiter pair. +- **Wired both into `concept-profile.md`** β€” in both the bundled source AND the vault copy, immediately after the "render a mermaid codefence here" instruction so the rules are in scope at the moment of generation. The placement matters: rules need to be in scope BEFORE generation, not in a post-hoc "now check your work" section, because drafting models ignore later self-check instructions more often than they violate earlier in-scope rules. + +The vault-drift insight generalizes beyond this fix. Any partial or template improvement we ship needs both a bundled-source update AND a manual vault-copy update for users who've already seeded the folder. The re-seed button doesn't help β€” it only writes missing files, not updated ones. The long-term fix is probably a "diff and patch user files" mode for re-seed (or a per-file content-hash that flags when the user's copy diverges from the shipped version); the short-term fix is that every partial or template change ships with an explicit vault-copy step in the release notes. + +**One follow-up the diagnosis raises:** rendering-rule partials might belong as system-level **preambles** rather than user-prompt **partials**. Preambles get auto-attached to every request and live in the model's persistent-discipline scope rather than appearing inline next to template content. The current partial-include pattern requires every template author to remember to include the rules in every template that emits diagrams β€” and the vault-drift problem we just diagnosed is precisely the case where that discipline fails. Promoting to a preamble would make the rules apply to every run automatically, including the cases where `sonar-deep-research` decides to render a diagram spontaneously without the template asking for one. Deferred β€” we want a few cycles of evidence that the strengthened partial actually moves the 4/5 broken rate before adding the preamble plumbing. + +## A different editorial discipline for each template + +The three deep-research templates now in the shipped set β€” market-map, standards-and-specs, market-category β€” share a common technical substrate (sonar-deep-research, idle-only timeout, max-tokens 24000, return-images false) and a common structural posture (named entities everywhere, inline citations, anti-padding discipline, deep-research analyst voice). But their editorial discipline diverges in ways that matter: + +- **Market-map** caps big tech at 1-of-5-10 in any sub-bucket to counteract training-data bias. The analyst's job is to name the named operators driving the curve. +- **Standards-and-specs** names big tech freely in the Incumbents tier (large dominant implementations) and aggressively in the Innovators tier (where the next extensions come from). Suppression would distort the implementation landscape. +- **Market-category** names big tech as the FIRST move in Incumbents because for a category profile, knowing the incumbents IS the work product. Suppression would be lying about what an enterprise buyer sees. + +Each template's system prompt states its editorial stance explicitly so a future template-modifier doesn't accidentally copy the wrong one across templates. This is the kind of thing a single comment in a single template can prevent days of "why did the market-map start naming Microsoft six times" debugging. + +## Files touched + +- `src/docs/templates/standards-and-specs-profile.md` β€” new. Twelve-section skeleton, five-way authority typing, three-tier structural adoption framing, named editors / stewardship-transition discipline, named-critics framing. `max-tokens: 24000`, `request-timeout-ms: 0`. +- `src/docs/templates/market-category-profile.md` β€” new. Ten-section skeleton with explicit financial-stage tier definitions, separate Why Now / What's Happening sections, sub-grouped Industry Coverage (Market Reports / Industry Articles / Financial News). Same plumbing. +- `src/docs/templates/concept-profile.md` β€” wired in `{{include: latex-discipline}}` alongside the existing `{{include: mermaid-discipline}}`, both immediately after the "render a mermaid codefence here" instruction so the rendering rules land in scope at the moment of generation. +- `src/docs/templates/market-map-profile.md` β€” added `max-tokens: 24000` (previously masked from this issue by the wall-clock cap; needed now that `request-timeout-ms: 0` disables that cap). +- `src/docs/templates/README.md` β€” added both new templates to the shipped-templates table; updated the cft-key list to mention the new `max-tokens:` key. +- `src/docs/partials/mermaid-discipline.md` β€” rewritten. ~250 tokens β†’ ~440 tokens. Paired BAD/GOOD examples throughout, `&` escape rule, explicit `\n` vs `
` ban, six-item self-check before emit, simplify-rather-than-break ethos. Existing broader character list and shape rules preserved. +- `src/docs/partials/latex-discipline.md` β€” new. ~165 tokens covering Obsidian MathJax pitfalls (block vs inline delimiters, escaping literal `$` in prose). +- `src/services/templateSeederService.ts` β€” registered both new templates and the new latex partial so they auto-seed into vaults and are bundled into `main.js`. +- `src/services/directoryTemplateService.ts` β€” added `max_tokens?: number` to the `PerplexityPayload` interface and `max-tokens` resolution + payload-setting in `buildPayload` (function at line 451; resolution roughly 15 lines). +- `docs/directory-templates.md` β€” added a *Per-template `max-tokens:` override* section with the wall-clock-vs-max-tokens diagnostic table; updated the cft-key list at the top. + +Bundle size: `main.js` went from 325 KB to 351 KB across the day (the two new template files + the rewritten/new partials account for most of the increase). Both repo and vault copies deploy to identical timestamps via the existing dev wire. + +## What's deferred + +Three follow-ups surfaced today, all worth their own future entries: + +- **The `cf_finish_reason` frontmatter stamp.** A small (10-line) addition that would write Perplexity's `finish_reason` value into the target file's frontmatter on every run. Would have made the ARM diagnostic instant ("oh, `cf_finish_reason: length`, that's a max-tokens issue, not a wall-clock issue") instead of an investigation. Deferred only because the user wanted to get the two new templates out first. +- **A "Continue directory template on current file" command.** For runs that get cut off by `max_tokens` (or by user cancellation), a resume command would read the existing body, identify which template sections are present versus missing via skeleton diff, and re-prompt the model with only the missing sections plus a URL-keyed citation merge to keep `[N]` markers coherent across the two runs. Designed but not built; the design lives in this entry's conversation thread and will be promoted to a `context-v/specs/` doc if the demand materializes. +- **README documentation correctness.** The shipped-templates README still says "first template wins" when multiple templates match a path. The actual behavior (per `main.ts:1239`) is that the plugin opens a picker modal with all matching templates and the user chooses. The README hasn't been updated; today's matching scenario (`concepts/Market-Categories/` files match both `concept-profile` and `market-category-profile`) made the discrepancy visible. +- **Promote rendering-discipline partials to system preambles.** The current partial-include pattern relies on every template author remembering to wire in the rules β€” and the vault-drift bug we caught today is precisely the case where that discipline fails. Preambles auto-attach to every request and live in persistent-discipline scope, which would catch the case where `sonar-deep-research` decides to render a diagram spontaneously without the template asking for one. Deferred until we have a few cycles of evidence that the strengthened partial actually moves the 4/5 broken-mermaid rate. +- **A "diff and patch user files" mode for re-seed.** Today's vault-drift diagnosis surfaced a long-running plugin pathology β€” re-seed only writes missing files, so improvements to bundled templates and partials never reach users who've already seeded the folder. A diff-and-patch mode (with explicit per-file confirmation) would close this loop. Filed as a follow-up because the short-term workaround β€” manually copy each updated partial/template to the vault β€” is fine for a single-user dev workflow but won't scale to community-plugin distribution. + +## References + +- [Yesterday's market-map template ship + per-template timeout override](./2026-05-26_01.md) β€” the precedent this entry builds on for shipping a deep-research template + the cft-knob pattern. +- [The structural idle-timeout fix](./2026-05-26_02.md) β€” the port that made `request-timeout-ms: 0` safe to set and made today's `max-tokens` diagnosis tractable (without idle-timeout discipline, we'd have been chasing two truncation pathologies simultaneously). +- [`docs/directory-templates.md`](../docs/directory-templates.md) β€” the new *Per-template `max-tokens:` override* section with the diagnostic table. +- [`src/docs/templates/standards-and-specs-profile.md`](../src/docs/templates/standards-and-specs-profile.md) and [`src/docs/templates/market-category-profile.md`](../src/docs/templates/market-category-profile.md) β€” the two new templates as shipped. +- [`src/docs/partials/mermaid-discipline.md`](../src/docs/partials/mermaid-discipline.md) β€” the strengthened partial with paired BAD/GOOD examples and the six-item pre-emit self-check. +- [`src/docs/partials/latex-discipline.md`](../src/docs/partials/latex-discipline.md) β€” the new Obsidian MathJax partial. +- The Lossless Group's [`open-specs-and-standards` study](https://github.com/Lossless-Group/open-specs-and-standards) β€” the 17-spec curated reference collection that informed the standards-and-specs template's authority-typing taxonomy and stewardship-transition framing. diff --git a/changelog/releases/0.3.0.md b/changelog/releases/0.3.0.md new file mode 100644 index 0000000..8ad03dd --- /dev/null +++ b/changelog/releases/0.3.0.md @@ -0,0 +1,176 @@ +--- +title: "Perplexed 0.3.0 β€” the analyst-grade release: three deep-research templates for VC / PE / equities workflows, and the four infrastructure fixes they needed to ship" +lede: "0.3.0 turns Perplexed into a serious analyst tooling layer. Three new deep-research templates (market-map, standards-and-specs, market-category) produce 6-9K-word cited analyst drafts in a single run. Underneath them, four infrastructure fixes β€” a per-template wall-clock override, an idle-timeout structural port, a max_tokens output-budget knob, and a strengthened mermaid + new LaTeX rendering discipline β€” turn deep-research from a brittle 'might work' workflow into something you can put real analyst time behind." +date_authored_initial_draft: 2026-05-27 +date_authored_current_draft: 2026-05-27 +date_first_published: 2026-05-27 +date_last_updated: 2026-05-27 +at_semantic_version: 0.3.0 +status: Published +category: Release +tags: + - Release-Narrative + - Directory-Templates + - Deep-Research + - Market-Maps + - Standards-and-Specs + - Market-Categories + - Three-Tier-Framing + - Streaming-Timeouts + - Idle-Timeout + - Max-Tokens + - Silent-Truncation + - Mermaid-Rendering + - LaTeX-Rendering + - Partials + - Vault-Drift + - Obsidian-Community-Plugin + - Venture-Capital + - Private-Equity + - Equities-Trading +authors: + - Michael Staton +augmented_with: Claude Code (Opus 4.7, 1M context) +release_tag: "0.3.0" +prior_release_tag: "0.2.1" +--- + +## Why care? + +If your work involves naming the actual companies in a market category, mapping who's playing where in an emerging market, or profiling the open specs and protocols an investment thesis depends on β€” **0.3.0 is the release where Perplexed becomes a serious analyst tooling layer**, not just a notes-with-citations plugin. + +Three new directory templates do the bulk of the work. Each produces a 6-9K-word cited analyst draft in a single Perplexity Deep Research run. Each was designed for the kind of document a VC analyst, PE associate, or equities-trading desk strategist hands to a partner or IC: + +- **Market-category profiles** name the companies in a category, sorted into three explicit financial-stage tiers β€” Incumbents (public / late-stage private / PE-owned), Challengers (Series C+ scale-ups), Innovators (Pre-Seed through Series B) β€” with separate Why Now / What's Happening sections covering CAGR and category-creation momentum, and a fully-sourced Industry Coverage section sub-grouped into Market Reports (Gartner, IDC, Forrester) / Industry Articles / Financial News (Bloomberg, FT, Pitchbook). +- **Market-map drafts** produce the analyst memo itself β€” 4-8 sub-segments, 20-40 named innovator cards with funding stage and lead investor, Market Dynamics with cited sizing data and capital concentration, and a Frontier section with open questions framed as questions. +- **Standards-and-specs profiles** turn an open protocol or convention into a strategic-context document β€” five-way authority typing (de-jure / consortium / vendor-led-open / community / de-facto), three-tier structural adoption framing (incumbents / challengers / innovators) plus notable holdouts, named editors, stewardship-transition stories, named public critics with their arguments. The framing surfaces exactly what an analyst needs to know: is this protocol surviving its originating vendor, who has it captured, who's working to break the capture. + +If you author any directory template that runs on `sonar-deep-research`, the four infrastructure fixes that made these templates ship reliably matter more than any single template. Each fix solved a class of "deep-research silently produces broken output" failure mode that would otherwise have kept the analyst-grade workflow brittle. + +If you ever ran a long template and got a draft cut off mid-sentence, the streaming-timeout refactor fixes that β€” the timer that decides "is this stream still alive?" is now armed per-chunk, not once per request, so a healthy slow stream completes naturally and a silently-stalled one fails in seconds. If you ever ran a long template and got a draft that looked complete but was missing the back half of the section skeleton, the new `max-tokens:` override fixes that β€” Perplexity's default output cap was silently truncating thorough drafts by ending the stream cleanly mid-template. If you've been frustrated that Perplexity-authored Mermaid diagrams come back broken 4-of-5 times, the strengthened mermaid partial (plus a new LaTeX partial for Obsidian MathJax) addresses that β€” provided your vault's `concept-profile.md` is wired to include them, which the release notes here will help you verify. + +## What's new? + +``` +zz-cf-lib/ +β”œβ”€β”€ templates/ +β”‚ β”œβ”€β”€ concept-profile.md +β”‚ β”œβ”€β”€ vocabulary-profile.md +β”‚ β”œβ”€β”€ source-profile.md +β”‚ β”œβ”€β”€ toolkit-profile.md +β”‚ β”œβ”€β”€ market-map-profile.md (new in 0.3.0) +β”‚ β”œβ”€β”€ standards-and-specs-profile.md (new in 0.3.0) +β”‚ └── market-category-profile.md (new in 0.3.0) +β”œβ”€β”€ partials/ +β”‚ β”œβ”€β”€ mermaid-discipline.md (strengthened in 0.3.0 β€” ~250 β†’ ~440 tokens) +β”‚ └── latex-discipline.md (new in 0.3.0) +└── preambles/ + β”œβ”€β”€ inline-citation.md + β”œβ”€β”€ image-placement.md + └── research-framing.md +``` + +Three new templates, three new cft-block knobs, one new partial, one strengthened partial: + +- **`market-map-profile.md`** β€” analyst-grade market-map drafts in Known Category (Humanoid Robots, Quantum Computing) or Thesis-Driven (Neural Network Hardware as Brains for Robotics) shape. Matches `lost-in-public/market-maps/**` or `market-maps/**`. Ships with `sonar-deep-research`, `request-timeout-ms: 2400000` (40-min ceiling), `max-tokens: 24000`. +- **`standards-and-specs-profile.md`** β€” open-spec/standard profiles with five-way authority typing and three-tier adoption framing. Matches `Sources/Standards-and-Specs/**`. Ships with `sonar-deep-research`, `request-timeout-ms: 0` (idle-only safety), `max-tokens: 24000`. +- **`market-category-profile.md`** β€” concept-folder reference card for a named market category with three financial-stage tiers. Matches `concepts/Market-Categories/**`. Ships with `sonar-deep-research`, `request-timeout-ms: 0`, `max-tokens: 24000`. +- **`request-timeout-ms:` cft override** β€” per-template absolute wall-clock ceiling that wins over the plugin-level setting. Set to `0` to disable and rely on idle-only safety. +- **`stream-idle-timeout-ms:` cft override** β€” per-template per-chunk idle timer (default 270s for deep-research, 90s otherwise). The primary safety mechanism after the structural port from the legacy modal flow. +- **`max-tokens:` cft override** β€” per-template Perplexity output-token budget override. Lifts the silent ~8K-token default ceiling that was clean-truncating thorough drafts. Both new deep-research templates and the market-map template now declare 24,000. +- **Strengthened `mermaid-discipline.md` partial** β€” paired BAD/GOOD examples throughout, the `&` escape rule, explicit `\n` vs `
` ban, six-item self-check before emit, simplify-rather-than-break ethos. Existing broader special-character list and shape rules preserved. +- **New `latex-discipline.md` partial** β€” Obsidian MathJax delimiters (`$$...$$` for block, `$...$` for inline, NOT `\[...\]` / `\(...\)`), plus `\$` escaping for literal dollar amounts to avoid accidental inline-math spans. + +The plugin-level default request timeout was also bumped from 600,000 ms (10 min) to 1,800,000 ms (30 min). Settings-pane description rewrote to call out the cost framing β€” "$10-$50 of analyst time per good output is worth waiting for." + +## The streaming-timeout story β€” wall-clock to per-chunk idle + +The first thing 0.3.0 fixes is also the most invisible. Until 0.2.x, the directory-template runtime armed a single `setTimeout(controller.abort, timeoutMs)` at fetch time. If the stream took longer than the ceiling, even by one byte, abort. If the stream went silent for 27 of 30 allotted minutes, wait the full 30 anyway before noticing. + +We hit both pathologies in the same week. The first real run of `market-map-profile` on a Humanoid Robots map produced ~7,500 words then cut off mid-sentence at "R" in the Frontier section because the deep-research run legitimately needed longer than the ceiling. Two days later, the first real run of `standards-and-specs-profile` on the ARM architecture stopped at a clean section break with all 50 citations rendered correctly β€” and was missing six of fourteen template sections. Same template, two different failure modes, neither one obvious from the output until you knew what to look for. + +The fixes ran in two waves: + +**Wave 1 β€” pressure-relief valves.** Per-template `request-timeout-ms:` cft override, plugin-level default bumped 10 β†’ 30 min. The market-map template declares 40 min. Bought headroom; didn't fix the structural problem. + +**Wave 2 β€” port the idle-timeout discipline from the legacy modal flow.** The `PerplexityModal` flow had moved to per-chunk idle-timeout discipline two iterations ago β€” each `reader.read()` raced against a fresh timer; as long as bytes keep arriving, the timer is cleared and re-armed; only a stream that goes *quiet* for `idleMs` is killed. The directory-template flow was forked from an earlier iteration and never received the backport. 0.3.0 ports it. The wall-clock timer becomes an optional belt-and-suspenders ceiling; per-chunk idle (270s for deep-research, 90s otherwise) is the primary safety. + +The result: healthy slow streams complete naturally regardless of how long they take. Stalled streams fail in seconds, not at the ceiling. Setting `request-timeout-ms: 0` in a cft block disables the ceiling entirely and relies on idle-only safety β€” the recommended setting for the two new analyst-grade templates. + +## The max_tokens story β€” when "healthy completion" lies + +The second silent failure was different in flavor. The ARM spec-profile run completed cleanly β€” full sources footer, all 50 citations, no Notice, no truncation flag, clean section-end cutoff. Every diagnostic signal pointed to "healthy completion." Eight of fourteen sections were missing. + +The cause: Perplexity's default `max_tokens` cap (~8,192 tokens, ~6,000 words for `sonar-deep-research`). The stream completed via `finish_reason: "length"` β€” the model self-rationed across the skeleton, ran out of output-token budget, and wrapped up with a clean tail. The plugin's cleanup pipeline saw a clean stream end, rendered the sources footer, stamped frontmatter β€” exactly as designed for a healthy completion, because from the stream's perspective it WAS a healthy completion. + +The fix is a new `max-tokens:` cft override that passes through to Perplexity's `max_tokens` parameter. Both new deep-research templates and the market-map template declare `max-tokens: 24000` (~18K-word budget). The new section of `docs/directory-templates.md` includes the diagnostic table operators need to distinguish the two pathologies: + +| Symptom | Cause | +|---|---| +| Mid-sentence cutoff (literally ends with a partial word); no sources footer; user sees "stream went idle" or no Notice at all | Wall-clock or idle timer fired during streaming | +| Clean section-end cutoff; **sources footer renders correctly with all citations**; no Notice | Perplexity `max_tokens` cap; stream completed via `finish_reason: "length"` | + +The wall-clock, idle, and max-tokens knobs are independent. A template can hit any of them, all of them, or none. The three new templates set all three generously, and the cleanup pipeline behaves correctly for whichever (if any) fires. + +## The rendering-discipline story β€” and the vault-drift bug it surfaced + +The third silent failure was a model output issue layered on top of a configuration issue. Concept-profile runs were producing broken Mermaid diagrams roughly 4 of 5 times β€” unquoted multi-word labels with parentheses, bare `&` characters, `\n` instead of `
`, subgraphs with spaces in titles but no quoted display title. Each broken diagram meant another round-trip to Claude to repair before the entry could ship. + +The instinctive fix was to strengthen the rendering rules. The bundled `mermaid-discipline.md` partial already existed and was already pulled into `concept-profile` via `{{include: mermaid-discipline}}` β€” but the existing rules were stated abstractly and didn't include paired BAD/GOOD examples (which is what actually shifts model behavior). + +The deeper diagnosis showed the rules weren't the bottleneck at all: + +**The vault's `concept-profile.md` had drifted from the bundled source and was missing the `{{include: mermaid-discipline}}` directive entirely.** The model was generating Mermaid with no rendering rules in scope at all. Strengthening the partial wouldn't have changed a thing until the include was wired up in the running prompt. + +This is the kind of bug a long-running plugin generates when seeded templates are user-editable and re-seed only fills in MISSING files: as the bundled template improves over time, the vault copy stays frozen at whatever version was originally seeded. The improvements never reach the running prompt. + +The 0.3.0 fix is twofold: + +- **`mermaid-discipline.md`** rewritten with paired BAD/GOOD examples throughout, the `&` escape rule, the explicit `\n` vs `
` ban, a six-item self-check before emit, and a simplify-rather-than-break ethos (no escape hatch like "describe in prose if unsure" β€” the discipline forces a working diagram with shortened text rather than a broken one with the labels the model wanted). +- **`latex-discipline.md`** added as a new partial covering Obsidian MathJax delimiters and `$` escaping in prose. + +Both wired into `concept-profile.md` in both the bundled source AND the vault copy. **If you've been seeding the templates folder before today, run a quick check after upgrading**: open `Content-Dev/Templates/concept-profile.md` and verify it contains both `{{include: mermaid-discipline}}` and `{{include: latex-discipline}}` directives in the Defining and Describing section. If they're missing (as they were in our vault), add them manually β€” the re-seed button won't update files that already exist. + +The vault-drift insight generalizes beyond this fix and is filed as a 0.4.x follow-up: any partial or template improvement we ship needs both a bundled-source update AND a manual vault-copy update for users who've already seeded the folder. A "diff and patch user files" mode for re-seed would close the loop properly. + +## Editorial discipline by template β€” three different stances on big tech + +The three new deep-research templates share a common technical substrate (sonar-deep-research, idle-only timeout, max-tokens 24000, return-images false) and a common structural posture (named entities everywhere, inline citations, anti-padding discipline). Their editorial discipline diverges in ways that matter for analyst output quality: + +- **Market-map** caps big tech at 1-of-5-10 in any sub-bucket to counteract training-data bias. The analyst's job is to name the named operators driving the curve, not the incumbents profiting from it. +- **Standards-and-specs** names big tech freely in the Incumbents tier (large dominant implementations) and aggressively in the Innovators tier (where the next extensions come from). Suppression would distort the implementation landscape. +- **Market-category** names big tech as the FIRST move in Incumbents because for a category profile, naming the incumbents IS the work product. Suppression would be lying about what an enterprise buyer sees. + +Each template's system prompt states its editorial stance explicitly so a future template-modifier doesn't accidentally copy the wrong one across templates. + +## Upgrade notes + +**Vault template drift β€” the one you have to check by hand.** The biggest gotcha in 0.3.0 is the rendering-discipline fix described above. If your `Content-Dev/Templates/concept-profile.md` was seeded before today, it may be missing the `{{include: mermaid-discipline}}` and `{{include: latex-discipline}}` directives β€” meaning even after upgrading the plugin and getting the strengthened partials in the vault, the running prompt won't include them until you wire the includes into the template manually. The README's Directory Templates section has the placement; insert both directives immediately after the "render a `mermaid` codefence here" instruction in the *Defining and Describing* section. Five seconds of editing; saves you from continued 4-of-5 broken Mermaid. + +**Re-seed pulls in the new templates and the new latex partial.** The seven shipped templates plus the two partials are all bundled into `main.js`. After upgrading to 0.3.0, click *Settings β†’ Directory templates β†’ Re-seed templates* to write the new files into your vault (the button only adds files whose filenames don't already exist; nothing existing gets overwritten). You'll see `market-map-profile.md`, `standards-and-specs-profile.md`, `market-category-profile.md` appear in your templates folder, and `latex-discipline.md` appear in your partials folder. + +**Backward compatibility.** All cft-block additions (`request-timeout-ms`, `stream-idle-timeout-ms`, `max-tokens`) are opt-in. Templates that don't declare them inherit sensible defaults. The four pre-0.3.0 templates (concept, vocabulary, source, toolkit) run unchanged. + +**Plugin-level default request timeout bumped 10 min β†’ 30 min.** Visible in *Settings β†’ Directory templates β†’ Request timeout (ms)*. If you had previously set a custom value, it's preserved. + +## What's deferred + +Three follow-ups surfaced during the 0.3.0 work cycle and are worth tracking for 0.4.x: + +- **A `cf_finish_reason` frontmatter stamp** on every template run. Would have made the ARM max-tokens diagnosis instant ("oh, `cf_finish_reason: length`, that's a max-tokens issue, not a wall-clock issue") instead of an investigation. +- **A "Continue directory template on current file" command.** For runs that get cut off by `max_tokens` or by user cancellation, a resume command would read the existing body, identify which template sections are present vs missing via skeleton diff, and re-prompt the model with only the missing sections plus a URL-keyed citation merge to keep `[N]` markers coherent across the two runs. +- **A "diff and patch user files" mode for re-seed.** The vault-drift bug we caught today is structural: re-seed only writes missing files, so improvements to bundled templates and partials never reach users who've already seeded the folder. A diff-and-patch mode (with explicit per-file confirmation) would close the loop. + +## Engineering changelog references + +The full work that landed in 0.3.0 is captured across three engineering-changelog entries: + +- [`changelog/2026-05-26_01.md`](../2026-05-26_01.md) β€” the market-map template + the per-template `request-timeout-ms:` override + the plugin-level default bump. +- [`changelog/2026-05-26_02.md`](../2026-05-26_02.md) β€” the structural idle-timeout port from `perplexityService.ts:659-668` into `streamPerplexityToFile`. Closed the "Wall-clock timeout cuts off long deep-research streams" issue. +- [`changelog/2026-05-27_01.md`](../2026-05-27_01.md) β€” the two new templates (standards-and-specs, market-category), the `max-tokens` cft override, the strengthened mermaid + new LaTeX partials. + +The 0.2.1 β†’ 0.3.0 commit range will appear in the GitHub release page when this version is tagged. + +## Compatibility + +`minAppVersion: 1.8.10` β€” unchanged from 0.2.1. Desktop-only (`isDesktopOnly: true`) β€” unchanged. No new external dependencies; the three new templates and two partials are bundled into `main.js` at build time. diff --git a/context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md b/context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md index b43340f..cd8ec6b 100644 --- a/context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md +++ b/context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md @@ -3,13 +3,14 @@ lede: "The directory-template runtime caps every stream by total wall-clock duration, but the legacy modal flow already moved to per-chunk idle-timeout discipline two iterations ago β€” and the discrepancy is now actively truncating analyst-grade market-map drafts mid-sentence." date_created: 2026-05-26 date_modified: 2026-05-26 +date_resolved: 2026-05-26 authors: - Michael Staton augmented_with: - Claude Opus 4.7 (1M context) -semantic_version: 0.0.0.1 +semantic_version: 0.0.0.2 type: issue -status: open +status: resolved target_repo: perplexed tags: - Issue-Resolution @@ -103,12 +104,21 @@ ## Proposed structural fix The diff is moderate β€” `streamPerplexityToFile` is ~140 lines today; the refactor touches roughly the first 40 of those (the timer setup and the `reader.read()` call inside the loop). The post-stream cleanup pipeline (`wrapThinkBlocks`, `processContentWithImages`, `buildSourcesFooter`) is unaffected. -## Open items before this becomes a spec +## Resolution (2026-05-26) -- [ ] Decide whether the cft-block key stays named `request-timeout-ms:` (which becomes a misnomer once idle-timeout is what's actually applied) or migrates to `stream-idle-timeout-ms:`. Compatibility-alias plan if the latter. -- [ ] Decide whether to retain a wall-clock absolute ceiling alongside the idle timeout, and if so what the value is (60 min? 120 min? infinite with a Notice that surfaces "this run has been going for X min" past a threshold?). -- [ ] Audit other long-stream callsites in the codebase that may have the same wall-clock pathology (Gemini service, LM Studio service, Claude service streaming flows) β€” the idle-timeout discipline should be the house style across all of them. -- [ ] Confirm the model-name-based deep-research detection (`/deep-research/i` regex on the resolved model) is robust against future Perplexity model name changes; consider exposing the idle-timeout values as plugin settings so they can be tuned without a code change. +The structural fix shipped the same day as the issue was filed. See [`changelog/2026-05-26_02.md`](../../changelog/2026-05-26_02.md) for the full ship note. + +What landed: + +- [x] **The port itself.** `streamPerplexityToFile` now wraps each `reader.read()` in a `readWithIdleTimeout()` race, ported from `perplexityService.ts:659-668`. Signature changed from `timeoutMs: number` to `timeouts: { idleMs: number; ceilingMs: number }`. AbortController retained for cancel + ceiling. +- [x] **Dual-key naming decision.** Kept `request-timeout-ms:` as the legacy key (now semantically the *absolute wall-clock ceiling*) and added `stream-idle-timeout-ms:` as the new key (the per-chunk idle timer, primary safety). No rename, no compatibility alias needed β€” existing template values for `request-timeout-ms:` still work and now mean exactly what their name suggests. +- [x] **Ceiling-vs-idle defaults.** Idle defaults match the legacy modal flow: 270s for deep-research models (`/deep-research/i` on resolved model name), 90s otherwise. Ceiling defaults to `settings.requestTimeoutMs` (30 min); explicit `0` in either settings or cft disables the ceiling entirely. + +Deferred (now their own follow-ups, not blocking): + +- [ ] **Cross-service audit.** The same wall-clock pathology may live in the Gemini service, the LM Studio service, and the Claude streaming flows. The idle-timeout discipline should be the house style across all of them. Track as its own issue. +- [ ] **Settings-pane exposure of idle defaults.** Today the 270s/90s values are hardcoded in `streamPerplexityToFile` (matching `perplexityService.ts`). Exposing them as plugin settings is a follow-up if we ever need to tune them without a code change. +- [ ] **Robustness of deep-research detection.** `/deep-research/i` against the resolved model name is the current heuristic β€” same as `perplexityService.ts`. Revisit if Perplexity ever ships a long-running model under a different naming convention. ## Related diff --git a/docs/directory-templates.md b/docs/directory-templates.md index d233131..70781c4 100644 --- a/docs/directory-templates.md +++ b/docs/directory-templates.md @@ -79,7 +79,7 @@ # User Notes ### The three zones 1. **Frontmatter** (between `---` lines at the top) β€” carries `title`, `applies-to-paths` (array of globs), and an optional `description`. The runtime uses `applies-to-paths` to match a template to a target file. -2. **`cft` block** (a code fence with language `cft`) β€” YAML carrying `provider`, `model`, optional `search-recency`, `return-citations`, `return-images`, optional `request-timeout-ms:` (per-template wall-clock override β€” see *Per-template timeout override* below), and a multi-line `system:` prompt. Everything **above** the `cft` block is dropped from the request. +2. **`cft` block** (a code fence with language `cft`) β€” YAML carrying `provider`, `model`, optional `search-recency`, `return-citations`, `return-images`, optional `stream-idle-timeout-ms:` and `request-timeout-ms:` (per-template timeout overrides β€” see *Per-template timeout overrides* below), optional `max-tokens:` (per-template output-token budget β€” see *Per-template max-tokens override* below), and a multi-line `system:` prompt. Everything **above** the `cft` block is dropped from the request. 3. **Heading skeleton** β€” the markdown structure between the `cft` block's closing fence and the first `***` divider. This becomes the user prompt. Bullets under each heading are *instructions to the model*, not literal output. Everything **below** the first `***` is excluded from the request. ### Interpolation tokens @@ -200,33 +200,84 @@ ## Commands --- -## Per-template timeout override +## Per-template timeout overrides -The runtime applies a wall-clock timeout to every Perplexity stream via an `AbortController` (see [`directoryTemplateService.ts`](../src/services/directoryTemplateService.ts) `streamPerplexityToFile`). The plugin-level default (*Plugin settings β†’ Directory templates β†’ Request timeout (ms)*) is **30 min** (1,800,000 ms) β€” generous because most templates are well under that ceiling, and the few that aren't (the analyst-grade deep-research templates) shouldn't be cut off in the middle of a draft worth $10-$50 of analyst time. +The runtime applies **two complementary timers** to every Perplexity stream (see [`directoryTemplateService.ts`](../src/services/directoryTemplateService.ts) `streamPerplexityToFile`): -Templates can override the plugin-level default by declaring `request-timeout-ms:` inside their `cft` block: +| Timer | What it does | Default | +|---|---|---| +| **Stream-idle timeout** (per-chunk) | Each `reader.read()` is raced against a fresh timer. As long as bytes keep arriving, the timer is cleared and re-armed β€” the stream only dies if it goes **quiet** for the idle duration. | `270_000` ms (4.5 min) for deep-research models (matched by `/deep-research/i` against the resolved model name); `90_000` ms (1.5 min) otherwise. Matches the legacy modal flow in `perplexityService.ts:659`. | +| **Wall-clock ceiling** (absolute) | An optional `AbortController` deadline armed once at fetch time. Belt-and-suspenders backstop. | Plugin-level `settings.requestTimeoutMs` (default **30 min**). Set to **0** in settings to disable; the idle timer alone is responsible for safety. | + +The idle timer is the primary safety mechanism. It catches the two pathologies that a single wall-clock timer handles poorly: + +- **Silently stalled stream** (Perplexity rate-limit, socket close, upstream stall): the idle timer surfaces the failure within `idleMs` seconds rather than making the user wait the full ceiling. +- **Slow-but-healthy stream**: as long as some byte arrives every `idleMs` seconds, the stream completes naturally β€” it does not get cut off mid-sentence at the ceiling regardless of whether it's still producing. + +Templates can override either or both via their `cft` block: ```cft provider: perplexity model: sonar-deep-research -request-timeout-ms: 2400000 # 40 min +stream-idle-timeout-ms: 270000 # per-chunk idle (default already 270s for deep-research) +request-timeout-ms: 2400000 # absolute ceiling (40 min) system: | ... ``` Override semantics: -- A numeric value or a numeric string both work; non-positive / non-numeric values are silently ignored and the plugin-level default is used. -- The override applies only to that template's runs. Other templates continue to use the plugin-level value. -- The runtime tolerates mid-stream truncation gracefully: a stream cut off by timeout leaves a partial draft on disk with `truncated: true` flagged in the result and a Notice telling the user to re-run. +- A numeric value or a numeric string both work. +- For `stream-idle-timeout-ms:`: non-positive / non-numeric values fall through to the model-class default (270s deep, 90s normal). +- For `request-timeout-ms:`: an explicit `0` disables the ceiling entirely (idle-only). Non-numeric or missing falls through to `settings.requestTimeoutMs`. +- Overrides apply only to that template's runs. +- The runtime tolerates mid-stream truncation gracefully: a stream killed by either timer leaves a partial draft on disk with `truncated: true` flagged in the result and a Notice telling the user to re-run. -When to bump above the plugin-level default: +When to override the defaults: -- `sonar-deep-research` on a multi-section analyst template (market maps, in-depth research reports, sector overviews). These routinely emit 6–8K-word bodies and the tail sections (Frontier, Adjacent Concepts, Open Questions) are the first to be cut. -- Templates with deeply nested skeleton structure and many `[IMAGE N: …]` markers β€” Perplexity's image-result population time stretches the wall clock. -- Templates that declare a large `include-sources:` block (per the [multi-stage exploration](../context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md)) when that feature ships β€” RAG context inflation adds prompt-processing time. +- `sonar-deep-research` on a multi-section analyst template (market maps, in-depth research reports, sector overviews) routinely emits 6–8K-word bodies. The 270s idle default lets the slow trickle complete naturally β€” no override needed for the *idle* key in most cases. +- Bump `request-timeout-ms:` (or set to `0`) when you want truly unbounded healthy runs and trust the idle timer to catch stalls. +- Lower `stream-idle-timeout-ms:` for short non-deep-research templates that should fail fast (the 90s default already does this for them). +- Templates that declare a large `include-sources:` block (per the [multi-stage exploration](../context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md)) when that feature ships β€” RAG context inflation adds first-byte latency; consider bumping `stream-idle-timeout-ms:` if first-byte routinely exceeds the default. -Market-map-profile ships with `request-timeout-ms: 2400000` (40 min) for exactly the reason above. Concept, vocabulary, source, and toolkit profiles do not declare the field and inherit the plugin-level 30-min default β€” they all finish comfortably under that. +Market-map-profile ships with `request-timeout-ms: 2400000` (40 min ceiling) as a safety cap on absolutely-runaway runs; the per-chunk idle timer (270s, inherited) is what actually keeps the stream healthy. Concept, vocabulary, source, and toolkit profiles declare neither key and inherit `settings.requestTimeoutMs` for the ceiling and the 90s idle default β€” they all finish comfortably under both. + +--- + +## Per-template `max-tokens:` override + +Perplexity's default output-token cap silently truncates long analyst-grade templates by ending the stream **cleanly** with `finish_reason: "length"` mid-skeleton. The symptom is hostile: the run looks like a healthy completion (sources footer renders, no Notice, no truncation warning), but the back half of the section skeleton never appears. Default is approximately 8,192 tokens for `sonar-deep-research` (~6,000 English words). + +Templates can override the default by declaring `max-tokens:` inside their `cft` block: + +```cft +provider: perplexity +model: sonar-deep-research +max-tokens: 24000 # ~18K-word budget +system: | + ... +``` + +Override semantics: + +- A numeric value or a numeric string both work; non-positive / non-numeric values are silently ignored and Perplexity's default applies. +- The override is per-template. Other templates continue to use Perplexity's default. +- Perplexity may itself cap the request below your declared `max-tokens:` based on the model's context window minus the prompt length. If you request more than the model can deliver, Perplexity returns the model's actual cap; your override is the ceiling you're willing to accept. + +When to bump above the default: + +- **Any analyst-grade template that asks for >6K words of body.** `market-map-profile` and `standards-and-specs-profile` both ship with `max-tokens: 24000` for this reason. +- **Templates with deeply nested skeletons** (3+ heading levels) β€” the model rations its token budget across the skeleton, and a deep skeleton means more sections competing for the same budget. +- **Templates that demand multiple cards per section** (e.g., the three-tier adoption skeleton in `standards-and-specs-profile` with deeper implementation cards). Each card takes ~150-300 tokens; 15-30 cards Γ— 200 tokens = 3,000-6,000 tokens just for the card sections. + +**The diagnostic to distinguish max-tokens truncation from wall-clock truncation:** + +| Symptom | Cause | +|---|---| +| Mid-sentence cutoff (literally ends with a partial word or trailing punctuation); no sources footer; user sees "stream went idle" or no Notice at all | Wall-clock timer (`request-timeout-ms:` ceiling) or idle timer (`stream-idle-timeout-ms:`) fired during streaming | +| Clean section-end cutoff (last byte is a paragraph break or full sentence); **sources footer renders correctly with all citations**; no Notice | Perplexity `max_tokens` cap; the stream completed cleanly via `finish_reason: "length"` | + +The wall-clock and idle timers are streaming-time controls; `max-tokens` is an output-budget control. They are independent β€” a template can hit either, neither, or both. Set both knobs generously for thorough deep-research templates. ## Settings diff --git a/manifest.json b/manifest.json index e63f209..f3064f5 100644 --- a/manifest.json +++ b/manifest.json @@ -1,7 +1,7 @@ { "id": "perplexed", "name": "Perplexed", - "version": "0.2.1", + "version": "0.3.0", "minAppVersion": "1.8.10", "description": "Generate source-cited research content from Perplexity, Anthropic Claude, Google Gemini, Perplexica (now Vane), or local LM Studio β€” directly into your notes.", "author": "The Lossless Group", diff --git a/package.json b/package.json index 82fd8b2..141f6f3 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "perplexed", - "version": "0.2.1", + "version": "0.3.0", "description": "Generate source-cited research content through prompts and templates. From Perplexity, Anthropic Claude, Google Gemini, Perplexica (now Vane), or local LM Studio β€” directly into your notes.", "main": "main.js", "scripts": { diff --git a/release-notes/0.3.0.md b/release-notes/0.3.0.md new file mode 100644 index 0000000..cfaf8e4 --- /dev/null +++ b/release-notes/0.3.0.md @@ -0,0 +1,59 @@ +**0.3.0 turns Perplexed into a serious analyst tooling layer for Venture Capital, Private Equity, and equities-trading workflows. Three new directory templates produce 6-9K-word cited analyst drafts in a single Perplexity Deep Research run. Underneath them, four infrastructure fixes turn deep-research from a brittle "might work" workflow into something you can put real analyst time behind.** + +## Why care? + +If your work involves naming the actual companies in a market category, mapping who's playing where in an emerging market, or profiling the open specs an investment thesis depends on β€” 0.3.0 is the release where Perplexed becomes a serious analyst tooling layer, not just a notes-with-citations plugin. + +**Three new directory templates.** Each produces a 6-9K-word cited analyst draft in a single Perplexity Deep Research run. Each was designed for the kind of document a VC analyst, PE associate, or equities-trading desk strategist hands to a partner or IC: + +- **Market-category profiles** (`concepts/Market-Categories/`) name the companies in a category, sorted into three explicit financial-stage tiers β€” Incumbents (public / late-stage private / PE-owned), Challengers (Series C+ scale-ups), Innovators (Pre-Seed through Series B) β€” with separate *Why Now* / *What's Happening* sections covering CAGR and category-creation momentum, and a fully-sourced Industry Coverage section sub-grouped into Market Reports (Gartner, IDC, Forrester, ABI) / Industry Articles / Financial News (Bloomberg, FT, Pitchbook). +- **Market-map drafts** (`lost-in-public/market-maps/`) produce the analyst memo itself β€” 4-8 sub-segments, 20-40 named innovator cards with funding stage and lead investor, Market Dynamics with cited sizing data and capital concentration, and a Frontier section with open questions framed as questions. +- **Standards-and-specs profiles** (`Sources/Standards-and-Specs/`) turn an open protocol or convention into a strategic-context document β€” five-way authority typing (de-jure / consortium / vendor-led-open / community / de-facto), three-tier structural adoption framing plus notable holdouts, named editors, stewardship-transition stories, named public critics with their arguments. Surfaces what an analyst needs to know about a spec: is it surviving its originating vendor, who has it captured, who's working to break the capture. + +**Four infrastructure fixes** β€” each one solving a class of "deep-research silently produces broken output" failure mode that would otherwise have kept the analyst-grade workflow brittle: + +- **Per-chunk idle-timeout discipline.** The timer that decides "is this stream still alive?" is now armed per-chunk, not once per request. A healthy slow stream completes naturally regardless of total duration; a silently-stalled one fails in seconds, not at the wall-clock ceiling. Ported from the legacy `PerplexityModal` flow into the directory-template runtime. +- **Per-template `request-timeout-ms:` cft override.** Becomes the optional absolute wall-clock ceiling (belt-and-suspenders backstop on top of the idle timer). Set to `0` to disable entirely and rely on idle-only safety β€” the recommended setting for the two new analyst-grade templates. +- **Per-template `max-tokens:` cft override.** Fixes the silent-truncation pathology where Perplexity's default output cap (~8,192 tokens, ~6K words for `sonar-deep-research`) was ending the stream cleanly mid-template β€” full sources footer, no Notice, no truncation flag, looking like a healthy completion that just happened to be missing the back half. The three deep-research templates ship with `max-tokens: 24000` (~18K-word budget). +- **Strengthened rendering-discipline partials.** `mermaid-discipline.md` rewritten with paired BAD/GOOD examples throughout, the `&` escape rule, an explicit `\n` vs `
` ban, a six-item self-check before emit, and a simplify-rather-than-break ethos. New `latex-discipline.md` partial covers Obsidian MathJax delimiters and `$` escaping. Both wired into `concept-profile.md`. + +## Diagnostic vocabulary for streaming failures + +| Symptom | Cause | +|---|---| +| Mid-sentence cutoff (ends with a partial word); no sources footer; "stream went idle" Notice or no Notice at all | Wall-clock timer (`request-timeout-ms:`) or idle timer (`stream-idle-timeout-ms:`) fired during streaming | +| Clean section-end cutoff; **sources footer renders correctly with all citations**; no Notice | Perplexity `max_tokens` cap; stream completed via `finish_reason: "length"` | + +The three streaming knobs (`request-timeout-ms`, `stream-idle-timeout-ms`, `max-tokens`) are independent β€” a template can hit any of them, all of them, or none. The three new deep-research templates set all three generously so analyst-grade runs land reliably. + +## Editorial discipline diverges by template + +The three new deep-research templates share a common technical substrate but apply different editorial stances on big tech: + +- **Market-map** caps big tech at 1-of-5-10 in any sub-bucket to counteract training-data bias β€” the analyst's job is to name the named operators driving the curve. +- **Standards-and-specs** names big tech freely in the Incumbents tier (large dominant implementations) and aggressively in the Innovators tier (where the next extensions come from). Suppression would distort the implementation landscape. +- **Market-category** names big tech as the FIRST move in Incumbents because for a category profile, knowing the incumbents IS the work product. + +Each template's system prompt states its editorial stance explicitly so a future template-modifier doesn't accidentally cross-pollinate the wrong stance across templates. + +## Upgrade notes + +**Vault template drift β€” the gotcha worth five seconds of attention.** If your `Content-Dev/Templates/concept-profile.md` was seeded before today, it may be missing the `{{include: mermaid-discipline}}` and `{{include: latex-discipline}}` directives. Without them, every concept-profile run generates Mermaid with no rendering rules in scope at all, producing broken diagrams 4-of-5 times. Open the file, find the *Defining and Describing* section, and insert both directives immediately after the "render a `mermaid` codefence here" instruction. The re-seed button won't add them β€” it only writes missing files. + +**Re-seed pulls in the new templates and the new partial.** The seven shipped templates plus the two partials are bundled into `main.js`. After upgrading, click *Settings β†’ Directory templates β†’ Re-seed templates* to write the new files into your vault. The button only adds files whose filenames don't already exist; nothing existing gets overwritten. You'll see `market-map-profile.md`, `standards-and-specs-profile.md`, `market-category-profile.md` appear in templates and `latex-discipline.md` appear in partials. + +**Backward compatibility.** All cft-block additions are opt-in. Templates that don't declare the new knobs inherit sensible defaults. The four pre-0.3.0 templates (concept, vocabulary, source, toolkit) run unchanged. + +**Plugin-level default request timeout bumped 10 min β†’ 30 min.** Visible in *Settings β†’ Directory templates β†’ Request timeout (ms)*. If you had previously set a custom value, it's preserved. + +## Deferred to 0.4.x + +Three follow-ups that surfaced during the 0.3.0 work cycle: + +- A `cf_finish_reason` frontmatter stamp on every run β€” would have made the max-tokens diagnostic instant instead of an investigation. +- A "Continue directory template on current file" command β€” for runs cut off by max-tokens, would re-prompt with only the missing sections plus URL-keyed citation merge. +- A "diff and patch user files" mode for re-seed β€” the vault-drift bug we caught is structural: re-seed only writes missing files, so partial/template improvements never reach users who've already seeded. + +## Compatibility + +`minAppVersion: 1.8.10` β€” unchanged from 0.2.1. Desktop-only β€” unchanged. No new external dependencies; the three new templates and two partials are bundled into `main.js` at build time. The cleanup pipeline behavior (think-block wrap, image-marker swap, sources footer) is unchanged. diff --git a/src/docs/partials/latex-discipline.md b/src/docs/partials/latex-discipline.md new file mode 100644 index 0000000..1a9336a --- /dev/null +++ b/src/docs/partials/latex-discipline.md @@ -0,0 +1,18 @@ +**Obsidian MathJax discipline β€” Obsidian uses `$$...$$` for block math and `$...$` for inline math, NOT the `\[...\]` / `\(...\)` form.** Math wrapped in `\[...\]` or `\(...\)` renders as literal escape sequences, not as a formula. + +**Block math** uses `$$...$$`: + +- BAD: `\[\text{CAGR} = (V_e / V_b)^{1/n} - 1\]` +- GOOD: `$$\text{CAGR} = (V_e / V_b)^{1/n} - 1$$` + +**Inline math** uses `$...$`: + +- BAD: `where \(n\) is the number of years` +- GOOD: `where $n$ is the number of years` + +**Escape literal dollar signs as `\$` in prose.** If you mention a dollar amount anywhere in the document, escape the currency symbol so Obsidian does not pair it with another `$` and try to render the span between as inline math. + +- BAD: `revenue grew from $500,000 to $2.5M` +- GOOD: `revenue grew from \$500,000 to \$2.5M` + +This applies throughout the document, not just near math blocks. Two unescaped `$` characters in the same paragraph will be parsed as an inline-math span no matter how far apart they are. diff --git a/src/docs/partials/mermaid-discipline.md b/src/docs/partials/mermaid-discipline.md index 5e07e12..db9a449 100644 --- a/src/docs/partials/mermaid-discipline.md +++ b/src/docs/partials/mermaid-discipline.md @@ -1,12 +1,54 @@ -**Mermaid syntax discipline β€” read every rule before emitting a fence; broken mermaid is the #1 failure mode of these templates:** +**Mermaid syntax discipline β€” read every rule before emitting a fence; broken mermaid is the #1 failure mode of these templates.** Mermaid is a strict parser. Every chart you emit must follow these rules or the page renders a syntax error instead of the diagram. Each rule below pairs a BAD example with a GOOD example β€” match the GOOD shape exactly. -- **Always quote node labels** with double quotes whenever the label contains ANY of: parentheses `( )`, equals `=`, plus `+`, comma `,`, colon `:`, slash `/`, backtick, brackets `[ ]`, braces `{ }`, angle brackets `< >`, pipe `|`, ampersand `&`, hash `#`, or LaTeX. Example: write `A["F(x) + x"]`, never `A[F(x) + x]`. When in doubt, quote. -- **Never put parentheses, math, or LaTeX inside an unquoted label.** No `\(...\)`, no `$...$`, no `H(x)=F(x)+x` bare β€” wrap in `"..."`. -- **Subgraph titles must be quoted and given an id:** `subgraph RB["Residual Block"] ... end`, never `subgraph Residual Block`. -- **Edge labels** with special chars use the `-->|"label"|` form when the label contains punctuation; plain words can stay unquoted. -- **One statement per line.** No semicolons. No trailing whitespace inside labels. -- **Allowed shapes:** rectangle `A["x"]`, rounded `A("x")`, circle `A(("x"))`, rhombus `A{"x"}`. Pick one consistently; don't mix exotic shapes. -- **Direction header is required:** start with `flowchart LR` or `flowchart TD` (prefer `LR` for process flows, `TD` for hierarchies/taxonomies). -- **Node ids are short alphanumerics** (`X`, `F1`, `Add`, `H`), distinct from labels. Never use spaces, parens, or punctuation in an id. -- **No HTML, no `
` outside quoted labels, no markdown inside labels.** If a label needs a line break, use `"line one
line two"` inside quotes. -- Before finalizing, mentally parse the fence: every `[`, `(`, `{` must close; every label containing punctuation must be wrapped in `"..."`. +**Quote every multi-word node and edge label in double quotes.** The parser cannot handle `(`, `)`, `&`, `:`, `,`, `/`, `#`, `=`, `+`, `[`, `]`, `{`, `}`, `<`, `>`, `|`, backtick, or LaTeX syntax inside a bare label. When in doubt, quote. + +- BAD: `A[Raw inputs (text, audio)]` +- GOOD: `A["Raw inputs (text, audio)"]` + +- BAD: `B -->|Sends data & status| C` +- GOOD: `B -->|"Sends data & status"| C` + +**Escape `&` as `&` inside quoted labels.** Bare `&` is interpreted as the start of an HTML entity even when the label is quoted. + +- BAD: `D["Search & filter"]` +- GOOD: `D["Search & filter"]` + +**No nested double quotes.** The outer `"..."` is the only quoting layer the parser supports. Drop the inner quotes or rephrase. + +- BAD: `B["Legal view
\"Which laws apply?\""]` +- GOOD: `B["Legal view
Which laws apply?"]` + +**Line breaks inside labels are `
`, never `\n`.** `\n` renders as the literal two characters backslash-n. + +- BAD: `B1[Membrane potential\nintegrates spikes]` +- GOOD: `B1["Membrane potential
integrates spikes"]` + +**Subgraph titles with spaces need an ID plus a quoted display title.** A bare `subgraph My Title` breaks in stricter renderers. + +- BAD: `subgraph Spiking neuron dynamics` +- GOOD: `subgraph Spiking_neuron_dynamics ["Spiking neuron dynamics"]` + +**Quote decision-node (diamond) labels too.** Same rules as `[...]` labels apply to `{...}` shapes. + +- BAD: `B2{Threshold reached?}` +- GOOD: `B2{"Threshold reached?"}` + +**Allowed shapes:** rectangle `A["x"]`, rounded `A("x")`, circle `A(("x"))`, decision/rhombus `A{"x"}`. Pick one consistently per fence; don't mix exotic shapes. + +**Structural rules every fence must follow:** + +- Start with a direction header: `flowchart LR` for process flows, `flowchart TD` for hierarchies or taxonomies. +- Node IDs are short alphanumerics (`X`, `F1`, `Add`, `H`) β€” distinct from labels, no spaces, no punctuation. +- One statement per line. No semicolons. No trailing whitespace inside labels. +- No HTML outside quoted labels. No markdown inside labels. + +**Self-check before emitting the ` ```mermaid ` block.** Mentally walk every node and edge label and confirm all six: + +1. Every multi-word label is wrapped in `"..."`. +2. Every `&` inside a label is written `&`. +3. No label contains nested `"`. +4. Every line break inside a label is `
`, not `\n`. +5. Every subgraph whose title contains a space uses the `id ["display title"]` form. +6. Every `[`, `(`, `{` has a matching close. + +If you cannot satisfy all six, **simplify the labels** (drop the parenthetical sub-text, replace `&` with "and", shorten the phrasing) rather than emit a broken diagram. Do not emit a fence you cannot validate. A working diagram with shortened labels is always better than a broken diagram with the labels you wanted. diff --git a/src/docs/templates/README.md b/src/docs/templates/README.md index f9e79c9..9c61c93 100644 --- a/src/docs/templates/README.md +++ b/src/docs/templates/README.md @@ -18,6 +18,8 @@ ## Shipped templates | `source-profile.md` | `Sources/**` | Profiles of trusted sources β€” books, people, channels, publications, journals, reports, events. Adapts emphasis to the source's type. | | `toolkit-profile.md` | `Tooling/**` | Profiles of tools, products, platforms, frameworks. | | `market-map-profile.md` | `lost-in-public/market-maps/**`, `market-maps/**` | Analyst-grade market-map drafts β€” both Known Category (e.g., Humanoid Robots) and Thesis-Driven (e.g., Neural Network Hardware as Brains for Robotics). Runs on `sonar-deep-research`. Single-stage v1; multi-stage RAG + Claude-edit pass is planned. | +| `standards-and-specs-profile.md` | `Sources/Standards-and-Specs/**`, `Standards-and-Specs/**` | Analyst-grade profiles of open specs and standards. Five-way authority typing (de-jure / consortium / vendor-led-open / community / de-facto). Three-tier structural adoption framing (incumbents / challengers / innovators) plus notable holdouts. Named editors, stewardship transitions, named critics. Runs on `sonar-deep-research` with idle-only timeout safety. | +| `market-category-profile.md` | `concepts/Market-Categories/**`, `Market-Categories/**` | Concept-folder reference card for a named market category. Three-tier company landscape with explicit FINANCIAL-STAGE definitions: Incumbents (public / late-stage private / PE-owned) β†’ Challengers (Series C+ scale-ups, recently public) β†’ Innovators (Pre-Seed through Series B). Big tech belongs in Incumbents β€” no anti-incumbent cap here, unlike the market-map template. Separate Why Now / What's Happening sections covering CAGR + category-creation momentum. Industry Coverage section sub-grouped into Market Reports / Industry Articles / Financial News. Runs on `sonar-deep-research`. | ## How a template works @@ -60,7 +62,7 @@ # User Notes ### The three zones 1. **Frontmatter** (top, between `---` lines) β€” carries `title`, `applies-to-paths` (array of glob patterns), and an optional `description`. The plugin uses `applies-to-paths` to match a template to a target file. -2. **`cft` block** (a code fence with language `cft`) β€” YAML config: `provider`, `model`, `search-recency`, `return-citations`, `return-images`, optional `request-timeout-ms:` (overrides the plugin-level *Request timeout (ms)* setting for this template β€” useful for long deep-research runs that need 20-40 min), plus a multi-line `system:` prompt. Anything above the `cft` block is treated as documentation and dropped from the request. +2. **`cft` block** (a code fence with language `cft`) β€” YAML config: `provider`, `model`, `search-recency`, `return-citations`, `return-images`, optional `stream-idle-timeout-ms:` (per-chunk idle timer; defaults 270s for deep-research / 90s otherwise β€” the primary safety mechanism), `request-timeout-ms:` (optional absolute wall-clock ceiling; falls back to plugin-level *Request timeout (ms)*; set to `0` to disable and rely on idle-only), `max-tokens:` (optional Perplexity output-token budget override; defaults to Perplexity's per-model cap β€” ~8192 for sonar-deep-research, which silently truncates long analyst-grade templates), plus a multi-line `system:` prompt. Anything above the `cft` block is treated as documentation and dropped from the request. 3. **Heading skeleton** (everything between the `cft` block's closing fence and the first `***`) β€” the user prompt. This is the markdown structure the model fills in. Bullets under each heading are *instructions to the model*, not literal output. The `***` divider terminates the user prompt. Anything below it (the User Notes zone) is for your own scratch work and never reaches the model. diff --git a/src/docs/templates/concept-profile.md b/src/docs/templates/concept-profile.md index 2937f6b..79e8f20 100644 --- a/src/docs/templates/concept-profile.md +++ b/src/docs/templates/concept-profile.md @@ -60,6 +60,8 @@ # Defining and Describing {{basename}} - If this concept involves a process, hierarchy, taxonomy, or part-relationship that a diagram clarifies, render a `mermaid` codefence here. If a diagram does not add insight, omit it entirely β€” do not force one. {{include: mermaid-discipline}} + +{{include: latex-discipline}} - Write a one-sentence italicized lede (a zinger or kicker) that captures the core insight in plain language. Use markdown italics: `_..._`. - Then write a 2–4 sentence paragraph giving more context: what the concept is, when it applies, and why it matters. diff --git a/src/docs/templates/market-category-profile.md b/src/docs/templates/market-category-profile.md new file mode 100644 index 0000000..0294a48 --- /dev/null +++ b/src/docs/templates/market-category-profile.md @@ -0,0 +1,316 @@ +--- +title: Market Category Profile (Analyst Draft) +applies-to-paths: + - "concepts/Market-Categories/**" + - "Market-Categories/**" +description: Generates an analyst-grade profile of a named market category β€” its definition, the forces that made it coherent right now, current CAGR and momentum, and the three-tier company landscape (incumbents / challengers / innovators) with explicit financial-stage definitions for each tier. +date_created: 2026-05-27 +date_modified: 2026-05-27 +--- + +# About this template + +Use this for files under `concepts/Market-Categories/` whose body is empty or whose curated lead-in (a thesis paragraph or stake-in-the-ground take) has been authored but the analytical body is missing. + +A **market category profile** is the mental-model entry an innovation consultant draws on when asked "what is this market, who's playing in it, and where is it going?" Unlike a [[market-map-profile]] (which is a published analyst memo with sub-segments and lighthouse examples), this profile is a concept-folder entry β€” a reference card the curator returns to when reading or briefing on the category. + +The structure is opinionated about ONE thing in particular: the three-tier company landscape is **structural**, not editorial. Each tier has an explicit financial-stage definition, and the template asks the model to populate each tier separately: + +- **Incumbents** β€” large public companies, tech giants, late-stage private, PE-owned behemoths. Legacy footprint, named everywhere in the category. Big tech BELONGS HERE β€” do not suppress. +- **Challengers** β€” well-funded scale-ups (Series C and beyond, or recently public via SPAC/IPO). Rapidly growing, hype-driven, eating share from the incumbents. +- **Innovators** β€” Pre-Seed through Series B startups. Early-stage, often founder-led, novel-bet positioning, the frontier of the category. + +This is a different editorial discipline from the `market-map-profile` template β€” for a market map, the analyst is enumerating sub-segments and innovators within them, and big tech gets capped to 1 of 5-10 to counteract training-data over-representation. For a market-category profile, the goal is the *full* landscape of who plays in this market, sorted by financial stage. Both incumbents and innovators are first-class. + +This template runs on `sonar-deep-research` and skips image embedding by design β€” deep research returns better citation density and analytical length but unreliable image metadata. Banner / portrait / square imagery for category entries lives in frontmatter, generated separately (Ideogram). + +```cft +provider: perplexity +model: sonar-deep-research +return-citations: true +return-images: false +# Idle-only safety β€” ceiling disabled. The per-chunk idle timer (270s +# for deep-research, inherited from the model-class default) is the +# sole safety mechanism: as long as Perplexity sustains bytes, the run +# is allowed to complete. A complete market-category profile can stretch +# 6-9K words given the three-tier company landscape + named market +# reports + What's Happening section; any wall-clock cap risks cutting +# the tail. Set this to a positive value if you want a hard ceiling. +request-timeout-ms: 0 +# Generous output-token budget. Perplexity's default for sonar-deep-research +# (~8192 tokens, ~6K words) silently truncates this template's back half +# by ending the stream cleanly with finish_reason: length mid-skeleton. +# Bumped to 24000 (~18K-word budget) so the full three-tier landscape + +# deeper cards + industry coverage section can land with headroom. +max-tokens: 24000 +system: | + You are writing the analyst-grade profile of a MARKET CATEGORY named + "{{basename}}" for an innovation consultant's vault. + + A market-category profile is the mental-model entry an analyst returns + to when asked "what is this market, who's in it, and where is it + going?" It is NOT a published market map (those live in + `lost-in-public/market-maps/` and run a different template). It IS a + concept-folder reference card with disciplined three-tier company + framing and explicit market-data sourcing. + + A CATEGORY PROFILE answers four questions a partner would ask: + + - WHAT is this market category β€” how are its boundaries drawn, what's + in and what's out? + - WHY NOW β€” what forces aligned to make this category coherent or + expandable at this moment? + - WHAT'S HAPPENING β€” what is the CAGR, the momentum, the category + creation or coalescence dynamics right now, named with specific + market-report figures? + - WHO IS PLAYING in three explicit financial-stage tiers β€” incumbents, + challengers, innovators β€” with named cited entities in each? + + THREE-TIER COMPANY FRAMING β€” STRUCTURAL, NOT EDITORIAL. + + Each tier has an explicit financial-stage definition. Sort each + named company into exactly one tier: + + - INCUMBENTS β€” large public companies, tech giants, late-stage + private companies (typically post-Series E or with $1B+ valuation + and 10+ years of operation), and private-equity-owned behemoths. + The defining trait is legacy footprint with huge market presence β€” + these are the companies an enterprise buyer ALREADY has a contract + with, even if not in this category yet. Big tech (Microsoft, + Google, Amazon, Apple, Meta, Oracle, Salesforce, IBM, Adobe, + SAP, Cisco, Nvidia post-2020) belongs HERE if it has a relevant + offering in the category β€” DO NOT suppress. + - CHALLENGERS β€” well-funded scale-ups, typically Series C through + pre-IPO, or recently public via SPAC/IPO with under 7 years of + operation. The defining traits are rapid growth, public hype + (analyst coverage, founder press, conference keynotes), and the + capital position to credibly threaten incumbent market share. + "Recently IPO'd unicorn" usually belongs here, not in Incumbents. + - INNOVATORS β€” Pre-Seed through Series B funded startups. The + defining traits are early-stage funding, novel-bet positioning + (often a contrarian thesis on the category's shape), and + typically founder-led with under 100 employees. This is where + the next category-redefinition will come from. + + For each tier, produce 5-8 named entities. Render each entry in + `[Company Name](https://url) β€” one-line on their role in this category` + form, then produce 2-3 deeper CARDS per tier for the most + strategically significant entries (the ones a partner would + expect to be briefed on by name). + + Frontmatter for "{{basename}}": + {{frontmatter}} + + TIER-CARD SHAPE (deeper treatment for top entries per tier): + + Each deeper card follows this shape exactly. The emphasis SHIFTS by + tier β€” for Incumbents the Footprint line is heaviest, for Innovators + the Funding line is heaviest: + + #### [Company Name](https://homepage.url) + **Stage**: public (EXCHANGE: TICKER) | late-stage private (last + round date) | PE-owned (sponsor name + acquisition year) | + scale-up | Series B (date) | Pre-Seed / Seed (date) + **Funding**: total raised + most recent round + lead investor + year. + For public companies: market cap (as of recent quarter) + last reported + revenue. For PE-owned: known acquisition price if disclosed. Cite. [N] + **Footprint**: revenue / employees / customer count / countries + served β€” whatever's most visible. Heaviest for INCUMBENTS. Cite. [N] + **Why they're in this category**: one specific sentence on what + they ship into this market and what their angle is. Not marketing + adjectives; a concrete differentiator or footprint claim. Cite. [N] + **Coverage**: 1-2 references in trade press, founder podcasts, or + analyst notes. Format: `[Outlet, Title](url)`. Cite. [N] + + INDUSTRY DATA AND COVERAGE β€” name the reports, not just the figures. + + The "Industry Coverage and Market Data" section is sub-grouped into + three explicit subsections: + + - Market Reports β€” Gartner, IDC, Forrester, McKinsey, ABI Research, + Bain, BCG, Deloitte Insights, Frost & Sullivan, Grand View Research, + Mordor, Markets and Markets, Cabinet Office reports, etc. Name the + REPORT (with title and year) AND the firm. Quote the figure plus the + methodology framing (top-down vs bottom-up, geography, year range). + - Industry Articles β€” specialized trade press (sector verticals from + TechCrunch, The Information, Stratechery, sector-specific trade + publications, named founder/operator blogs). Prefer NAMED journalists + and operator-thinkers over generic outlet citations. + - Financial News β€” Bloomberg, FT, WSJ, Pitchbook articles, Reuters, + Crunchbase News, dealroom.co, Seeking Alpha, Tegus. These should be + the source for funding-round figures, M&A activity, public-company + earnings calls referencing the category. + + RESEARCH DISCIPLINE: + + - Use Perplexity's web search aggressively. For a market-category + profile, breadth of named entities and cited market data both + matter β€” you are populating a reference card that the analyst will + return to dozens of times. + - For every factual claim β€” funding round, market sizing, CAGR + figure, revenue, market cap, customer name β€” append an inline + numeric citation marker [1], [2], etc. + - Quote phrasing from primary sources where useful (founder interviews, + earnings notes, analyst report excerpts, regulatory filings). + - Prefer primary surfaces: company homepage, S-1 filings, earnings + calls, founder Twitter/X, technical blog posts, conference keynotes. + Aggregator pages (Crunchbase summaries, PitchBook profiles, + Wikipedia) are fallbacks for funding stage and headcount only. + - Do NOT cite this Perplexity response itself, only the underlying + sources. + + WHY NOW AND WHAT'S HAPPENING β€” DIFFERENT BEATS. + + These two sections are easy to conflate; keep them distinct: + + - WHY NOW asks: what FORCES aligned that made this category coherent + (or expandable) right now? Technological unlocks, regulatory shifts, + capital-formation patterns, customer-behavior shifts. The reader + leaves Why Now understanding the ENABLING CONDITIONS. + - WHAT'S HAPPENING asks: what is the current MOMENTUM β€” CAGR figures + with sources, category-creation events (a defining IPO, a defining + acquisition, a defining product launch that crystallized the + category name), recent capital concentration. The reader leaves + What's Happening understanding the CURRENT VELOCITY and where the + capital is flowing. + + EDITORIAL STANCE: + + - This is a market-category profile, NOT a market-map analyst memo. + There is no anti-incumbent cap β€” name the big tech in Incumbents, + name the well-funded scale-ups in Challengers, name the Pre-Seed + bets in Innovators. The three-tier sorting IS the discipline. + - Where a company could plausibly fit in two tiers (a unicorn that + just IPO'd; a late-stage private that's behaving like a scale-up), + pick the tier that best matches the company's CURRENT behavior and + note the ambiguity in the card's "Why they're in this category" + line. + - Surface CATEGORY DISPUTES β€” sentences in the prose where credible + operators disagree about whether the category boundary should + include or exclude a particular sub-area. Disputes are signal. + + LINKS AND WIKILINKS: + + - For company names, founder names, and source links, use + `[Name](https://url)` form. + - Do NOT invent `[[wikilink]]` syntax. The curator will promote + names to vault wikilinks during the curation pass. + - If you happen to know a canonical adjacent concept the curator + has already vaulted (e.g., "Agentic Workspaces", "Compliance + Automation"), surface the name as plain text in the Adjacent + Concepts section so the curator can wikilink it later. + + CALIBRATION ON LENGTH: + + This is a deep-research run. Lean long, not short. A complete + market-category profile is roughly 6,000-9,000 words of body, with + 15-24 named companies across three tiers (5-8 per tier) + 2-3 deeper + cards per tier, a fully-populated Industry Coverage & Market Data + section with 4-6 named reports / articles / news pieces per + sub-grouping, and explicit cited CAGR / TAM figures in What's + Happening. Better to over-enumerate and let the curator prune than + to under-enumerate and leave the analyst guessing. +``` + +# Snapshot + +- One-paragraph italicized lede (max 2 sentences) that captures the category in a single beat. Voice: an analyst introducing the category to a partner who's never heard of it. Use markdown italics: `_..._`. +- Then the headline stat β€” one cited statistic that signals scale, velocity, or category momentum (current TAM, current CAGR, a recent landmark funding round, a defining exit). Format as a blockquote (`> "..."`). +- Then 2-3 sentences orienting the reader: what is this category, what timeframe is the profile capturing, why is it worth a reference card right now. + +# What is this Market Category? + +- 3-5 sentences defining the category. What problems do its products solve? What customer is it sold to? What does the category EXCLUDE β€” what does the boundary leave out that a naive reader might wrongly include? +- One sentence on where the boundary is fuzzy or disputed β€” name the specific edges where operators disagree about whether something belongs. + +# Why Now? + +- 3-5 cited bullets, each one a specific FORCE that aligned to make this category coherent or expandable in the current quarter. The reader should leave understanding the ENABLING CONDITIONS. +- Force types to consider: technological unlocks (a capability crossing a cost / latency / accuracy threshold), regulatory or standards shifts, capital-formation patterns (a fund vintage, an exit precedent that opened the LP appetite), customer-behavior shifts, an open-source release that lowered the floor for new entrants. +- Cite each force. Where possible, quote a founder, analyst, or operator who named the force in public. + +# What's Happening? + +What's the current momentum, in cited numbers and named events. + +- **CAGR and TAM:** 2-3 cited bullets covering the current CAGR estimate, TAM, and forecast year range β€” naming both the figure AND the report it comes from. Where two credible sources disagree (Gartner says $X by Y, IDC says $Z by Y), surface the disagreement explicitly. +- **Category creation events:** 2-3 cited bullets covering the defining moments β€” a landmark IPO that crystallized the category name; a defining acquisition that signaled incumbent recognition; a defining product launch that became the category's reference implementation; a defining analyst-report-naming-the-category event. +- **Capital concentration:** 2-3 cited bullets covering where funding has concentrated by tier (e.g., "$3B raised by Series B-D scale-ups in the category in 2025, led by [named funds]"), with named lead investors where public. + +# Market Incumbents + +Large public companies, tech giants, late-stage private (post-Series E or $1B+ valuation with 10+ years of operation), or private-equity-owned behemoths. Legacy with huge market footprint. Big tech belongs here when they have a relevant offering β€” do not suppress. + +- 5-8 named entities. Format: `[Company Name](https://url) β€” one-line on their role and footprint in this category`. Cite each. +- After the flat list, produce 2-3 deeper TIER-CARDS for the most strategically significant entries per the card shape in the system prompt. For Incumbents, the **Footprint** line is the heaviest field β€” market cap, revenue, customer count, geographic reach. + +# Market Challengers + +Well-funded scale-ups (Series C and beyond, or recently public via SPAC/IPO with under 7 years of operation). Rapidly growing, hype-driven, public analyst coverage, capital position to credibly threaten incumbent share. + +- 5-8 named entities. Same format as Incumbents. +- After the flat list, produce 2-3 deeper TIER-CARDS for the most strategically significant. For Challengers, both **Funding** and **Footprint** matter equally β€” funding shows the war chest, footprint shows the traction. + +# Market Innovators + +Pre-Seed through Series B funded startups. Early-stage, often founder-led, novel-bet positioning, typically under 100 employees. This is where the next category-redefinition will come from. + +- 5-8 named entities. Same format as Incumbents. +- After the flat list, produce 2-3 deeper TIER-CARDS for the most strategically significant. For Innovators, the **Funding** line is the heaviest field β€” round size, lead investor, date, total raised, and the contrarian thesis the round capitalized. + +# Industry Coverage and Market Data + +The sourcing layer β€” where to read about this category and where the figures came from. Group into three subsections: + +## Market Reports + +- 4-6 named market reports. Format: `**[Report Title, Year](url)** β€” Firm β€” one-line on the report's signature finding or methodology. [N]` +- Prefer specialized analyst firms (ABI Research, Forrester, Gartner, IDC, Frost & Sullivan, Grand View Research) over generalist business publications. +- Where a report's headline figure has been quoted in this profile (CAGR, TAM), the citation should resolve to the report itself, not a press release recapping it. + +## Industry Articles + +- 4-6 named articles, blog posts, or essays from specialized trade press and operator-thinkers. Format: `**[Title](url)** β€” Outlet / Author β€” one-line on the angle. [N]` +- Prefer named journalists and operator-bloggers over generic outlet citations. Founder posts on Substack / personal blog count and are often the most useful. + +## Financial News Sources + +- 4-6 named financial news pieces covering funding rounds, M&A activity, public-company earnings calls referencing the category, or sell-side analyst notes. Format: `**[Title](url)** β€” Outlet β€” one-line on what it reports. [N]` +- Prefer Bloomberg, FT, WSJ, Reuters, Pitchbook, Crunchbase News, dealroom.co for primary funding/M&A signal. Use Seeking Alpha / Motley Fool / Yahoo Finance only for public-company earnings recaps when the primary source isn't available. + +# Frontier and Open Questions + +- 4-6 bullets, each a specific open question about where the category is going. Frame each as a question, not a statement. +- Pair each question with a one-sentence note on which tier (Incumbents / Challengers / Innovators) or which named operators are most likely to drive the resolution. +- Surface category-boundary disputes explicitly β€” where credible operators disagree about whether the category should expand to include adjacent functions, or contract to focus on a narrower core. + +# Adjacent Concepts and Categories + +- 4-8 plain-text concept names (no wikilink syntax β€” the curator wikilinks during curation) that an operator working in this category would want to explore next. +- Mix of: adjacent market categories (other categories this one borders), foundational concepts (mental models the category sits on), and vocabulary terms (specific jargon the curator may want to define in `Vocabulary/`). +- Format: `- β€” one-line on how it adjoins this category`. + +*** + +# User Notes + +Anything below the `***` line is excluded from the request. Use this zone for: + +- The stake-in-the-ground take or thesis paragraph you want to fold into the model's context. To do that, move the paragraph ABOVE the `***` divider before running the template. +- Curator's wikilink resolution notes during the curation pass β€” which named companies, founders, or adjacent concepts have vault entries that need linking back. +- Tuning notes on which sections came back thin and need a re-run. +- Hand-curated notes on the category's relevance to your portfolio or thesis. + +## Relationship to other shipped templates + +- **`market-map-profile`** β€” for published analyst-grade maps in `lost-in-public/market-maps/`. That template produces a memo with sub-segments and lighthouse examples in flowing analyst prose. THIS template (market-category-profile) is a concept-folder reference card with the disciplined three-tier financial-stage framing. +- **`concept-profile`** β€” for general `concepts/**` entries. The market-category template targets the narrower `concepts/Market-Categories/` sub-folder and overrides the general concept template's editorial stance (which suppresses big tech) β€” for a category profile, naming the incumbents IS the goal. +- **`toolkit-profile`** β€” for individual tools/products/platforms in `Tooling/**`. Many companies named in this template's tiers will have their own toolkit-profile entries that the curator wikilinks back to. + +## Multi-stage roadmap (deferred) + +This v1 template is intentionally single-stage. The deferred multi-stage version mirrors the market-map and standards-and-specs roadmaps, tracked in `context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md`: + +1. **RAG pre-flight** β€” pull canonical Lossless sources adjacent to this category (vaulted company profiles in `Tooling/`, adjacent vaulted categories, prior commentary in `concepts/`) as primed context. +2. **Perplexity research stage** β€” deep research with the RAG context as primer, eliminating wikilink invention and letting the model name actual vault entries. +3. **Claude editing stage** β€” editorial pass that enforces tier discipline, prunes over-enumeration, sharpens tier boundaries (especially for the unicorn-just-IPO'd ambiguity), and emits the final `[[wikilink]]` form. diff --git a/src/docs/templates/market-map-profile.md b/src/docs/templates/market-map-profile.md index 422c334..8ec3714 100644 --- a/src/docs/templates/market-map-profile.md +++ b/src/docs/templates/market-map-profile.md @@ -26,13 +26,24 @@ # About this template model: sonar-deep-research return-citations: true return-images: false -# 40-minute wall-clock budget for this template specifically. Market-map deep -# research runs routinely produce 6-8K-word drafts with 20-40 named innovators -# across 4-8 sub-segments, and the tail of that generation (Frontier and -# Adjacent Concepts sections) is the most cuttable under aggressive timeouts. -# Overrides the plugin-level default. A complete run is worth $10-$50 of -# analyst time; let it cook. +# 40-minute absolute wall-clock ceiling β€” the belt-and-suspenders cap. +# The primary safety is the per-chunk idle timer (270s for deep-research, +# inherited from the model-class default), which lets a slow-but-healthy +# stream complete naturally while killing a silently-stalled one fast. +# This ceiling is the "do not run longer than 40 min under any +# circumstances" cap on top of that. Market-map deep-research runs +# routinely produce 6-8K-word drafts with 20-40 named innovators across +# 4-8 sub-segments; a complete run is worth $10-$50 of analyst time, so +# the cap is generous. Set this key to 0 to disable the ceiling entirely +# and rely on idle-only safety. request-timeout-ms: 2400000 +# Generous output-token budget. Perplexity's default for sonar-deep-research +# (~8192 tokens, ~6K words) silently truncates market-map drafts mid-skeleton +# by ending the stream cleanly with finish_reason: length. Bumped to 24000 +# (~18K-word budget) so a complete map can land with headroom for 20-40 +# innovator cards across 4-8 sub-segments plus Market Dynamics, Frontier, +# and Adjacent Concepts. +max-tokens: 24000 system: | You are writing the analyst-grade draft of a MARKET MAP titled "{{basename}}". diff --git a/src/docs/templates/standards-and-specs-profile.md b/src/docs/templates/standards-and-specs-profile.md new file mode 100644 index 0000000..8476937 --- /dev/null +++ b/src/docs/templates/standards-and-specs-profile.md @@ -0,0 +1,385 @@ +--- +title: Standard or Spec Profile (Analyst Draft) +applies-to-paths: + - "Sources/Standards-and-Specs/**" + - "Standards-and-Specs/**" +description: Generates an analyst-grade profile of an open spec or standard β€” covering authorship, current stewardship, ecosystem position, three-tier adoption, named critics, and stewardship transitions. Tuned for the strategist's lens, not the implementer's. +date_created: 2026-05-27 +date_modified: 2026-05-27 +--- + +# About this template + +Use this for files under `Sources/Standards-and-Specs/` whose body is empty or whose curated lead-in (a thesis paragraph, a stake-in-the-ground take) has been authored but the analytical body is missing. + +A **spec profile** is the draft an innovation consultant hands to a partner who has just asked "should we care about this spec?" It is not a tutorial, not implementer documentation, not marketing for the spec. It explains who created it, who steers it now, what coordination problem it solves, who has implemented it across three tiers (incumbents / challengers / innovators), who has explicitly declined, and what the public critics are saying. + +The template handles five flavors of authority: + +1. **De-jure** β€” formal standards body with quasi-legal authority (W3C, IETF/RFC, ISO, IEEE, ECMA, NIST). +2. **Industry consortium** β€” non-vendor-controlled multi-stakeholder body (Khronos, Linux Foundation, OpenJS, OASIS, CNCF). +3. **Vendor-led-open** β€” single vendor created and primarily maintains it but has published under an open license (MCP from Anthropic, OpenAPI's early years at Wordnik). +4. **Community** β€” published by an individual or informal group with no parent org (llms.txt by Jeremy Howard, AGENTS.md after the OpenAI handoff). +5. **De-facto** β€” never formally published but treated as a spec because the dominant implementation defines it (README convention, package.json shape, the curl interface). + +The model classifies the authority type from the spec name, primary URL, and search results before drafting Identity & Status. Every downstream section's treatment depends on which type. + +This template runs on `sonar-deep-research` and skips image embedding by design β€” deep research returns better citation density and analytical length but unreliable image metadata. Visual identity for specs (logos, diagrams) lives in frontmatter, generated separately. + +```cft +provider: perplexity +model: sonar-deep-research +return-citations: true +return-images: false +# Idle-only safety β€” ceiling disabled. The per-chunk idle timer (270s for +# deep-research, inherited from the model-class default) is the sole +# safety mechanism: as long as Perplexity sustains bytes, the run is +# allowed to complete. A complete spec profile can stretch 6-9K words +# given the three-tier adoption section + named editors + critique +# framing; any wall-clock cap risks cutting the tail. Set this to a +# positive value (e.g., 3600000 for 60 min) if you want a hard ceiling. +request-timeout-ms: 0 +# Generous output-token budget. Perplexity's default for sonar-deep-research +# is ~8192 tokens (~6K words), which silently truncates this template's +# back half by ending the stream cleanly with finish_reason: length +# mid-skeleton (looks like a healthy completion except the last several +# sections never appear). Bumped to 24000 (~18K-word budget) so the full +# 6-9K-word draft can land with headroom for the three-tier adoption +# section, deeper implementation cards, named critics, and the frontier. +max-tokens: 24000 +system: | + You are writing the analyst-grade profile of a STANDARD or SPEC named + "{{basename}}" for an innovation consultant's vault. + + This is NOT an implementer's reference. Your reader is a strategist, + founder, or operator who needs to understand the spec's significance, + governance, and adoption landscape β€” not how to write a conformant + implementation. Conformance matrices, MUST/SHOULD/MAY discipline, and + wire-format details are out of scope. Stewardship, adoption tiers, + political fault lines, and named critics are in scope. + + A SPEC PROFILE answers four questions a partner would ask: + + - WHAT is this spec, who wrote it, and who currently steers it? + - WHY does it matter β€” what does it unlock, what has shifted because + it exists, what has died because it exists? + - WHO is using it (in three tiers: incumbents, challengers, innovators) + and who is publicly NOT using it? + - WHERE is the frontier β€” pending disputes, likely next versions, the + political fault lines that will shape the spec's evolution? + + CLASSIFY THE AUTHORITY TYPE FIRST. + + Every spec falls into one of these five governance models, and your + treatment of every section downstream depends on which one: + + - DE-JURE β€” formal standards body with legal or quasi-legal authority + (W3C, IETF/RFC, ISO, IEEE, ECMA, NIST). Editors are publicly named + in the spec itself; working group archives are public; decisions + follow formal procedure. + - INDUSTRY CONSORTIUM β€” non-vendor-controlled multi-stakeholder body + (Khronos, Linux Foundation, OpenJS Foundation, OASIS, CNCF). + Members are named companies; governance is documented in foundation + bylaws. + - VENDOR-LED-OPEN β€” a single vendor created and primarily maintains + the spec but has published it under an open license and accepts + community contributions (MCP from Anthropic, OpenAPI's early years + at Wordnik). + - COMMUNITY β€” published by an individual or informal group with no + parent org (llms.txt by Jeremy Howard, AGENTS.md in its current + form after the OpenAI handoff). Stewardship is by reputation and PR. + - DE-FACTO β€” never formally published as a spec but treated as one + by the ecosystem because the dominant implementation defines the + behavior (the README convention, package.json's shape, the curl + interface). Stewardship is implicit and shifts with the dominant + implementation. + + Determine the authority type from the spec name, its primary URL, and + search results before drafting Identity & Status. State the type + explicitly in the Identity & Status section AND in the Snapshot + callout line. + + Frontmatter for "{{basename}}": + {{frontmatter}} + + FRONTMATTER FIELDS TO SURFACE FOR LATER PROMOTION: + + A curator promotes the named-creator and named-steward findings into + structured frontmatter after the draft is written. To make that pass + easy, surface these explicitly in the Snapshot lede callout AND in + full detail in the Identity & Status section: + + - created_by β€” list of authors at inception (people OR orgs; prefer + named persons + their affiliation at inception where public) + - created_year β€” four-digit year of first public release + - original_publisher β€” the org that first published it (if different) + - maintained_by β€” current stewards (people, orgs, or "community") + - stewardship_type β€” one of: de-jure | consortium | vendor-led-open + | community | de-facto + + RESEARCH DISCIPLINE: + + - Use Perplexity's web search aggressively. For a spec profile, + accuracy on governance and adoption matters more than encyclopedic + breadth. + - For every factual claim β€” author name, version date, implementation, + transition story, critique, license terms β€” append an inline numeric + citation marker [1], [2], etc. corresponding to the search-result + order. + - Quote phrasing from primary sources where useful: the spec text + itself, editor blog posts, working-group meeting minutes, GitHub + issues that captured the dispute, conference talks where the + editors presented the spec. + - Prefer primary surfaces: the spec's canonical URL, the working + group's mailing list archive or GitHub repo, the editors' own + posts, the implementing-org's announcements. Aggregator pages + (Wikipedia, blog roundups, "X explained" articles) are fallbacks + for orientation only. + - Do NOT cite this Perplexity response itself, only the underlying + sources. + + EDITORIAL STANCE β€” name the humans, not just the orgs: + + Specs are written by people. Training data over-represents the + sponsoring org's logo and under-represents the named editors who + actually drafted the text and made the design choices. Counteract + this systematically: + + - In `created_by` and `maintained_by`, prefer NAMED PERSONS where + public. "Anthropic" is acceptable; "Anthropic + David Soria Parra + + Justin Spahr-Summers" is what an analyst wants. + - In Governance & Stewardship, name the editors, working group + chairs, and sponsoring partners with their roles. Not "the + working group decided" β€” name the chair and the decision. + - For DE-JURE specs, the editor names are on the cover of the spec + itself; surface them. + - For VENDOR-LED-OPEN specs, name both the originating team (the + people who shipped it) AND the cross-vendor partners who have + since contributed substantively. + - For COMMUNITY specs, the originator's identity is the spec's + political center; name them prominently. + + THREE-TIER ADOPTION FRAMING β€” this is STRUCTURAL, not editorial: + + The Adoption section is partitioned into three sub-buckets, and each + has its own discipline: + + - INCUMBENTS β€” the dominant implementations that the ecosystem + treats as canonical or near-canonical. This includes the + reference implementation (if there is one), the most-deployed + OSS implementation, and the commercial implementations from + market leaders. 4-8 named entries. Big tech BELONGS HERE if it + has implemented the spec β€” do not suppress. + - CHALLENGERS β€” production-grade alternative implementations that + compete with the incumbents on completeness or performance, + often from mid-sized companies or well-funded startups. 4-8 + entries. These are the implementations that keep the incumbents + honest. + - INNOVATORS β€” early-stage, experimental, or research + implementations exploring the spec's edges, novel extensions, or + unusual integration patterns. Often single-developer or + research-lab projects. 4-8 entries. This is where the next + extensions will come from. + + Within each bucket, render entries in `[Name](url) β€” one-line + description` form. After the three buckets, surface NOTABLE HOLDOUTS + β€” orgs that explicitly declined to implement, forked, or are running + incompatible alternatives β€” as a separate short paragraph. The + holdouts are often as informative as the adopters about where the + spec's design choices have made enemies. + + IMPLEMENTATION CARD SHAPE (deeper treatment for top entries): + + For the top 2-3 implementations PER TIER β€” the ones a partner would + expect to be briefed on by name β€” produce a deeper card: + + #### [Implementation Name](https://homepage.url) + **Steward**: the org or individual maintaining this implementation, + with one-line context on their relationship to the spec (authoring + participant, early adopter, late convert, fork). Cite. [N] + **Coverage of the spec**: which parts of the spec are implemented, + which are explicitly not, any extensions added beyond spec. Cite. [N] + **Adoption signal**: who uses this implementation in production + (named customer, named project, GitHub stars / downloads if + relevant). Cite. [N] + **Why it matters**: one sentence on the strategic significance β€” + the distribution channel it commands, the conformance bar it sets, + the political coalition it represents. Cite. [N] + + STEWARDSHIP TRANSITIONS β€” surface these explicitly: + + When the created-by org and the maintained-by org are different, that + transition is itself a story worth its own paragraph. Canonical + examples to model the depth of treatment on: + + - AGENTS.md: OpenAI β†’ community (via Sourcegraph) + - OpenAPI: Wordnik β†’ SmartBear β†’ Linux Foundation (OpenAPI Initiative) + - HTTP: Tim Berners-Lee/CERN β†’ IETF / W3C + - JSON: Douglas Crockford β†’ IETF (RFC 8259) + Ecma (ECMA-404) β€” two + parallel stewards + + In Governance & Stewardship, when a transition has happened, give it + a dedicated paragraph: WHEN the transition was announced, WHO handed + to WHOM, WHAT triggered the handoff, what changed in governance pace + or direction as a result. Cite the transition announcement. + + CRITIQUE β€” be candid, do not adjudicate: + + A spec without public critics is either (a) too young to have + attracted critique, or (b) so dominant that nobody bothers. For any + spec older than 18 months, there should be at least 2-3 publicly + named critics worth surfacing in Critique & Open Disputes. Name them, + link to their critique, summarize their argument in one sentence + each. Do not rebut β€” the analyst's job here is to surface the + disagreement so the partner reading the memo knows the political + landscape, not to settle it. + + LINKS AND WIKILINKS: + + - For implementation names, editor names, and source links, use + `[Name](https://url)` form. + - Do NOT invent `[[wikilink]]` syntax. The curator will promote + names to vault wikilinks during the curation pass. If you happen + to know a canonical adjacent spec the curator has already vaulted + (e.g., "JSON-RPC", "OAuth 2.0", "OpenAPI"), surface the name as + plain text in the Adjacent Specs section so the curator can + wikilink it later. + + CALIBRATION ON LENGTH: + + This is a deep-research run. Lean long, not short. A complete spec + profile is roughly 6,000-9,000 words of body, with 12-24 named + implementations across three tiers + holdouts, 2-3 deeper cards per + tier for the most strategically significant implementations, a + documented stewardship transition where relevant, and 2-4 named + critics in Critique & Open Disputes. Better to over-enumerate and + let the curator prune than to under-enumerate and leave the analyst + guessing. +``` + +# Snapshot + +- One-paragraph italicized lede (max 2 sentences) that captures the spec's significance in a single beat. Voice: the analyst opening their memo. Use markdown italics: `_..._`. +- Immediately after the lede, a one-line bold callout in this exact shape: `**Created by** {name(s)} ({year}) Β· **Maintained by** {name(s)} Β· **Type:** {de-jure | consortium | vendor-led-open | community | de-facto}`. This is the spec's identity at a glance. +- Then the headline stat or framing quote β€” one cited statement (an adoption number, a developer-survey result, a notable adopter's framing of why they picked it, a critic's signature objection) from a credible source. Format as a blockquote (`> "..."`). +- Then 2-3 sentences orienting the reader: what is this spec, what does it constrain or enable, and why is it worth profiling right now. + +# The Question this Spec Answers + +- One paragraph (3-5 sentences) stating the coordination failure, interop gap, or pain point this spec was created to address. +- What was the world like in the period before this spec existed? What were people doing instead β€” bespoke per-vendor integrations, fragmented forks, lock-in to proprietary APIs? +- What does the spec's existence allow that wasn't possible (or was prohibitively expensive) before? Be specific β€” name the kinds of products, integrations, or workflows that the spec made tractable. + +# Identity & Status + +- **Full name** and commonly-used abbreviation. +- **Type:** protocol / data format / API / behavior spec / schema / process spec / RFC / convention. +- **Authority type:** state explicitly which of the five β€” de-jure / industry consortium / vendor-led-open / community / de-facto β€” and cite the evidence. +- **Created by:** named persons and orgs at inception, with affiliations as of the inception date. +- **Created year:** four-digit year of first public release. +- **Original publisher:** if different from creators. +- **Maintained by:** current stewards (people, orgs, foundation, or "community"). +- **Current version** and **lifecycle stage** β€” use whatever taxonomy the spec itself uses (Draft / Working / Recommendation / Stable / Deprecated; or numbered milestones; or "alive / abandoned" for community specs). +- **License:** publishing license of the spec text itself (CC-BY? MIT? bespoke?) and patent-grant terms if relevant. +- **Canonical URL** of the spec. + +# Why It Matters + +What this spec unlocks, and what has shifted because it exists. + +- 3-5 cited bullets covering WHAT IT UNLOCKS: the interop, coordination, or portability story that the spec enables. Be specific about what kinds of integrations, products, or customer escapes from lock-in become possible. Where possible, quote a founder, editor, or implementer who named the unlock. +- 3-5 cited bullets covering WHAT IMPACT IT HAS HAD (or is having): named adopters that shipped because of it, named projects that died because of it (or that the spec rendered unnecessary), named market shifts in vendor positioning, named regulatory or procurement-policy changes that referenced the spec. Surface both adoption AND resistance β€” who explicitly chose to ignore or fork, and why their stated reasoning matters. +- If the spec is too young to have demonstrable impact yet (under 12 months from first public release), say so explicitly and pivot to predicted impact based on the early-adoption signals β€” naming the early-adopting orgs and quoting their stated reasons for adopting early. + +# Position in the Ecosystem Stack + +- **What it depends on** β€” the layers underneath that the spec assumes. Name the underlying specs or de-facto conventions (e.g., MCP depends on JSON-RPC and HTTP; A2A depends on HTTP/SSE; AGENTS.md depends on markdown's conventions). For each, cite where the dependency is documented in the spec text. +- **What depends on it** β€” the layers that have been built on top of this spec by other authors. 4-8 named downstream specs, protocols, or conventions with one-line each on how they extend or consume this spec. +- **Companion specs** β€” specs deliberately designed to work alongside this one, often by the same authoring group (e.g., OAuth 2.0 companions: PKCE, dPoP; MCP companions: capability negotiation specs). One paragraph naming 3-5 companions. +- **Strategic positioning:** one sentence β€” what part of the stack does this spec colonize, and why does that position matter for whoever wants to compete with the dominant implementations? + +# Lineage + +- **Predecessors** β€” earlier specs or de-facto conventions in the same problem space. For each, one-line on what they got wrong, what they got right that this spec inherited, and why the ecosystem ultimately moved on. +- **Parallel efforts** β€” concurrent specs from other authoring groups solving the same coordination problem differently. Name 2-4, with one-line each on their distinguishing bet and current adoption signal relative to this spec. +- **Likely successors** β€” emerging specs or research efforts that may eventually supersede this one, OR an explicit statement that no successor is visible yet. If a successor exists, name its authoring group and current status. + +# Governance & Stewardship + +- **Editors / chairs / sponsoring partners** β€” named individuals with their roles. For DE-JURE specs, lift the editor list from the spec's cover. For VENDOR-LED-OPEN specs, name both the originating team and the cross-vendor contributors who have committed substantively. For COMMUNITY specs, the originator's identity is the political center β€” name them prominently. +- **Where decisions get made** β€” the mailing list, GitHub org, working-group meeting, or governance forum. Cite the URL of the public archive. +- **Pace** β€” release cadence, date of last meaningful update, typical time between major versions. +- **Versioning policy** β€” semver / calendar / named milestones / RFC numbering / "live spec, no versions." +- **STEWARDSHIP TRANSITIONS** β€” if `created_by` and `maintained_by` differ, this gets its own paragraph: when the transition was announced, who handed to whom, what triggered the handoff (loss of vendor interest, community pressure, foundation absorption, fork-takeover), what changed in pace or direction as a result. Cite the transition announcement post or RFC. +- **Political fault lines** β€” 2-3 sentences naming the working-group disputes (or implementer disputes) that have been publicly archived. Where is the coalition holding together and where is it under strain? Cite the GitHub issue, mailing-list thread, or talk where the dispute is most visible. + +# Adoption β€” by Tier + +The three-tier framing is structural. Each tier has its own subsection with its own discipline. After the three tiers, surface notable holdouts as a separate short paragraph. + +## Incumbents + +- 4-8 dominant implementations the ecosystem treats as canonical or near-canonical. The reference implementation belongs here if there is one. Big tech belongs here if it has implemented the spec β€” do not suppress. +- Format: `[Implementation Name](https://url) β€” one-line on what they ship and their relationship to the spec`. Cite each. +- After the flat list, produce 2-3 deeper IMPLEMENTATION CARDS for the most strategically significant entries per the card shape defined in the system prompt. + +## Challengers + +- 4-8 production-grade alternative implementations that compete with the incumbents on completeness, performance, or coverage of optional spec capabilities. Often from mid-sized companies or well-funded startups. +- Same format as Incumbents, with 2-3 deeper cards for the most significant. + +## Innovators + +- 4-8 early-stage, experimental, or research implementations exploring the spec's edges, novel extensions, or unusual integration patterns. Often single-developer projects, research-lab outputs, or open-source projects with small but engaged communities. +- Same format as Incumbents, with 2-3 deeper cards for the most significant. + +## Notable Holdouts + +- One short paragraph (3-6 sentences) naming 2-5 orgs or projects that explicitly declined to implement, forked the spec, or are running incompatible alternatives. Name each holdout, link to where they stated their position, summarize their reasoning in one sentence. The holdouts are often as informative as the adopters about where the spec's design choices have made enemies. + +# Critique & Open Disputes + +- 2-4 named critics (people or orgs) with their argument summarized in one sentence each. Format: `**Critic Name** ([affiliation](url)) β€” "their argument in one sentence" [N]`. +- 1-2 sentences on the known limitations the editors themselves have admitted (in mailing-list posts, GitHub issues, conference Q&A). +- 1-2 sentences on the working-group fault lines that are publicly archived β€” where members have voted against each other, where a proposal was withdrawn under pressure, where a fork was threatened. +- Do NOT rebut the critique here. Surface the disagreement; do not adjudicate it. The partner reading the memo will form their own view. + +# Frontier & Open Questions + +- 4-6 bullets, each a specific open question that the working group, implementers, or critics are actively debating. Frame each as a question, not a statement. +- Pair each question with a one-sentence note on which editors, working groups, or implementations are most likely to drive the resolution. +- Where pending RFCs / extensions / next-version drafts are public, link to them and note their current status (proposed / under review / rejected / merged). + +# Media, Voices, and Coverage + +- 6-12 bullets covering the editors' own posts and talks, the credible critics' coverage, the implementer blogs that document real-world deployment, and the podcasts/conferences where the spec is regularly debated. +- Format: `**Name** β€” Platform β€” one-line note on their angle / why they are worth following`. Include a primary URL link. +- Sub-group with `## Editor & Maintainer Voices`, `## Implementer Coverage`, `## Critic Coverage`, `## Conferences & Working Group Forums` if the list is long enough to warrant it. +- Prefer specialized voices over generalist tech press. Prefer the editors' own posts over recapped coverage. + +# Adjacent Specs and Standards + +- 4-8 plain-text spec names (no wikilink syntax β€” the curator wikilinks during curation) that an operator working with this spec would want to explore next. +- Mix of: predecessor and successor specs, companion specs, competing specs from parallel efforts, and foundational specs this one depends on. +- Format: `- β€” one-line on how it adjoins this spec`. + +*** + +# User Notes + +Anything below the `***` line is excluded from the request. Use this zone for: + +- Hand-curated notes on the spec's relevance to your own work, portfolio, or thesis. +- Curator's wikilink resolution notes during the curation pass β€” which named editors, implementations, or adjacent specs have vault entries that need linking back. +- Iteration history while you refine the template for specific spec types. +- Tuning notes on which sections came back thin and need a re-run. + +## Multi-stage roadmap (deferred) + +This v1 template is intentionally single-stage: one Perplexity Deep Research run produces the full draft. The curator promotes implementation names to `[[wikilink]]` form, adds vault-specific cross-references to other vaulted specs, and adds `:::tool-showcase` blocks for vault-tracked implementations during the curation pass. + +The deferred multi-stage version mirrors the market-map roadmap and is tracked in `context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md`: + +1. **RAG pre-flight** β€” pull canonical Lossless sources adjacent to this spec (other vaulted specs in `Sources/Standards-and-Specs/`, the `open-specs-and-standards` study, any prior commentary in `concepts/` or `Tooling/`) as primed context. +2. **Perplexity research stage** β€” deep research with the RAG context as primer, eliminating the wikilink-invention problem and letting the model name actual vault entries. +3. **Claude editing stage** β€” an editorial pass that enforces the analyst voice, prunes over-enumeration, sharpens the three-tier adoption boundaries, and emits the final `[[wikilink]]` form. diff --git a/src/services/directoryTemplateService.ts b/src/services/directoryTemplateService.ts index 4fc975a..b694633 100644 --- a/src/services/directoryTemplateService.ts +++ b/src/services/directoryTemplateService.ts @@ -44,6 +44,7 @@ interface PerplexityPayload { return_related_questions: boolean; search_recency_filter?: string; search_domain_filter?: string[]; + max_tokens?: number; } export interface PerplexitySource { @@ -493,6 +494,23 @@ function buildPayload( payload.search_domain_filter = domainFilter; } + // Per-template `max-tokens:` override β€” Perplexity's default output cap + // (~8192 for sonar-deep-research) silently truncates long analyst-grade + // templates by ending the stream cleanly with finish_reason: "length" + // mid-skeleton. The symptom looks identical to a healthy run except + // that the back half of the section skeleton never appears. Templates + // that legitimately want 6-9K-word bodies must bump this explicitly. + // Accept number or numeric-string; ignore non-positive / non-numeric. + const maxTokensRaw = cfg['max-tokens']; + const maxTokens = typeof maxTokensRaw === 'number' + ? maxTokensRaw + : typeof maxTokensRaw === 'string' + ? parseInt(maxTokensRaw, 10) + : NaN; + if (Number.isFinite(maxTokens) && maxTokens > 0) { + payload.max_tokens = maxTokens; + } + return payload; } @@ -501,14 +519,21 @@ async function streamPerplexityToFile( apiKey: string, endpoint: string, payload: PerplexityPayload, - timeoutMs: number, + timeouts: { idleMs: number; ceilingMs: number }, file: TFile, initialContent: string, isCancelled: () => boolean, ): Promise<{ streamed: string; sources: PerplexitySource[]; images: PerplexityImage[]; truncated: boolean }> { payload.stream = true; const controller = new AbortController(); - const timer = activeWindow.setTimeout(() => controller.abort(), timeoutMs); + // Optional absolute wall-clock ceiling β€” belt-and-suspenders backstop. + // The per-chunk idle timer below is the primary safety mechanism; the + // ceiling only fires if a stream sustains bytes past the ceiling AND + // a hard cap is desired (e.g. settings/cft override > 0). Set ceilingMs + // to Infinity to disable the ceiling entirely. + const ceilingTimer = Number.isFinite(timeouts.ceilingMs) && timeouts.ceilingMs > 0 + ? activeWindow.setTimeout(() => controller.abort(), timeouts.ceilingMs) + : null; let response: Response; try { @@ -524,21 +549,42 @@ async function streamPerplexityToFile( cache: 'no-store', }); } catch (err) { - activeWindow.clearTimeout(timer); + if (ceilingTimer !== null) activeWindow.clearTimeout(ceilingTimer); throw err; } if (!response.ok) { - activeWindow.clearTimeout(timer); + if (ceilingTimer !== null) activeWindow.clearTimeout(ceilingTimer); throw new Error(`Perplexity HTTP ${response.status.toString()}`); } const reader = response.body?.getReader(); if (!reader) { - activeWindow.clearTimeout(timer); + if (ceilingTimer !== null) activeWindow.clearTimeout(ceilingTimer); throw new Error('Perplexity returned no response body'); } + // Race each reader.read() against a fresh per-chunk idle timer. As long + // as bytes keep arriving the timer is cleared and re-armed, so a healthy + // slow stream completes naturally β€” only a stream that goes *quiet* for + // `idleMs` (Perplexity stall, rate-limit, socket close) is killed. This + // is the structural fix for [[Wall-Clock-Timeout-Cuts-Off-Long-Deep- + // Research-Streams]] β€” ported from the legacy PerplexityModal flow in + // perplexityService.ts:659-668, where the same pattern has been load- + // bearing for sonar-deep-research runs for two iterations. + const idleMs = timeouts.idleMs; + const readWithIdleTimeout = (): Promise> => { + let timer: number | undefined; + const timeout = new Promise((_, reject) => { + timer = activeWindow.setTimeout(() => { + reject(new Error(`stream went idle for ${(idleMs / 1000).toString()}s (likely API stall, rate limit, or socket close)`)); + }, idleMs); + }); + return Promise.race([reader.read(), timeout]).finally(() => { + if (timer !== undefined) activeWindow.clearTimeout(timer); + }); + }; + const decoder = new TextDecoder(); let sseBuffer = ''; let streamed = ''; @@ -557,14 +603,15 @@ async function streamPerplexityToFile( let value: Uint8Array | undefined; let done = false; try { - ({ value, done } = await reader.read()); + ({ value, done } = await readWithIdleTimeout()); } catch { - // Timeout (AbortController fired), user cancel, or a dropped - // socket. Don't throw away what already arrived β€” - // sonar-deep-research delivers the whole document plus its - // citations in the FIRST SSE event, so a later disconnect - // should still leave a usable, cited file. Mark the run - // truncated and fall through to the final flush + return. + // Idle timer fired, AbortController fired (ceiling or user + // cancel), or a dropped socket. Don't throw away what + // already arrived β€” sonar-deep-research delivers the whole + // document plus its citations in the FIRST SSE event, so a + // later disconnect should still leave a usable, cited file. + // Mark the run truncated and fall through to the final + // flush + return. truncated = true; break; } @@ -626,7 +673,7 @@ async function streamPerplexityToFile( } } } finally { - activeWindow.clearTimeout(timer); + if (ceilingTimer !== null) activeWindow.clearTimeout(ceilingTimer); try { reader.releaseLock(); } catch { @@ -775,27 +822,42 @@ export async function applyTemplate( // Set initial state before streaming begins. await app.vault.modify(target, initialContent); - // Template-level `request-timeout-ms:` in the cft block overrides the - // plugin-level default. Deep-research templates routinely need 20-40min - // to fully render a long analyst draft and shouldn't be capped by a - // setting tuned for short concept-profile runs. Accept number or - // numeric string. Ignore non-positive / non-numeric values silently. - const cftTimeoutRaw = template.cftConfig['request-timeout-ms']; - const cftTimeoutMs = typeof cftTimeoutRaw === 'number' - ? cftTimeoutRaw - : typeof cftTimeoutRaw === 'string' - ? parseInt(cftTimeoutRaw, 10) - : NaN; - const effectiveTimeoutMs = Number.isFinite(cftTimeoutMs) && cftTimeoutMs > 0 - ? cftTimeoutMs - : settings.requestTimeoutMs; + // Two complementary timeout knobs: + // - `stream-idle-timeout-ms:` (cft) β€” per-chunk idle timer; primary + // safety. Defaults: 270s deep-research, 90s normal β€” matches the + // legacy modal flow in perplexityService.ts:659. + // - `request-timeout-ms:` (cft, legacy key retained) β€” absolute + // wall-clock ceiling; opt-in backstop. Falls back to the + // plugin-level `settings.requestTimeoutMs`. Set to 0 (in cft or + // settings) to disable the ceiling and rely entirely on idle. + // Both accept number or numeric-string; non-positive / non-numeric + // falls through to the fallback chain silently. + const parseMs = (raw: unknown): number => { + if (typeof raw === 'number') return raw; + if (typeof raw === 'string') return parseInt(raw, 10); + return NaN; + }; + const isDeepResearch = /deep-research/i.test(effectiveModel); + const idleDefaultMs = isDeepResearch ? 270_000 : 90_000; + const cftIdleMs = parseMs(template.cftConfig['stream-idle-timeout-ms']); + const effectiveIdleMs = Number.isFinite(cftIdleMs) && cftIdleMs > 0 + ? cftIdleMs + : idleDefaultMs; + + const cftCeilingMs = parseMs(template.cftConfig['request-timeout-ms']); + const settingsCeilingMs = settings.requestTimeoutMs; + // Negative or non-numeric β†’ fall through to settings; explicit 0 in + // cft β†’ disable ceiling entirely (Infinity, idle-only). + const effectiveCeilingMs = Number.isFinite(cftCeilingMs) + ? (cftCeilingMs > 0 ? cftCeilingMs : Infinity) + : (settingsCeilingMs > 0 ? settingsCeilingMs : Infinity); const { streamed, sources, images, truncated } = await streamPerplexityToFile( app, settings.perplexityApiKey, settings.perplexityEndpoint, payload, - effectiveTimeoutMs, + { idleMs: effectiveIdleMs, ceilingMs: effectiveCeilingMs }, target, initialContent, isCancelled, diff --git a/src/services/templateSeederService.ts b/src/services/templateSeederService.ts index 8272f14..a249340 100644 --- a/src/services/templateSeederService.ts +++ b/src/services/templateSeederService.ts @@ -7,9 +7,12 @@ import vocabularyProfile from '../docs/templates/vocabulary-profile.md'; import sourceProfile from '../docs/templates/source-profile.md'; import toolkitProfile from '../docs/templates/toolkit-profile.md'; import marketMapProfile from '../docs/templates/market-map-profile.md'; +import standardsAndSpecsProfile from '../docs/templates/standards-and-specs-profile.md'; +import marketCategoryProfile from '../docs/templates/market-category-profile.md'; import partialsReadme from '../docs/partials/README.md'; import mermaidDisciplinePartial from '../docs/partials/mermaid-discipline.md'; +import latexDisciplinePartial from '../docs/partials/latex-discipline.md'; import preamblesReadme from '../docs/preambles/README.md'; import inlineCitationPreamble from '../docs/preambles/inline-citation.md'; @@ -32,11 +35,14 @@ const TEMPLATE_FILES: SeedFile[] = [ { name: 'source-profile.md', content: sourceProfile }, { name: 'toolkit-profile.md', content: toolkitProfile }, { name: 'market-map-profile.md', content: marketMapProfile }, + { name: 'standards-and-specs-profile.md', content: standardsAndSpecsProfile }, + { name: 'market-category-profile.md', content: marketCategoryProfile }, ]; const PARTIALS_README: SeedFile = { name: 'README.md', content: partialsReadme }; const PARTIAL_FILES: SeedFile[] = [ { name: 'mermaid-discipline.md', content: mermaidDisciplinePartial }, + { name: 'latex-discipline.md', content: latexDisciplinePartial }, ]; const PREAMBLES_README: SeedFile = { name: 'README.md', content: preamblesReadme }; diff --git a/versions.json b/versions.json index ec721e2..68f13f0 100644 --- a/versions.json +++ b/versions.json @@ -3,5 +3,6 @@ "0.1.1": "1.8.10", "0.1.2": "1.8.10", "0.2.0": "1.8.10", - "0.2.1": "1.8.10" + "0.2.1": "1.8.10", + "0.3.0": "1.8.10" } \ No newline at end of file