From 3ae4012f1fb8f506d444d8628255024a8b2a2732 Mon Sep 17 00:00:00 2001 From: mpstaton Date: Fri, 22 May 2026 15:28:53 -0500 Subject: [PATCH] =?UTF-8?q?feat(release):=200.2.1=20=E2=80=94=20deep=20res?= =?UTF-8?q?earch=20that=20actually=20keeps=20the=20research?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Directory-template runs with sonar-deep-research were silently discarding ~99% of every response. Perplexity's deep-research model delivers its whole finished document in the first SSE event's message.content; the streaming reader took delta.content (correct for the token-streaming sonar models) and captured only a ~75-character tail fragment of a ~10,000-character report. toolkit-profile shipped with sonar-deep-research as its default model, so the default was the broken one. This release fixes the streaming, hardens it against mid-stream disconnects, and adds the tooling the directory-template research workflow was missing. Both streaming paths now read Perplexity's cumulative message.content snapshot and append only the new tail — correct for token-streaming sonar models and for deep research's first-event document dump alike. A timed-out or dropped stream now keeps the partial content and the citations it already received instead of discarding the whole run. "Apply directory template" gained a run dialog with a per-run model selector, so the model is no longer locked to the template file, and the loading notice names the model that is actually running instead of a hardcoded "deep research" string. Streaming (perplexityService.ts, directoryTemplateService.ts): - Read message.content (cumulative snapshot), diff against written text, append the tail; a startsWith guard skips a changed prefix rather than corrupting the note. - streamPerplexityToFile returns a partial result with a `truncated` flag on a read error instead of throwing; applyTemplate writes the partial with its sources footer and a truncation notice. The modal catch block salvages its citation metadata the same way. - Deep-research idle timeout 90s -> 270s; request-timeout default 5 -> 10 min. Model selection (DirectoryTemplateRunModal.ts, main.ts): - New run dialog: a template dropdown plus a Perplexity model dropdown that defaults to the template's cft model and overrides it for that run only. - cf_last_run_model now stamps the model that actually ran. Search scoping (directoryTemplateService.ts): - search_domain_filter support via cf_search_domains (target frontmatter) and search-domains (cft block); allow/deny entries, capped at 10; job boards are denied automatically. Template (src/docs/templates/toolkit-profile.md): - System prompt rebuilt: anchors the entity on its real name rather than the scraped marketing tagline, makes source quality a triage task instead of an allow/deny list, adds a search-don't-recall rule and per-section length ceilings. Default model -> sonar-pro. Files changed: - src/services/perplexityService.ts - src/services/directoryTemplateService.ts - src/modals/DirectoryTemplateRunModal.ts - main.ts - src/docs/templates/toolkit-profile.md - manifest.json - package.json - versions.json - changelog/releases/0.2.1.md - release-notes/0.2.1.md Also included: - manifest.json description aligned with package.json ("Perplexica (now Vane)"). Co-Authored-By: Claude Opus 4.7 (1M context) --- changelog/releases/0.2.1.md | 110 ++++++++++++++++++++ main.ts | 27 +++-- manifest.json | 4 +- package.json | 4 +- release-notes/0.2.1.md | 69 ++++++++++++ src/docs/templates/toolkit-profile.md | 122 +++++++++++++++++++--- src/modals/DirectoryTemplateRunModal.ts | 125 ++++++++++++++++++++++ src/services/directoryTemplateService.ts | 127 ++++++++++++++++++++--- src/services/perplexityService.ts | 87 +++++++++++----- versions.json | 3 +- 10 files changed, 606 insertions(+), 72 deletions(-) create mode 100644 changelog/releases/0.2.1.md create mode 100644 release-notes/0.2.1.md create mode 100644 src/modals/DirectoryTemplateRunModal.ts diff --git a/changelog/releases/0.2.1.md b/changelog/releases/0.2.1.md new file mode 100644 index 0000000..0f3c5d1 --- /dev/null +++ b/changelog/releases/0.2.1.md @@ -0,0 +1,110 @@ +--- +title: "Perplexed 0.2.1 — deep research that actually keeps the research" +lede: "Run a directory template with sonar-deep-research and you'd get a near-empty note — the streaming reader was taking the wrong field and discarding ~99% of every response before it reached the file. 0.2.1 fixes that, keeps the citations even when a long research stream drops mid-flight, adds a per-run model picker to the run dialog, and rebuilds the toolkit-profile template so the model searches the product's real name and triages SEO slop instead of citing it." +date_authored_initial_draft: 2026-05-22 +date_authored_current_draft: 2026-05-22 +date_first_published: 2026-05-22 +date_last_updated: 2026-05-22 +at_semantic_version: 0.2.1 +status: Published +category: Release +tags: + - Release-Narrative + - Directory-Templates + - Deep-Research + - SSE-Streaming + - Citation-Resilience + - Source-Quality + - Model-Picker + - Obsidian-Community-Plugin +authors: + - Michael Staton +augmented_with: Claude Code (Opus 4.7, 1M context) +release_tag: "0.2.1" +release_url: "https://github.com/lossless-group/perplexed-plugin/releases/tag/0.2.1" +prior_release_tag: "0.2.0" +release_commit: "pending — stamp after the release commit" +--- + +## Why care? + +If you generate research notes in Obsidian with Perplexed's directory templates — the "Apply directory template to current file" workflow that fills a `Tooling/` or concept file from a heading skeleton — 0.2.1 is the release where the deep-research path actually works. + +**The headline fix: `sonar-deep-research` runs now keep their output.** Perplexity's deep-research model does all its work server-side, then delivers the entire finished document — often 10,000+ characters — in the *first* event of its streaming response, inside a field the plugin wasn't reading. The streaming reader took `delta.content`, which is correct for the lighter `sonar` models that stream token-by-token, and on a deep-research response that field holds only a ~75-character tail fragment of a 10,000-character report. The other 99% arrived on the wire and was thrown away. If you ever ran a directory template on the deep-research model and got a near-empty note, that was this bug — and `toolkit-profile` shipped with `sonar-deep-research` as its *default* model, so it was the default that was broken. Fixed. + +**Citations survive a dropped connection.** Deep research can hold a stream open for minutes; a WiFi roam or an idle socket mid-stream used to discard everything — including the citations, which deep research delivers up front in that same first event. A cut-off run now keeps whatever content and sources already arrived, writes the citations footer anyway, and tells you it was truncated so you can re-run for the rest. + +**You pick the model when you run, not in a file you have to remember to edit.** "Apply directory template" now opens a run dialog with a model dropdown. And the loading toast finally names the model that's actually running, instead of the hardcoded "deep research" string it had been showing regardless. + +## What's new? + +### Deep-research streaming — read the snapshot, not the fragment + +Both streaming paths (the directory-template runner and the modal "Ask Perplexity" command) now treat Perplexity's cumulative `message.content` snapshot as the source of truth, diff it against what's already been written, and append only the new tail. This is correct for every model: + +| Model | How it streams | Old reader (`delta.content` only) | 0.2.1 reader (`message.content` snapshot) | +|---|---|---|---| +| `sonar`, `sonar-pro` | Token-by-token deltas | Worked | Works | +| `sonar-deep-research` | Whole document in event 1, then tiny deltas | Captured ~75 of ~10,234 chars | Captures the full document | + +A `startsWith` guard means a prefix that ever changes is skipped rather than corrupting the note — Perplexity's stream is append-only in practice, but the reader doesn't assume it. + +### Citations that survive a timeout + +`streamPerplexityToFile` no longer throws away a run when the socket drops. It catches the read error, returns whatever `streamed` / `sources` / `images` it has, and flags the run `truncated`. `applyTemplate` then writes the partial content *with* its sources footer and shows a notice — "stream cut off … saved partial content with N sources. Re-run to complete." The modal path got the same treatment: its catch block now salvages the citation metadata it already received before writing the error line. + +The deep-research idle timeout was also lengthened — 90s → 270s in the modal path, since deep research legitimately holds the socket silent for minutes while it researches server-side; 90s was firing on healthy runs. + +### A model picker in the run dialog + +"Apply directory template to current file" used to be a bare fuzzy template picker. It now opens `DirectoryTemplateRunModal`: a template dropdown (when more than one template matches the file), a **model dropdown** — `sonar-pro`, `sonar-reasoning-pro`, `sonar`, `sonar-reasoning`, `sonar-deep-research` — and Run / Cancel. The model defaults to the template's `cft` `model:` and overrides it for that run only; the template file is never touched. The `cf_last_run_model` frontmatter stamp now records the model that *actually ran*, override included. + +### Domain scoping — `cf_search_domains` + +New `search_domain_filter` support. A target file can carry a `cf_search_domains:` list in its frontmatter (entity-specific), and a template can carry a `search-domains:` key in its `cft` block (generic). Plain entries allowlist, a leading `-` denylists, capped at Perplexity's 10-domain limit. Job boards — Indeed, Glassdoor, ZipRecruiter, USAJobs — are denied on every run automatically, since a job listing is never a product source. + +Allowlisting is positioned as a *last resort*, not the workflow: hand-curating domains is the same labour as searching yourself. It's there for names so collision-prone that nothing else recovers them — `NATS` the messaging system versus `NATS` the UK air traffic authority being the canonical case. + +### The `toolkit-profile` template, rebuilt + +The shipped `toolkit-profile` template got a full system-prompt overhaul, because the old one was quietly steering the model wrong: + +- **Anchored on the real name, not the scraped tagline.** The prompt used `{{title}}` as the entity identity — but in scraped Tooling files `title` is the page's `og:title`, a marketing tagline. Profiling `Kubernetes.md` told the model the entity was named "Production-Grade Container Orchestration," so it searched the tagline. The prompt now anchors on `{{basename}}` (the filename = the real name) and explicitly flags `{{title}}` as a tagline. +- **Source-quality is a triage task, not an allow/deny list.** The old prompt hard-banned "consulting vendors." The new one tells the model it's the editor: search rankings are SEO-driven, so judge each page on its merits — favour editorial authority (the entity's own docs, standards bodies, established trade press) over rank, and treat a vendor's blog post as fine *if the page itself is substantive*. The only hard reject is a page about a different same-named entity. +- **"Search, don't recall."** An explicit instruction that training knowledge is stale and uncitable, and that a profile assembled from memory rather than search results is a failed profile. +- **Length discipline.** A `LENGTH AND ECONOMY` block turns the per-section sentence/paragraph counts already in the skeleton into hard ceilings, bans filler, and tells the model a thin section should be short — "length is not evidence of quality." +- **Default model is now `sonar-pro`** — fast, clean, grounded — with `sonar-deep-research` a dialog click away when you want depth. + +### Longer request timeout + +The directory-template request timeout default moved from 5 minutes to 10 (`DEFAULT_SETTINGS` in `main.ts`), so a `sonar-deep-research` run picked from the model dialog has room to finish. + +## Under the hood + +0.2.1 is a single concentrated session of work — there are no intermediate commits between the 0.2.0 release and this one. The changeset, eight files: + +| File | Change | +|---|---| +| `src/services/perplexityService.ts` | `message.content`-snapshot streaming; metadata salvage in the catch block; deep-research idle timeout 90s → 270s | +| `src/services/directoryTemplateService.ts` | Same streaming fix; `truncated` partial-result salvage; `search_domain_filter` support (`cf_search_domains` + `search-domains` + job-board denylist); per-run model override | +| `src/modals/DirectoryTemplateRunModal.ts` *(new)* | The template + model run dialog | +| `main.ts` | Run-modal wiring; 10-minute timeout default | +| `src/docs/templates/toolkit-profile.md` | Rebuilt system prompt — entity anchoring, source triage, concision, `sonar-pro` default | +| `manifest.json`, `package.json`, `versions.json` | 0.2.1; description aligned ("Perplexica (now Vane)") | + +The streaming bug was found the honest way — a raw `curl` against `sonar-deep-research` with `stream: true`, watching the SSE events scroll past. Event one carried a 10,169-character `message.content` and a 10-character `delta.content`. That was the whole diagnosis. + +## What's next + +- **Validate the rebuilt template on clean runs.** The `toolkit-profile` overhaul ships in this release; the next pass confirms entity-anchoring and source-triage hold up across a spread of `Tooling/` entries. +- **Carry the prompt rebuild to the other three templates.** `concept-profile`, `vocabulary-profile`, and `source-profile` still use the old prompt shape and likely mis-anchor the same way. The entity-name fix and the triage instruction generalize directly. +- **A `max-tokens` ceiling.** If prompt-level concision isn't enough to rein in deep research, a `max-tokens:` `cft` key would give a hard cap — held back here because a hard cap truncates mid-sentence and prompt discipline is the cleaner lever. +- **Diff-aware regeneration.** Still true from 0.2.0's list: a re-run on a file that already has `cf_last_run` stamped should offer to *replace* rather than silently append. Right now stacked re-runs pile up. + +## References + +- **Release on GitHub:** https://github.com/lossless-group/perplexed-plugin/releases/tag/0.2.1 +- **Install from Obsidian:** Settings → Community Plugins → Browse → "Perplexed" +- **Prior release:** [`0.2.0`](./0.2.0.md) — Google Gemini joins the provider lineup +- **Perplexity models:** [docs.perplexity.ai](https://docs.perplexity.ai/) — `sonar`, `sonar-pro`, `sonar-reasoning-pro`, `sonar-deep-research` +- **Sibling Lossless plugins:** [`image-gin`](https://github.com/lossless-group/image-gin), [`cite-wide`](https://github.com/lossless-group/cite-wide), [`metafetch`](https://community.obsidian.md/plugins/metafetch) diff --git a/main.ts b/main.ts index 5adcef9..920453c 100644 --- a/main.ts +++ b/main.ts @@ -20,6 +20,8 @@ import { ArticleGeneratorModal } from './src/modals/ArticleGeneratorModal'; import { TextEnhancementModal } from './src/modals/TextEnhancementModal'; import { TextEnhancementWithImagesModal } from './src/modals/TextEnhancementWithImagesModal'; import { DirectoryTemplatePickerModal } from './src/modals/DirectoryTemplatePickerModal'; +import { DirectoryTemplateRunModal } from './src/modals/DirectoryTemplateRunModal'; +import type { TemplateRunChoice } from './src/modals/DirectoryTemplateRunModal'; import { FolderPickerModal } from './src/modals/FolderPickerModal'; import { BatchConfirmModal } from './src/modals/BatchConfirmModal'; @@ -328,7 +330,7 @@ Structure the article as follows: { name: 'image-placement', when: 'return-images' }, ], directoryTemplatesFrontmatterWhitelist: ['title', 'og_description', 'tags', 'og_image'], - directoryTemplatesRequestTimeoutMs: 300000, + directoryTemplatesRequestTimeoutMs: 600000, // Find images for selection findImagesMaxImages: 3 @@ -1242,15 +1244,20 @@ export default class PerplexedPlugin extends Plugin { const dirSettings: DirectoryTemplateSettings = this.buildDirectoryTemplateSettings(); - new DirectoryTemplatePickerModal(this.app, matching, (chosen) => { - void (async () => { - const parsed = await loadDirectoryTemplate(this.app, chosen.file); - if (!parsed) { - new Notice('Template parse error: missing or malformed cft block.'); - return; - } - await applyDirectoryTemplate(this.app, dirSettings, target, parsed); - })(); + // Load every matching template up front so the run modal can show + // each one's cft model as the default in the model selector. + const choices: TemplateRunChoice[] = []; + for (const tf of matching) { + const parsed = await loadDirectoryTemplate(this.app, tf.file); + if (parsed) choices.push({ template: parsed, title: tf.title }); + } + if (choices.length === 0) { + new Notice('Template parse error: missing or malformed cft block.'); + return; + } + + new DirectoryTemplateRunModal(this.app, choices, (template, model) => { + void applyDirectoryTemplate(this.app, dirSettings, target, template, { modelOverride: model }); }).open(); } diff --git a/manifest.json b/manifest.json index 1efce8f..e63f209 100644 --- a/manifest.json +++ b/manifest.json @@ -1,9 +1,9 @@ { "id": "perplexed", "name": "Perplexed", - "version": "0.2.0", + "version": "0.2.1", "minAppVersion": "1.8.10", - "description": "Generate source-cited research content from Perplexity, Anthropic Claude, Google Gemini, Perplexica, or local LM Studio — directly into your notes.", + "description": "Generate source-cited research content from Perplexity, Anthropic Claude, Google Gemini, Perplexica (now Vane), or local LM Studio — directly into your notes.", "author": "The Lossless Group", "authorUrl": "https://lossless.group", "fundingUrl": "https://buymeacoffee.com/losslessgroup", diff --git a/package.json b/package.json index 81a9110..82fd8b2 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "perplexed", - "version": "0.2.0", - "description": "Generate source-cited research content from Perplexity, Anthropic Claude, Google Gemini, Perplexica, or local LM Studio — directly into your notes.", + "version": "0.2.1", + "description": "Generate source-cited research content through prompts and templates. From Perplexity, Anthropic Claude, Google Gemini, Perplexica (now Vane), or local LM Studio — directly into your notes.", "main": "main.js", "scripts": { "dev": "node esbuild.config.mjs", diff --git a/release-notes/0.2.1.md b/release-notes/0.2.1.md new file mode 100644 index 0000000..9bd1850 --- /dev/null +++ b/release-notes/0.2.1.md @@ -0,0 +1,69 @@ +**Directory-template runs with `sonar-deep-research` were quietly discarding ~99% of every response — the streaming reader was taking the wrong field. 0.2.1 fixes that, keeps citations even when a long research stream drops mid-flight, adds a per-run model picker to the run dialog, and rebuilds the toolkit-profile template so the model searches the product's real name and triages SEO slop instead of citing it.** + +## Why care? + +If you generate research notes with Perplexed's directory templates — the "Apply directory template to current file" workflow — 0.2.1 is the release where the deep-research path actually works. + +**`sonar-deep-research` runs now keep their output.** Perplexity's deep-research model does all its work server-side, then delivers the entire finished document — often 10,000+ characters — in the *first* event of its streaming response, in a field the plugin wasn't reading. The reader took `delta.content` (correct for the lighter `sonar` models, which stream token-by-token); on a deep-research response that field holds only a ~75-character tail fragment of a 10,000-character report. The other 99% arrived on the wire and was thrown away. `toolkit-profile` shipped with `sonar-deep-research` as its *default* model — so it was the default that was broken. Fixed. + +**Citations survive a dropped connection.** Deep research holds a stream open for minutes; a WiFi roam or idle socket mid-stream used to discard everything, including the citations deep research delivers up front. A cut-off run now keeps whatever content and sources arrived, writes the citations footer anyway, and tells you it was truncated. + +**You pick the model when you run.** "Apply directory template" now opens a run dialog with a model dropdown — and the loading toast finally names the model actually running, instead of a hardcoded "deep research" string. + +## What's new + +### Deep-research streaming — read the snapshot, not the fragment + +Both streaming paths now treat Perplexity's cumulative `message.content` snapshot as the source of truth and append only the new tail. + +| Model | How it streams | Old reader | 0.2.1 reader | +|---|---|---|---| +| `sonar`, `sonar-pro` | Token-by-token deltas | Worked | Works | +| `sonar-deep-research` | Whole document in event 1, then tiny deltas | ~75 of ~10,234 chars | Full document | + +### Citations that survive a timeout + +A dropped or timed-out stream no longer discards the run. The runner returns its partial content and the sources it already received, writes the citations footer, and shows a "stream cut off — saved partial content with N sources, re-run to complete" notice. The deep-research idle timeout was lengthened (90s → 270s) since deep research legitimately holds the socket silent while it researches. + +### A model picker in the run dialog + +"Apply directory template to current file" opens a dialog with a template dropdown and a **model dropdown** — `sonar-pro`, `sonar-reasoning-pro`, `sonar`, `sonar-reasoning`, `sonar-deep-research`. The model defaults to the template's `cft` `model:` and overrides it for that run only. The `cf_last_run_model` frontmatter stamp records the model that actually ran. + +### Domain scoping — `cf_search_domains` + +New `search_domain_filter` support: a `cf_search_domains:` list in a target file's frontmatter (entity-specific) or a `search-domains:` key in a template's `cft` block (generic). Plain entries allowlist, a leading `-` denylists, capped at 10. Job boards (Indeed, Glassdoor, ZipRecruiter, USAJobs) are denied automatically — a job listing is never a product source. Allowlisting is a last resort for collision-heavy names, not the everyday workflow. + +### The toolkit-profile template, rebuilt + +The shipped `toolkit-profile` template's system prompt was overhauled: it anchors the entity on its real name (`{{basename}}`) instead of the scraped marketing tagline; it tells the model to triage source quality — editorial authority over SEO ranking, judge each page on its merits — rather than trusting or hard-banning by domain; it adds a hard "search, don't recall" instruction and per-section length ceilings. Default model is now `sonar-pro`, with `sonar-deep-research` one dialog click away. + +### Longer request timeout + +The directory-template request timeout default moved 5 minutes → 10, so a `sonar-deep-research` run has room to finish. + +## Verify the release assets + +Every release asset is signed via [GitHub's artifact attestation system](https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations). Before manual install, verify provenance: + +```bash +gh attestation verify main.js --repo lossless-group/perplexed-plugin +gh attestation verify manifest.json --repo lossless-group/perplexed-plugin +gh attestation verify styles.css --repo lossless-group/perplexed-plugin +``` + +Exit code 0 means the file came out of a legitimate build of the source repo. + +## The Lossless Group plugin family + +| Plugin | What it does | +|---|---| +| [**Cite Wide**](https://community.obsidian.md/plugins/cite-wide) | Stable hex-identifier citations + URL-based dedupe + LLM-paste research conversion | +| [**Image Gin**](https://community.obsidian.md/plugins/image-gin) | Recraft + Ideogram image generation, Magnific stock search, ImageKit CDN, drag-gate | +| **Perplexed** *(this release)* | Source-cited research from Perplexity, Claude, Gemini, Perplexica, or LM Studio + per-directory templates | +| [**Metafetch**](https://community.obsidian.md/plugins/metafetch) | Pull OpenGraph metadata into note frontmatter via OpenGraph.io or Microlink | + +If Perplexed saves you time, [buy us a coffee](https://buymeacoffee.com/losslessgroup). The plugins are free; the coffee keeps the next one coming. + +## Full release narrative + +Marketing-grade narrative with the deeper context, the full eight-file changeset, and what's next: [`changelog/releases/0.2.1.md`](https://github.com/lossless-group/perplexed-plugin/blob/main/changelog/releases/0.2.1.md). diff --git a/src/docs/templates/toolkit-profile.md b/src/docs/templates/toolkit-profile.md index 766f5f6..cb2dbaf 100644 --- a/src/docs/templates/toolkit-profile.md +++ b/src/docs/templates/toolkit-profile.md @@ -3,46 +3,130 @@ applies-to-paths: - "Tooling/**" description: Generates a structured profile for a company / service / app / open-source repo. +date_created: 2026-05-09 +date_modified: 2026-05-22 --- # About this template -Use this for files under `Tooling/` whose body is empty or near-empty. Run "Apply directory template to current file" while viewing the target. The runtime sends the heading skeleton below as the user prompt to Perplexity Deep Research; the model returns markdown that follows the structure with inline citations. +Use this for files under `Tooling/` whose body is empty or near-empty. Run "Apply directory template to current file" while viewing the target — a dialog lets you pick the Perplexity model (the `model:` below is the default). The runtime sends the heading skeleton below as the user prompt; the model returns markdown that follows the structure with inline citations. + +Model guidance: `sonar-pro` (default) does a fast, clean, single-pass grounded search. `sonar-deep-research` does genuine multi-query research across many sources — pick it from the dialog when you want depth and can wait a few minutes. Avoid `sonar-reasoning-pro` here: reasoning models spend their budget thinking rather than searching and dump a large `` block into the file. The bullets under each heading are *instructions to the model*, not literal content. The model fills each section based on those instructions. +## Scoping search to the right entity + +Many tool/company names collide with unrelated namesakes (e.g. `NATS` the messaging system vs. `NATS` the UK air traffic authority). Two levers keep the model on-target: + +- The `system:` prompt above hard-pins the entity to its **name** (`{{basename}}`, the filename) and `{{url}}` — *not* the scraped `{{title}}`, which is usually a marketing tagline (e.g. Kubernetes' title is "Production-Grade Container Orchestration"). It then tells the model to **triage source quality itself** — favour editorial authority over SEO ranking, judge each page on its merits — rather than trusting the top search hits. Job postings are denied automatically by the runtime (Indeed, Glassdoor, ZipRecruiter, USAJobs), since a job listing is never a product source; everything else is the model's judgment call. +- Allowlisting is a last resort, not the workflow — curating domains by hand is the same labour as searching yourself. Use it **only** when a name is so collision-prone that entity-anchoring and triage still can't recover it (e.g. `NATS` the messaging system vs. `NATS` the UK air traffic authority). For those cases, add a **`cf_search_domains:`** list to the *target file's* frontmatter (not this template — this template is generic). Plain entries allowlist; a leading `-` denylists; max 10 total. Example for a `NATS.md` profiling `nats.io`: + + ```yaml + cf_search_domains: + - nats.io + - docs.nats.io + - github.com + - synadia.com + - cncf.io + ``` + + An allowlist restricts Perplexity to exactly those domains, so the wrong entity cannot come back. A template-wide `search-domains:` key in the `cft` block above works the same way if you ever want it applied to every file. + ```cft provider: perplexity -model: sonar-deep-research +model: sonar-pro search-recency: month return-citations: true return-images: false system: | - You are a research analyst profiling "{{title}}". Use Perplexity's web search - aggressively for every section. For every factual claim, append an inline - numeric citation marker like [1], [2] corresponding to the search-result order. - Quote phrasing from primary sources where useful. Prefer first-party sources, - official filings, and reputable industry publications. Use the entity's - existing metadata as starting context. + You are a research analyst writing a factual profile of ONE specific + entity and no other: the product, service, or company known as + "{{basename}}", whose canonical website is {{url}}. Note that "{{title}}" + is that website's tagline or page heading — useful context, but it is + NOT the entity's name. Search for, and reason about, "{{basename}}". + + SEARCH, do not recall. Your training knowledge is stale and is not + citable. Run a fresh web search for every section — value proposition, + architecture, history, funding, pricing, competitors, recent news — as + separate, specific queries. Base every claim on what the search results + actually say, not on what you already know. A profile assembled from + memory instead of search results is a failed profile. + + DISAMBIGUATION — read this before searching. Software, company, and + product names collide constantly. Many search results that match the + name "{{basename}}" will be about a DIFFERENT entity that merely shares + the name — a namesake company, a person, a place, a government body, an + acronym. Before using any result, confirm it is about the exact entity + at {{url}}. Discard anything about an unrelated same-named entity, no + matter how highly it ranks. Anchor every search query on "{{basename}}" + together with its own domain. + + SOURCE QUALITY — you are the editor; judge every source, do not just + take the top results. Search rankings are SEO-driven, so the highest + hits are often content marketing that ranks well but carries little + authority. Triage the full set of results you get and cite by merit. + + Higher authority — lead with these: the entity's own site, docs, GitHub + org, and changelog; official filings, regulatory documents, and standards + bodies; established journalism and trade press with real editorial + standards (e.g. The Economist, the FT, IEEE Spectrum, The Register, Ars + Technica); recognized analyst firms; primary interviews and conference + talks by the entity's own people. + + Lower authority — use only when nothing better covers a fact, and never + as the sole basis for a significant claim: thin "what is X" listicles and + content farms; scraper or aggregator pages; pages that exist to rank for + a keyword rather than to inform. Note: a consulting firm's or a vendor's + blog post is perfectly fine when that specific page is substantive, + first-hand, and accurate — judge the page on its merits, not the domain + it sits on. The only hard reject is a page about a different same-named + entity. + + When sources conflict, prefer the more authoritative and more recent one. + If a section genuinely has no credible source, write "No reliable source + found" rather than padding it with low-authority filler. + + LENGTH AND ECONOMY — this matters as much as accuracy. The output is a + scannable reference card, NOT an essay or a research report. Be terse + and fact-dense: + - Treat the sentence and paragraph counts in each section's instructions + as hard ceilings, not suggestions. "2-3 sentences" means at most three + sentences. "One short paragraph" means one. Never exceed them. + - One fact per sentence. Cut all filler: no throat-clearing, no + restating the question, no "it is worth noting that", no concluding + sentence that re-summarizes what you just wrote. + - Prefer bullets and tables over prose wherever a section allows it. + - A section with little substance should be SHORT. Never pad a thin + section to look thorough — a short correct section beats a long padded + one. Brevity is the goal; length is not evidence of quality. + - Output only the finished profile. Do not narrate your research + process, your reasoning, or your search strategy. + + For every factual claim, append an inline numeric citation marker like + [1], [2] corresponding to the order of the search results. Quote phrasing + from primary sources where useful. + + Entity metadata for grounding: + {{frontmatter}} ``` -# Features +# Value Proposition & Features +- Summarize the value proposition in 2-3 sentences. - Describe the core product features in 2–3 sentences each. - Bullet 5–8 features in priority order. -- If the product has a notable architecture, data flow, or pipeline that a diagram clarifies (e.g., an ingest → process → serve flow, an agent/orchestrator topology, a multi-tier deployment shape), render a `mermaid` codefence describing it. If a diagram does not add insight, omit it entirely — do not force one. - -{{include: mermaid-discipline}} ## Screenshots -- If three official screenshots are publicly available, list their URLs as bullets. -- For each, write a 1-sentence caption. If none are publicly available, write "No publicly available screenshots." +- If three official screenshots are publicly available, list their URLs wrapped in `![Alt Text](url)`, each one on a new line. +- For each, write a 1-sentence caption. If none are publicly available, skip. ## Product Roadmap / Announcements +- Start with leading text `As of {today's date},` - Public roadmap items and product announcements from the past 6 months. - Use dated bullets, most recent first. Cite each item. ## Recent Developments -- News and developments from the past 90 days. Cite sources inline. +- News and developments from the past 90 days. Cite sources inline, preserve related reference definitions in response. # History and Origin Story - Founding story, founders, key inflection points. One short paragraph. @@ -58,12 +142,15 @@ ## Notable Team Members # Market Sizing +## Category, Market Size, and Category Growth +- Define what category or categories this toolkit/tooling entry (the app, service, or company) is likely in based on available data. +- Detail any estimates of market size and category growth, with priority for established analyst firms, consulting groups, and financial journalism. ## Pricing - Markdown table of pricing tiers if published. - Note "no public pricing" if not. ## Revenue Trajectory Estimates -- Estimated or reported revenue / ARR. Cite source per figure. +- Estimated or reported revenue / ARR. If not available, skip. Cite source per figure. # Competitive Landscape @@ -73,6 +160,9 @@ ## Who it's for, who it's not for ## Viable Alternatives - 3–5 alternatives, one bullet each, with a brief rationale. +## Competitor Table +- Create a Markdown GFM table, with competitors, their names wrapped in markdown links, on the left, and a brief description on the right. + *** # User Notes diff --git a/src/modals/DirectoryTemplateRunModal.ts b/src/modals/DirectoryTemplateRunModal.ts new file mode 100644 index 0000000..af7b9d6 --- /dev/null +++ b/src/modals/DirectoryTemplateRunModal.ts @@ -0,0 +1,125 @@ +import { Modal, Setting, type App } from 'obsidian'; +import type { ParsedTemplate } from '../services/directoryTemplateService'; + +export interface TemplateRunChoice { + template: ParsedTemplate; + title: string; +} + +// Perplexity models offered in the run modal, in recommendation order. +const MODEL_CHOICES: { id: string; label: string }[] = [ + { id: 'sonar-pro', label: 'Sonar Pro — grounded search + citations, fast' }, + { id: 'sonar-reasoning-pro', label: 'Sonar Reasoning Pro — adds chain-of-thought' }, + { id: 'sonar', label: 'Sonar — lightweight, fastest' }, + { id: 'sonar-reasoning', label: 'Sonar Reasoning — lightweight + reasoning' }, + { id: 'sonar-deep-research', label: 'Sonar Deep Research — exhaustive, minutes-long' }, +]; + +function cftModel(choice: TemplateRunChoice): string { + const m = choice.template.cftConfig['model']; + return typeof m === 'string' && m.length > 0 ? m : 'sonar-pro'; +} + +/** + * Run dialog for "Apply directory template to current file": pick which + * template to apply (when more than one matches) and which Perplexity model + * to run it with. The model defaults to the template's cft `model:` but the + * selection here overrides it for this run only. + */ +export class DirectoryTemplateRunModal extends Modal { + private readonly choices: TemplateRunChoice[]; + private readonly onRun: (template: ParsedTemplate, model: string) => void; + private selectedIndex = 0; + private selectedModel: string; + + constructor( + app: App, + choices: TemplateRunChoice[], + onRun: (template: ParsedTemplate, model: string) => void, + ) { + super(app); + if (choices.length === 0) { + throw new Error('DirectoryTemplateRunModal requires at least one template choice'); + } + this.choices = choices; + this.onRun = onRun; + this.selectedModel = cftModel(this.selected()); + } + + /** The currently selected choice. choices is non-empty (checked in ctor). */ + private selected(): TemplateRunChoice { + const c = this.choices[this.selectedIndex]; + if (c) return c; + const first = this.choices[0]; + if (first) return first; + throw new Error('DirectoryTemplateRunModal: choices unexpectedly empty'); + } + + onOpen(): void { + this.render(); + } + + onClose(): void { + this.contentEl.empty(); + } + + private render(): void { + const { contentEl } = this; + contentEl.empty(); + contentEl.createEl('h3', { text: 'Apply directory template' }); + + if (this.choices.length > 1) { + new Setting(contentEl) + .setName('Template') + .setDesc('Which template to apply to this file.') + .addDropdown((dd) => { + this.choices.forEach((c, i) => { + dd.addOption(String(i), c.title); + }); + dd.setValue(String(this.selectedIndex)); + dd.onChange((v) => { + this.selectedIndex = Number(v); + // Reset the model to the newly chosen template's default. + this.selectedModel = cftModel(this.selected()); + this.render(); + }); + }); + } else { + new Setting(contentEl) + .setName('Template') + .setDesc(this.selected().title); + } + + const cftDefault = cftModel(this.selected()); + new Setting(contentEl) + .setName('Model') + .setDesc(`Perplexity model for this run. Template default: ${cftDefault}.`) + .addDropdown((dd) => { + for (const m of MODEL_CHOICES) dd.addOption(m.id, m.label); + // Keep an unlisted cft model selectable if the template names one. + if (!MODEL_CHOICES.some((m) => m.id === this.selectedModel)) { + dd.addOption(this.selectedModel, this.selectedModel); + } + dd.setValue(this.selectedModel); + dd.onChange((v) => { + this.selectedModel = v; + }); + }); + + new Setting(contentEl) + .addButton((b) => b + .setButtonText('Run') + .setCta() + .onClick(() => { + const chosen = this.selected(); + const model = this.selectedModel; + this.close(); + this.onRun(chosen.template, model); + })) + .addButton((b) => b + .setButtonText('Cancel') + .onClick(() => { + this.close(); + })); + } +} diff --git a/src/services/directoryTemplateService.ts b/src/services/directoryTemplateService.ts index 8dd1ac3..d8613bf 100644 --- a/src/services/directoryTemplateService.ts +++ b/src/services/directoryTemplateService.ts @@ -43,6 +43,7 @@ interface PerplexityPayload { return_images: boolean; return_related_questions: boolean; search_recency_filter?: string; + search_domain_filter?: string[]; } export interface PerplexitySource { @@ -413,13 +414,51 @@ export function interpolate(text: string, ctx: InterpolationContext): string { }); } +// Job boards and recruiting sites are never a credible source for a product +// profile, yet they rank highly for company names — especially names that +// collide with a large employer (e.g. "NATS" also = National Air Traffic +// Services). Denied on every run UNLESS the run declares an explicit +// allowlist: an allowlist already restricts results, and Perplexity caps +// search_domain_filter at 10 entries, so deny entries would only burn slots. +const JOB_BOARD_DENYLIST = [ + '-indeed.com', + '-glassdoor.com', + '-ziprecruiter.com', + '-usajobs.gov', +]; + +// Perplexity's hard cap on search_domain_filter length. +const SEARCH_DOMAIN_CAP = 10; + +/** + * Normalize a cft `search-domains:` value or a target file's + * `cf_search_domains:` frontmatter value into a clean string list. Accepts a + * YAML array or a comma-separated string. A leading `-` marks a denylist + * entry; everything else is an allowlist entry. + */ +function parseDomainList(raw: unknown): string[] { + const items = Array.isArray(raw) + ? raw + : typeof raw === 'string' + ? raw.split(',') + : []; + return items + .filter((x): x is string => typeof x === 'string') + .map((s) => s.trim()) + .filter((s) => s.length > 0); +} + function buildPayload( template: ParsedTemplate, systemPrompt: string, userPrompt: string, + targetDomains: string[] = [], + modelOverride?: string, ): PerplexityPayload { const cfg = template.cftConfig; - const model = typeof cfg.model === 'string' ? cfg.model : 'sonar-deep-research'; + const model = (modelOverride && modelOverride.length > 0) + ? modelOverride + : (typeof cfg.model === 'string' ? cfg.model : 'sonar-pro'); const recency = typeof cfg['search-recency'] === 'string' ? cfg['search-recency'] : undefined; @@ -438,6 +477,22 @@ function buildPayload( return_related_questions: false, }; if (recency) payload.search_recency_filter = recency; + + // search_domain_filter: template-level `search-domains:` (generic) + the + // target file's `cf_search_domains:` frontmatter (entity-specific) + the + // built-in job-board denylist. Declared entries win over the denylist on + // truncation to the 10-domain cap. + const declared = [ + ...parseDomainList(cfg['search-domains']), + ...targetDomains, + ]; + const hasAllowlist = declared.some((d) => !d.startsWith('-')); + const merged = hasAllowlist ? declared : [...declared, ...JOB_BOARD_DENYLIST]; + const domainFilter = [...new Set(merged)].slice(0, SEARCH_DOMAIN_CAP); + if (domainFilter.length > 0) { + payload.search_domain_filter = domainFilter; + } + return payload; } @@ -450,7 +505,7 @@ async function streamPerplexityToFile( file: TFile, initialContent: string, isCancelled: () => boolean, -): Promise<{ streamed: string; sources: PerplexitySource[]; images: PerplexityImage[] }> { +): Promise<{ streamed: string; sources: PerplexitySource[]; images: PerplexityImage[]; truncated: boolean }> { payload.stream = true; const controller = new AbortController(); const timer = activeWindow.setTimeout(() => controller.abort(), timeoutMs); @@ -489,6 +544,7 @@ async function streamPerplexityToFile( let streamed = ''; let sources: PerplexitySource[] = []; let images: PerplexityImage[] = []; + let truncated = false; let lastFlush = 0; const FLUSH_MS = 500; @@ -498,7 +554,20 @@ async function streamPerplexityToFile( controller.abort(); break; } - const { value, done } = await reader.read(); + let value: Uint8Array | undefined; + let done = false; + try { + ({ value, done } = await reader.read()); + } catch { + // Timeout (AbortController fired), user cancel, or a dropped + // socket. Don't throw away what already arrived — + // sonar-deep-research delivers the whole document plus its + // citations in the FIRST SSE event, so a later disconnect + // should still leave a usable, cited file. Mark the run + // truncated and fall through to the final flush + return. + truncated = true; + break; + } if (done) break; sseBuffer += decoder.decode(value, { stream: true }); @@ -516,10 +585,23 @@ async function streamPerplexityToFile( const choices = obj['choices']; if (Array.isArray(choices) && choices.length > 0) { const first = choices[0] as Record | undefined; + const message = first?.['message'] as Record | undefined; const delta = first?.['delta'] as Record | undefined; - const content = delta?.['content']; - if (typeof content === 'string') { - streamed += content; + const snapshot = message?.['content']; + const fragment = delta?.['content']; + // Perplexity streams a *cumulative* message.content + // snapshot on every SSE event. sonar-deep-research does + // all its research server-side and dumps the ENTIRE + // document into the first event's message.content, while + // delta.content only ever carries a short tail fragment + // — so reading delta alone silently loses ~99% of the + // response. Prefer the snapshot; fall back to delta. + if (typeof snapshot === 'string' + && snapshot.length > streamed.length + && snapshot.startsWith(streamed)) { + streamed = snapshot; + } else if (typeof fragment === 'string') { + streamed += fragment; } } const sr = obj['search_results']; @@ -555,7 +637,7 @@ async function streamPerplexityToFile( // Final flush of raw stream content before post-processing await app.vault.modify(file, initialContent + streamed); - return { streamed, sources, images }; + return { streamed, sources, images, truncated }; } export type ApplyOutcome = @@ -566,6 +648,8 @@ export type ApplyOutcome = export interface ApplyOptions { quiet?: boolean; isCancelled?: () => boolean; + /** Per-run Perplexity model; overrides the template's cft `model:`. */ + modelOverride?: string; } export async function applyTemplate( @@ -667,17 +751,31 @@ export async function applyTemplate( ? `${fmBlock}\n` : `${fmBlock}\n${existingBody}\n\n`; - const payload = buildPayload(template, systemPrompt, userPrompt); + // Effective model: an explicit per-run override (from the run modal) wins + // over the template's cft `model:`. Used for the payload, the loading + // notice, and the cf_last_run_model frontmatter stamp. + const cftModel = typeof template.cftConfig['model'] === 'string' + ? template.cftConfig['model'] + : ''; + const effectiveModel = (options.modelOverride && options.modelOverride.length > 0) + ? options.modelOverride + : (cftModel || 'sonar-pro'); + + // Entity-specific domain hints live in the target file's frontmatter so a + // collision-heavy name (e.g. NATS) can be pinned per-file without editing + // the shared template. + const targetDomains = parseDomainList(fm['cf_search_domains']); + const payload = buildPayload(template, systemPrompt, userPrompt, targetDomains, effectiveModel); let loadingNotice: Notice | null = null; if (!quiet) { - loadingNotice = new Notice('Streaming perplexity deep research…', 0); + loadingNotice = new Notice(`Streaming Perplexity · ${effectiveModel}…`, 0); } try { // Set initial state before streaming begins. await app.vault.modify(target, initialContent); - const { streamed, sources, images } = await streamPerplexityToFile( + const { streamed, sources, images, truncated } = await streamPerplexityToFile( app, settings.perplexityApiKey, settings.perplexityEndpoint, @@ -692,9 +790,7 @@ export async function applyTemplate( const provider = typeof template.cftConfig['provider'] === 'string' ? template.cftConfig['provider'] : 'unknown'; - const modelName = typeof template.cftConfig['model'] === 'string' - ? template.cftConfig['model'] - : 'unknown'; + const modelName = effectiveModel; const providerLabel = provider.length > 0 ? provider.charAt(0).toUpperCase() + provider.slice(1) : provider; @@ -758,6 +854,11 @@ export async function applyTemplate( }); if (!quiet) { + if (truncated) { + new Notice( + `"${target.basename}": stream cut off (timeout or lost connection) — saved partial content with ${sources.length.toString()} sources. Re-run to complete.`, + ); + } const verb = mode === 'fill' ? 'Filled' : 'Appended to'; new Notice(`${verb} "${target.basename}" using ${template.file.basename} (${sources.length.toString()} sources)`); } diff --git a/src/services/perplexityService.ts b/src/services/perplexityService.ts index 24ee3db..1455d9d 100644 --- a/src/services/perplexityService.ts +++ b/src/services/perplexityService.ts @@ -543,7 +543,7 @@ export class PerplexityService { } console.debug('✅ Streaming response received, starting to handle...'); - await this.handleStreamingResponse(response, editor, responseCursor, requestId, headerText); + await this.handleStreamingResponse(response, editor, responseCursor, requestId, headerText, isDeepResearch); } else { console.debug('🔄 Making non-streaming API request...'); // Use Obsidian's request method for non-streaming with cache busting @@ -627,16 +627,17 @@ export class PerplexityService { } private async handleStreamingResponse( - response: Response, - editor: Editor, + response: Response, + editor: Editor, responseCursor: { line: number; ch: number }, requestId?: number, - headerText?: string + headerText?: string, + isDeepResearch = false ): Promise { console.debug('🔄 handleStreamingResponse called'); const reader = response.body?.getReader(); if (!reader) throw new Error('No response body'); - + console.debug(`🔄 Starting streaming response handler [${requestId || 'unknown'}]`); // Hoist decoder out of the loop so the {stream: true} flag carries @@ -645,12 +646,17 @@ export class PerplexityService { let buffer = ''; const currentPos = { ...responseCursor }; let finalResponseData: PerplexityStreamChunk | null = null; + // Everything already written into the editor. Perplexity streams a + // cumulative message.content snapshot, so we diff against this. + let accumulated = ''; // Race each read against an idle timer — if no chunk arrives within // STREAM_IDLE_TIMEOUT_MS, throw so the catch surfaces a visible // "Streaming Error" instead of leaving reader.read() blocked forever - // on a dropped/idle SSE socket. - const STREAM_IDLE_TIMEOUT_MS = 90_000; + // on a dropped/idle SSE socket. sonar-deep-research does all its + // research server-side and can hold the socket silent for minutes + // before the first byte, so it gets a much longer leash. + const STREAM_IDLE_TIMEOUT_MS = isDeepResearch ? 270_000 : 90_000; const readWithIdleTimeout = (): Promise> => { let timer: number | undefined; const timeout = new Promise((_, reject) => { @@ -693,28 +699,41 @@ export class PerplexityService { }; } - if (parsed.choices?.[0]?.delta?.content) { - const content = parsed.choices[0].delta.content; - if (content) { - // console.debug('🎉 First content received! Clearing loading text...'); - // Clear any loading text before inserting content - this.clearLoadingText(editor); - - // Clear the animation interval if it exists - this.clearLoadingAnimation(); - - // console.debug(`📝 Inserting content at position:`, currentPos, `Content:`, content.substring(0, 50) + '...'); - editor.replaceRange(content, currentPos); - const contentLines = content.split('\n'); - if (contentLines.length === 1) { - currentPos.ch += content.length; - } else { - currentPos.line += contentLines.length - 1; - currentPos.ch = contentLines[contentLines.length - 1]?.length ?? 0; - } - // Scroll to follow the new content - editor.scrollIntoView({ from: currentPos, to: currentPos }, true); + // Perplexity streams a *cumulative* message.content + // snapshot on every SSE event. sonar-deep-research does + // all its research server-side and dumps the ENTIRE + // document into the first event's message.content, while + // delta.content only ever carries a short tail fragment + // — so reading delta alone silently drops ~99% of a deep + // research article. Treat message.content as the source + // of truth and write only the not-yet-written tail; fall + // back to accumulating delta.content if message is absent. + const choice = parsed.choices?.[0]; + let fullSoFar: string | undefined; + if (typeof choice?.message?.content === 'string') { + fullSoFar = choice.message.content; + } else if (typeof choice?.delta?.content === 'string') { + fullSoFar = accumulated + choice.delta.content; + } + // startsWith guard: Perplexity is append-only, but if a + // prefix ever changes we skip rather than corrupt the doc. + if (fullSoFar && fullSoFar.length > accumulated.length && fullSoFar.startsWith(accumulated)) { + const newText = fullSoFar.slice(accumulated.length); + // Clear any loading text / animation before first insert + this.clearLoadingText(editor); + this.clearLoadingAnimation(); + + editor.replaceRange(newText, currentPos); + const contentLines = newText.split('\n'); + if (contentLines.length === 1) { + currentPos.ch += newText.length; + } else { + currentPos.line += contentLines.length - 1; + currentPos.ch = contentLines[contentLines.length - 1]?.length ?? 0; } + // Scroll to follow the new content + editor.scrollIntoView({ from: currentPos, to: currentPos }, true); + accumulated = fullSoFar; } } } catch (e) { @@ -776,6 +795,18 @@ export class PerplexityService { this.clearLoadingText(editor); this.clearLoadingAnimation(); + + // Salvage citations/images already received before the drop — + // deep research delivers them in its first SSE event, so a later + // disconnect shouldn't discard a footer we already have the data + // for. processStreamingMetadata appends at the document tail. + if (finalResponseData) { + try { + this.processStreamingMetadata(finalResponseData, editor, headerText); + } catch (metaErr) { + console.warn('Could not salvage streaming metadata:', metaErr); + } + } editor.replaceRange(`\n**Streaming Error:** ${userMsg}\n\n`, currentPos); } } diff --git a/versions.json b/versions.json index a814160..ec721e2 100644 --- a/versions.json +++ b/versions.json @@ -2,5 +2,6 @@ "0.1.0": "0.15.0", "0.1.1": "1.8.10", "0.1.2": "1.8.10", - "0.2.0": "1.8.10" + "0.2.0": "1.8.10", + "0.2.1": "1.8.10" } \ No newline at end of file