mirror of
https://github.com/logancyang/obsidian-copilot.git
synced 2026-07-22 07:50:24 +00:00
Deprecate IntentAnalyzer (#2069)
* Deprecate IntentAnalyzer and replace with model-based tool planning in CopilotPlusChainRunner * Refine planning prompt to enforce XML-only responses and remove note extraction logic
This commit is contained in:
Logan Yang
GitHub
parent
71bf510acb
commit
1d668d3367
6 changed files with 337 additions and 258 deletions
|
|
@ -1,6 +1,6 @@
|
|||
# Deprecating `IntentAnalyzer` and Broca
|
||||
|
||||
This document captures the current responsibilities of the Copilot Plus intent analysis flow (`IntentAnalyzer` + Brevilabs “broca” API) and outlines a safe migration path to remove it while keeping Copilot Plus functionality aligned with the autonomous agent experience.
|
||||
This document captures the current responsibilities of the Copilot Plus intent analysis flow (`IntentAnalyzer` + Brevilabs “broca” API) and outlines a safe migration path to remove it while keeping Copilot Plus functionality aligned with the autonomous agent experience. The end state is simple: the user-facing chat model (same family as the agent chain) owns tool planning, salient term extraction, and time-expression handling; Broca is fully removed.
|
||||
|
||||
## Current Responsibilities
|
||||
|
||||
|
|
@ -15,7 +15,7 @@ This document captures the current responsibilities of the Copilot Plus intent a
|
|||
- `@websearch` / `@web` → forces `webSearch`.
|
||||
- `@memory` → forces `updateMemory`.
|
||||
- **Plus-specific salient term injection**: any Broca `salience_terms` array is passed into `localSearch` unchanged, altering recall compared with the agent chain. Our discovery shows this causes divergence between Plus and agent results; we need both flows to end up using the same salient term set.
|
||||
- **Implicit license enforcement**: `/broca` is the only per-turn API call guaranteed to touch Brevilabs. Although license validation also uses `/license` (`checkIsPlusUser`), Broca acts as the continuous touch point that can detect expired keys mid-session.
|
||||
- **Implicit license enforcement**: `/broca` is the only per-turn API call guaranteed to touch Brevilabs. Although license validation also uses `/license` (`checkIsPlusUser`), Broca acts as the continuous touch point that can detect expired keys mid-session. **Today**: `checkIsPlusUser` already runs per turn in both Copilot Plus and Autonomous Agent chain runners; keep that behavior when Broca goes away (no extra moving parts).
|
||||
|
||||
## Known Consumers and Side Effects
|
||||
|
||||
|
|
@ -28,11 +28,12 @@ This document captures the current responsibilities of the Copilot Plus intent a
|
|||
|
||||
- **Feature parity**: Copilot Plus must keep working with vault search, timeline questions, web search, file tree lookup, and memory updates.
|
||||
- **License verification**: every chat turn must continue to check Plus eligibility (either by retaining a per-turn Brevilabs call or by adding an explicit `/license` check with caching and backoff).
|
||||
- **Minimal prompt drift**: Copilot Plus prompts currently do not include the aggressive tool-calling instructions used by the autonomous agent; the migration should not break existing prompt tuning until the replacement strategy is ready.
|
||||
- **Minimal prompt drift**: Copilot Plus prompts currently do not include the aggressive tool-calling instructions used by the autonomous agent; the migration should not break existing prompt tuning until the replacement strategy is ready. Prefer reusing existing agent instructions instead of inventing new ones.
|
||||
- **Zero regression for `@` commands**: the existing inline overrides are user-facing affordances.
|
||||
- **Search recall**: any solution must surface the same or better salient term quality as the agent chain to avoid regressions highlighted in recent investigations.
|
||||
- **Search recall**: any solution must surface the same or better salient term quality as the agent chain to avoid regressions highlighted in recent investigations; salient terms should come from the user chat model (agent-style extraction), not Broca.
|
||||
- **Plus/agent parity**: after migration, the Plus chain must feed the vault search pipeline with the same query string, expanded variants, and salient term list that the agent chain would generate for the identical user input.
|
||||
- **Scoped automatic detection**: only the utility tools without explicit UI affordances (`getCurrentTime`, `convertTimeBetweenTimezones`, `getTimeRangeMs`, `getTimeInfoByEpoch`, `getFileTree`) need automatic invocation; everything else can be triggered through toggles or `@tool` commands.
|
||||
- **Self-host readiness**: future “self-host” mode must be able to (a) bypass live license checks (or use a short-lived cached verification) and (b) avoid all Brevilabs API calls while keeping the rest of the product functional, without branching code paths everywhere.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
|
|
@ -40,32 +41,39 @@ This document captures the current responsibilities of the Copilot Plus intent a
|
|||
|
||||
1. **Instrument current intent decisions**: add temporary logging (behind debug flag) to record Broca output, executed tools, and overrides. Purpose: baseline behavior before refactor.
|
||||
2. **Map tool usage**: capture how frequently each Broca tool is invoked to prioritise feature parity work.
|
||||
3. **Clarify license expectations**: confirm with product whether `/license` checks can replace Broca for per-turn validation, or if a lighter “heartbeat” endpoint is needed.
|
||||
3. **Clarify license expectations**: confirm with product whether `/license` checks can replace Broca for per-turn validation, or if a lighter “heartbeat” endpoint is needed. (Current code already calls `/license` per turn via `checkIsPlusUser`; keep this as the baseline.)
|
||||
|
||||
### Phase 1 – Surface-Agnostic Tool Planning
|
||||
|
||||
4. **Extract shared tool planner interface**: design a new planner (e.g., `PlusToolPlanner`) that returns the same `{ tool, args }[]` shape as `IntentAnalyzer`. Plan for multiple implementations (Broca, agent-driven, heuristic).
|
||||
5. **Port `@` command handling**: move `processAtCommands` logic into the new planner so overrides are planner-agnostic.
|
||||
6. **Adopt agent-style salient term generation**: introduce a reusable salience extractor (likely the agent instruction set or a deterministic tokenizer) so Plus mode produces the same salient terms the agent chain would compute.
|
||||
7. **Define utility auto-detection rules**: implement deterministic heuristics for the remaining auto-triggered tools (time/time-range/time-info/file-tree) so the planner can schedule them without LLM help.
|
||||
4. **Extract shared tool planner interface**: design a single planner abstraction (e.g., `PlusToolPlanner`) that returns `{ tool, args }[]`. Default implementation uses the same chat-model planning as the agent; keep Broca only as a temporary hidden fallback.
|
||||
5. **Port `@` command handling**: move `processAtCommands` into the planner; keep `chatHistory` enrichment for `@websearch/@web`. One place, one behavior.
|
||||
6. **Adopt agent-style salient term generation**: reuse the agent chain’s salience extraction via the chat model. Remove Broca salience entirely; no parallel salience paths.
|
||||
7. **Utility auto-detection**: rely on the chat-model planner for time/time-range/time-info/file-tree, with a minimal heuristic fallback for obvious time expressions. Preserve project-mode guardrails (skip vault-level `getFileTree` in projects).
|
||||
|
||||
### Phase 2 – Replace Broca for Tool Scheduling
|
||||
|
||||
8. **Reuse ModelAdapter-driven planning**: embed a constrained agent loop (single-iteration tool planner) that leverages the existing XML instructions used by autonomous agent models. Limit to deciding tool calls; keep Copilot Plus response streaming as-is.
|
||||
9. **Fallback heuristics**: provide deterministic patterns for the utility tools to guard against planner failures and avoid unneeded LLM calls for searches already controlled by UI toggles.
|
||||
10. **Feature flag & dogfood**: gate the new planner behind a runtime toggle (`settings.debugPlanner` or remote flag) to test against Broca in parallel, and compare Plus/agent query-expansion outputs in telemetry to ensure they match.
|
||||
8. **Reuse ModelAdapter-driven planning**: embed a single-iteration agent-style planner that emits localSearch/webSearch/time tools with the same parameters (query + salience terms + timeRange) the agent chain would produce.
|
||||
9. **Fallback heuristics**: keep deterministic fallbacks for time expressions; compute timeRange once and pass it to localSearch. Avoid extra tool calls.
|
||||
10. **Feature flag & dogfood**: gate the new planner behind a toggle to compare against Broca; collect salience/time parity telemetry. Keep toggles minimal.
|
||||
|
||||
### Phase 3 – License Enforcement Replacement
|
||||
|
||||
11. **Introduce explicit per-turn license check**: call `validateLicenseKey` (or a new lightweight endpoint) at the start of each Plus turn. Cache results for the current conversation with TTL to avoid redundant traffic within a single turn.
|
||||
12. **Graceful degradation**: if validation fails or is unreachable, surface the same error handling as today (e.g., show invalid key notice, disable Plus features).
|
||||
11. **Per-turn license check**: keep the current per-turn `checkIsPlusUser` in Copilot Plus, Agent, and Projects. If you cache, cache per turn only. Centralize this so there is one path.
|
||||
12. **Graceful degradation**: if validation fails or is unreachable, show the same invalid-key flow. Add a single override hook to bypass in self-host mode (no scattered flags).
|
||||
|
||||
### Phase 4 – Removal & Cleanup
|
||||
|
||||
13. **Switch default planner**: once the new planner reaches parity, flip the feature flag so Copilot Plus no longer calls Broca. Keep Broca behind a hidden fallback flag for one release.
|
||||
14. **Delete `IntentAnalyzer`**: remove the class, its tests, and references. Update imports (`initializeBuiltinTools` already handles tool registration outside IntentAnalyzer).
|
||||
15. **Retire `BrevilabsClient.broca`**: delete the method and any unused types. Keep `/license` and other endpoints.
|
||||
16. **Documentation & migration notes**: update `AGENTS.md`, `TODO.md`, and any user-facing Plus documentation to note the new flow.
|
||||
13. **Switch default planner**: flip the feature flag when parity is reached; Broca stays as a hidden one-release fallback.
|
||||
14. **Delete `IntentAnalyzer`**: remove the class, tests, and references; drop `initTools` from `src/main.ts` (tool registration already handled elsewhere).
|
||||
15. **Retire `BrevilabsClient.broca`**: remove the method and unused types; keep `/license` and others.
|
||||
16. **Documentation & migration notes**: update `AGENTS.md`, `TODO.md`, and user-facing Plus docs.
|
||||
|
||||
### Phase 5 – Prepare for Self-Host Mode
|
||||
|
||||
17. **Abstract Brevilabs dependencies**: one provider interface for license, web search, rerank, youtube, url/pdf ingestion. Default = Brevilabs; self-host = offline/no-op or user-supplied.
|
||||
18. **Configurable license gate**: single setting for “offline verified” (short-lived cache) or “skip verification” (self-host). Copilot Plus/Projects should read from the provider, not from BrevilabsClient directly.
|
||||
19. **Feature degradation without Brevilabs**: define one behavior for missing endpoints (disable specific tools or route to user-provided endpoints) without branching everywhere. Keep telemetry resilient when Broca/Brevilabs events are absent.
|
||||
20. **Testing and telemetry updates**: cover provider modes (online vs self-host) and update dashboards for the absence of Broca events.
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
|
|
|
|||
|
|
@ -6,20 +6,6 @@ import { turnOffPlus, turnOnPlus } from "@/plusUtils";
|
|||
import { getSettings } from "@/settings/model";
|
||||
import { arrayBufferToBase64 } from "@/utils/base64";
|
||||
|
||||
export interface BrocaResponse {
|
||||
response: {
|
||||
tool_calls: Array<{
|
||||
tool: string;
|
||||
args: {
|
||||
[key: string]: any;
|
||||
};
|
||||
}>;
|
||||
salience_terms: string[];
|
||||
};
|
||||
elapsed_time_ms: number;
|
||||
detail?: string;
|
||||
}
|
||||
|
||||
export interface RerankResponse {
|
||||
response: {
|
||||
object: string;
|
||||
|
|
@ -261,21 +247,6 @@ export class BrevilabsClient {
|
|||
return { isValid: true, plan: data?.plan };
|
||||
}
|
||||
|
||||
async broca(userMessage: string, isProjectMode: boolean): Promise<BrocaResponse> {
|
||||
const { data, error } = await this.makeRequest<BrocaResponse>("/broca", {
|
||||
message: userMessage,
|
||||
is_project_mode: isProjectMode,
|
||||
});
|
||||
if (error) {
|
||||
throw error;
|
||||
}
|
||||
if (!data) {
|
||||
throw new Error("No data returned from broca");
|
||||
}
|
||||
|
||||
return data;
|
||||
}
|
||||
|
||||
async rerank(query: string, documents: string[]): Promise<RerankResponse> {
|
||||
const { data, error } = await this.makeRequest<RerankResponse>("/rerank", {
|
||||
query,
|
||||
|
|
|
|||
|
|
@ -19,10 +19,14 @@ import { getSettings, getSystemPromptWithMemory } from "@/settings/model";
|
|||
import { writeToFileTool } from "@/tools/ComposerTools";
|
||||
import { ToolManager } from "@/tools/toolManager";
|
||||
import { ToolResultFormatter } from "@/tools/ToolResultFormatter";
|
||||
import { ToolRegistry } from "@/tools/ToolRegistry";
|
||||
import { initializeBuiltinTools } from "@/tools/builtinTools";
|
||||
import { localSearchTool, webSearchTool } from "@/tools/SearchTools";
|
||||
import { updateMemoryTool } from "@/tools/memoryTools";
|
||||
import { extractChatHistory } from "@/utils";
|
||||
import { ChatMessage, ResponseMetadata } from "@/types/message";
|
||||
import { getApiErrorMessage, getMessageRole, withSuppressedTokenWarnings } from "@/utils";
|
||||
import { BaseChatModel } from "@langchain/core/language_models/chat_models";
|
||||
import { IntentAnalyzer } from "../intentAnalyzer";
|
||||
import { BaseChainRunner } from "./BaseChainRunner";
|
||||
import { ActionBlockStreamer } from "./utils/ActionBlockStreamer";
|
||||
import { addChatHistoryToMessages } from "./utils/chatHistoryUtils";
|
||||
|
|
@ -47,8 +51,261 @@ import {
|
|||
import { ThinkBlockStreamer } from "./utils/ThinkBlockStreamer";
|
||||
import { deduplicateSources } from "./utils/toolExecution";
|
||||
import { recordPromptPayload } from "./utils/promptPayloadRecorder";
|
||||
import { ModelAdapterFactory, joinPromptSections } from "./utils/modelAdapter";
|
||||
import { parseXMLToolCalls } from "./utils/xmlParsing";
|
||||
import { extractParametersFromZod, SimpleTool } from "@/tools/SimpleTool";
|
||||
import ProjectManager from "@/LLMProviders/projectManager";
|
||||
import { isProjectMode } from "@/aiParams";
|
||||
|
||||
type ToolCallWithExecutor = {
|
||||
tool: any;
|
||||
args: any;
|
||||
};
|
||||
|
||||
export class CopilotPlusChainRunner extends BaseChainRunner {
|
||||
/**
|
||||
* Get available tools for Copilot Plus chain.
|
||||
* Uses a minimal set of utility tools: time tools and file tree.
|
||||
* Search tools are handled via @commands.
|
||||
*/
|
||||
protected getAvailableToolsForPlanning(): SimpleTool<any, any>[] {
|
||||
const registry = ToolRegistry.getInstance();
|
||||
|
||||
// Initialize tools if not already done
|
||||
if (registry.getAllTools().length === 0) {
|
||||
initializeBuiltinTools(this.chainManager.app?.vault);
|
||||
}
|
||||
|
||||
// Get all tools as SimpleTool instances
|
||||
const allTools = registry.getAllTools().map((def) => def.tool);
|
||||
|
||||
// Return only utility tools that need automatic detection
|
||||
// Other tools (@vault, @websearch, @memory) are handled by @command logic
|
||||
return allTools.filter((tool) => {
|
||||
return (
|
||||
tool.name === "getCurrentTime" ||
|
||||
tool.name === "convertTimeBetweenTimezones" ||
|
||||
tool.name === "getTimeInfoByEpoch" ||
|
||||
tool.name === "getTimeRangeMs" ||
|
||||
tool.name === "getFileTree"
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate tool descriptions in XML format for the model.
|
||||
* Reuses the same format as autonomous agent.
|
||||
*/
|
||||
public static generateToolDescriptions(availableTools: SimpleTool<any, any>[]): string {
|
||||
return availableTools
|
||||
.map((tool) => {
|
||||
let params = "";
|
||||
|
||||
// Extract parameters from Zod schema
|
||||
const parameters = extractParametersFromZod(tool.schema);
|
||||
if (Object.keys(parameters).length > 0) {
|
||||
params = Object.entries(parameters)
|
||||
.map(([key, description]) => `<${key}>${description}</${key}>`)
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
return `<${tool.name}>
|
||||
<description>${tool.description}</description>
|
||||
<parameters>
|
||||
${params}
|
||||
</parameters>
|
||||
</${tool.name}>`;
|
||||
})
|
||||
.join("\n\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Use model-based planning to determine which tools to call.
|
||||
* Replaces the Broca API call with chat model tool planning.
|
||||
*/
|
||||
private async planToolCalls(
|
||||
userMessage: string,
|
||||
chatModel: BaseChatModel
|
||||
): Promise<{ toolCalls: ToolCallWithExecutor[]; salientTerms: string[] }> {
|
||||
const availableTools = this.getAvailableToolsForPlanning();
|
||||
const adapter = ModelAdapterFactory.createAdapter(chatModel);
|
||||
const toolDescriptions = CopilotPlusChainRunner.generateToolDescriptions(availableTools);
|
||||
|
||||
// Build a lightweight planning prompt
|
||||
const registry = ToolRegistry.getInstance();
|
||||
const toolMetadata = availableTools
|
||||
.map((tool) => registry.getToolMetadata(tool.name))
|
||||
.filter((meta): meta is NonNullable<typeof meta> => meta !== undefined);
|
||||
|
||||
const allSections = adapter.buildSystemPromptSections(
|
||||
"You are a helpful AI assistant. Analyze the user's message and determine if any tools should be called.",
|
||||
toolDescriptions,
|
||||
availableTools.map((t) => t.name),
|
||||
toolMetadata
|
||||
);
|
||||
|
||||
let planningPrompt = joinPromptSections(allSections);
|
||||
|
||||
// Add salient term extraction instruction to the planning prompt
|
||||
planningPrompt += `\n\nIMPORTANT: Respond ONLY with XML blocks. Do not include any natural language explanations, apologies, or commentary.
|
||||
|
||||
Your response must contain:
|
||||
1. Tool calls (if needed): <use_tool>...</use_tool>
|
||||
2. Salient terms (always): <salient_terms><term>keyword</term></salient_terms>
|
||||
|
||||
Format for salient terms:
|
||||
<salient_terms>
|
||||
<term>first keyword</term>
|
||||
<term>second keyword</term>
|
||||
</salient_terms>
|
||||
|
||||
Guidelines for salient terms:
|
||||
- Extract meaningful nouns, topics, and specific concepts
|
||||
- Exclude common words like "what", "how", "my", "the", "a", "an"
|
||||
- Exclude time expressions (those are handled separately)
|
||||
- If the message has no meaningful search terms, output empty <salient_terms></salient_terms>
|
||||
- Use the EXACT words from the user's message (preserve language/spelling)
|
||||
|
||||
OUTPUT ONLY XML - NO OTHER TEXT.`;
|
||||
|
||||
// Create a simple planning request
|
||||
const planningMessages = [
|
||||
{
|
||||
role: getMessageRole(chatModel),
|
||||
content: planningPrompt,
|
||||
},
|
||||
{
|
||||
role: "user",
|
||||
content: userMessage,
|
||||
},
|
||||
];
|
||||
|
||||
logInfo("[CopilotPlus] Requesting tool planning from model...");
|
||||
|
||||
// Get model response for planning
|
||||
const response = await withSuppressedTokenWarnings(() => chatModel.invoke(planningMessages));
|
||||
|
||||
const responseText =
|
||||
typeof response.content === "string" ? response.content : String(response.content);
|
||||
|
||||
logInfo("[CopilotPlus] Model planning response:", responseText.substring(0, 500));
|
||||
|
||||
// Parse tool calls from response
|
||||
const parsedCalls = parseXMLToolCalls(responseText);
|
||||
|
||||
// Extract salient terms from <salient_terms> block
|
||||
const salientTerms = this.extractSalientTerms(responseText);
|
||||
|
||||
// Convert parsed calls to executor format
|
||||
const toolCalls: ToolCallWithExecutor[] = [];
|
||||
for (const parsedCall of parsedCalls) {
|
||||
const tool = availableTools.find((t) => t.name === parsedCall.name);
|
||||
if (tool) {
|
||||
toolCalls.push({ tool, args: parsedCall.args });
|
||||
}
|
||||
}
|
||||
|
||||
return { toolCalls, salientTerms };
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract salient terms from model response.
|
||||
* Looks for <salient_terms><term>...</term></salient_terms> format.
|
||||
*/
|
||||
private extractSalientTerms(responseText: string): string[] {
|
||||
const salientTermsMatch = responseText.match(/<salient_terms>([\s\S]*?)<\/salient_terms>/);
|
||||
if (!salientTermsMatch) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const termsBlock = salientTermsMatch[1];
|
||||
const termMatches = termsBlock.matchAll(/<term>(.*?)<\/term>/g);
|
||||
const terms: string[] = [];
|
||||
|
||||
for (const match of termMatches) {
|
||||
const term = match[1].trim();
|
||||
if (term) {
|
||||
terms.push(term);
|
||||
}
|
||||
}
|
||||
|
||||
return terms;
|
||||
}
|
||||
|
||||
/**
|
||||
* Process @commands in the user message and add corresponding tool calls.
|
||||
* Handles @vault, @websearch/@web, and @memory commands.
|
||||
*/
|
||||
private async processAtCommands(
|
||||
userMessage: string,
|
||||
existingToolCalls: ToolCallWithExecutor[],
|
||||
context: { salientTerms: string[]; timeRange?: any }
|
||||
): Promise<ToolCallWithExecutor[]> {
|
||||
const message = userMessage.toLowerCase();
|
||||
const cleanQuery = this.removeAtCommands(userMessage);
|
||||
const toolCalls = [...existingToolCalls];
|
||||
|
||||
// Handle @vault command
|
||||
if (message.includes("@vault")) {
|
||||
// Check if localSearch is already planned
|
||||
const hasLocalSearch = toolCalls.some((tc) => tc.tool.name === "localSearch");
|
||||
if (!hasLocalSearch) {
|
||||
toolCalls.push({
|
||||
tool: localSearchTool,
|
||||
args: {
|
||||
query: cleanQuery,
|
||||
salientTerms: context.salientTerms,
|
||||
timeRange: context.timeRange,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Handle @websearch command and also support @web for backward compatibility
|
||||
if (message.includes("@websearch") || message.includes("@web")) {
|
||||
const hasWebSearch = toolCalls.some((tc) => tc.tool.name === "webSearch");
|
||||
if (!hasWebSearch) {
|
||||
const memory = ProjectManager.instance.getCurrentChainManager().memoryManager.getMemory();
|
||||
const memoryVariables = await memory.loadMemoryVariables({});
|
||||
const chatHistory = extractChatHistory(memoryVariables);
|
||||
|
||||
toolCalls.push({
|
||||
tool: webSearchTool,
|
||||
args: {
|
||||
query: cleanQuery,
|
||||
chatHistory,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Handle @memory command
|
||||
if (message.includes("@memory")) {
|
||||
const hasUpdateMemory = toolCalls.some((tc) => tc.tool.name === "updateMemory");
|
||||
if (!hasUpdateMemory) {
|
||||
toolCalls.push({
|
||||
tool: updateMemoryTool,
|
||||
args: {
|
||||
statement: cleanQuery,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return toolCalls;
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove @command tokens from the user message.
|
||||
*/
|
||||
private removeAtCommands(message: string): string {
|
||||
return message
|
||||
.split(" ")
|
||||
.filter((word) => !AVAILABLE_TOOLS.includes(word.toLowerCase()))
|
||||
.join(" ")
|
||||
.trim();
|
||||
}
|
||||
|
||||
private async processImageUrls(urls: string[]): Promise<ImageProcessingResult> {
|
||||
const failedImages: string[] = [];
|
||||
const processedImages = await ImageBatchProcessor.processUrlBatch(
|
||||
|
|
@ -71,38 +328,6 @@ export class CopilotPlusChainRunner extends BaseChainRunner {
|
|||
return processedImages;
|
||||
}
|
||||
|
||||
private extractNoteContent(textContent: string): string {
|
||||
// Extract content from both <note_context> and <active_note> blocks, but not from <url_content> blocks
|
||||
const noteContextRegex = /<note_context>([\s\S]*?)<\/note_context>/g;
|
||||
const activeNoteRegex = /<active_note>([\s\S]*?)<\/active_note>/g;
|
||||
const contentRegex = /<content>([\s\S]*?)<\/content>/g;
|
||||
|
||||
let noteContent = "";
|
||||
let match;
|
||||
|
||||
// Find all note_context blocks
|
||||
while ((match = noteContextRegex.exec(textContent)) !== null) {
|
||||
const noteBlock = match[1];
|
||||
// Extract content from within this note_context block
|
||||
let contentMatch;
|
||||
while ((contentMatch = contentRegex.exec(noteBlock)) !== null) {
|
||||
noteContent += contentMatch[1] + "\n\n";
|
||||
}
|
||||
}
|
||||
|
||||
// Find all active_note blocks
|
||||
while ((match = activeNoteRegex.exec(textContent)) !== null) {
|
||||
const noteBlock = match[1];
|
||||
// Extract content from within this active_note block
|
||||
let contentMatch;
|
||||
while ((contentMatch = contentRegex.exec(noteBlock)) !== null) {
|
||||
noteContent += contentMatch[1] + "\n\n";
|
||||
}
|
||||
}
|
||||
|
||||
return noteContent.trim();
|
||||
}
|
||||
|
||||
private async extractEmbeddedImages(content: string, sourcePath?: string): Promise<string[]> {
|
||||
// Match both wiki-style ![[image.ext]] and standard markdown 
|
||||
const wikiImageRegex = /!\[\[(.*?\.(png|jpg|jpeg|gif|webp|bmp|svg))\]\]/g;
|
||||
|
|
@ -491,10 +716,10 @@ export class CopilotPlusChainRunner extends BaseChainRunner {
|
|||
}
|
||||
|
||||
try {
|
||||
logInfo("==== Step 1: Analyzing intent ====");
|
||||
let toolCalls;
|
||||
logInfo("==== Step 1: Planning tools ====");
|
||||
let toolCalls: ToolCallWithExecutor[];
|
||||
|
||||
// Extract L5 (raw user query) from envelope for intent analysis
|
||||
// Extract L5 (raw user query) from envelope for tool planning
|
||||
const envelope = userMessage.contextEnvelope;
|
||||
if (!envelope) {
|
||||
throw new Error(
|
||||
|
|
@ -505,7 +730,40 @@ export class CopilotPlusChainRunner extends BaseChainRunner {
|
|||
const messageForAnalysis = l5User?.text || userMessage.originalMessage || userMessage.message;
|
||||
|
||||
try {
|
||||
toolCalls = await IntentAnalyzer.analyzeIntent(messageForAnalysis);
|
||||
// Use model-based planning instead of Broca
|
||||
const chatModel = this.chainManager.chatModelManager.getChatModel();
|
||||
const planningResult = await this.planToolCalls(messageForAnalysis, chatModel);
|
||||
|
||||
// Execute getTimeRangeMs immediately if present (needed for localSearch timeRange)
|
||||
// We execute it once here and remove it from toolCalls to avoid double execution
|
||||
let timeRange: any = undefined;
|
||||
const timeRangeCall = planningResult.toolCalls.find(
|
||||
(tc) => tc.tool.name === "getTimeRangeMs"
|
||||
);
|
||||
if (timeRangeCall) {
|
||||
timeRange = await ToolManager.callTool(timeRangeCall.tool, timeRangeCall.args);
|
||||
logInfo("[CopilotPlus] Executed getTimeRangeMs, result:", timeRange);
|
||||
}
|
||||
|
||||
// Filter tool calls: skip getFileTree in project mode, skip getTimeRangeMs if already executed
|
||||
const filteredToolCalls = planningResult.toolCalls.filter((tc) => {
|
||||
if (tc.tool.name === "getFileTree" && isProjectMode()) {
|
||||
logInfo("Skipping getFileTree in project mode");
|
||||
return false;
|
||||
}
|
||||
if (tc.tool.name === "getTimeRangeMs" && timeRange) {
|
||||
logInfo("Skipping getTimeRangeMs - already executed during planning");
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
});
|
||||
|
||||
// Process @commands - this may add localSearch, webSearch, or updateMemory
|
||||
// Pass timeRange in context so @vault commands can use it
|
||||
toolCalls = await this.processAtCommands(messageForAnalysis, filteredToolCalls, {
|
||||
salientTerms: planningResult.salientTerms,
|
||||
timeRange,
|
||||
});
|
||||
} catch (error: any) {
|
||||
return this.handleResponse(
|
||||
getApiErrorMessage(error),
|
||||
|
|
@ -516,12 +774,8 @@ export class CopilotPlusChainRunner extends BaseChainRunner {
|
|||
);
|
||||
}
|
||||
|
||||
// Use the same removeAtCommands logic as IntentAnalyzer
|
||||
const cleanedUserMessage = userMessage.message
|
||||
.split(" ")
|
||||
.filter((word) => !AVAILABLE_TOOLS.includes(word.toLowerCase()))
|
||||
.join(" ")
|
||||
.trim();
|
||||
// Clean user message by removing @command tokens
|
||||
const cleanedUserMessage = this.removeAtCommands(userMessage.message);
|
||||
|
||||
const { toolOutputs, sources: toolSources } = await this.executeToolCalls(
|
||||
toolCalls,
|
||||
|
|
|
|||
|
|
@ -29,35 +29,36 @@ chainRunner/
|
|||
|
||||
## Tool Calling Systems Comparison
|
||||
|
||||
### 1. Legacy Tool Calling (CopilotPlusChainRunner)
|
||||
### 1. Model-Based Tool Planning (CopilotPlusChainRunner)
|
||||
|
||||
**How it works:**
|
||||
|
||||
- Uses Brevilabs API (`IntentAnalyzer.analyzeIntent()`) to analyze user intent
|
||||
- Determines which tools to call based on the analysis
|
||||
- Executes tools synchronously before sending to LLM
|
||||
- Uses chat model with tool descriptions to plan which tools to call
|
||||
- Model outputs tool calls in XML format (e.g., `<use_tool><name>...</name><args>...</args></use_tool>`)
|
||||
- Executes tools synchronously before sending to LLM for final response
|
||||
- Enhances user message with tool outputs as context
|
||||
- Supports `@` commands for explicit tool invocation (`@vault`, `@websearch`, `@memory`)
|
||||
|
||||
**Flow:**
|
||||
|
||||
```
|
||||
User Message → Intent Analysis → Tool Execution → Enhanced Prompt → LLM Response
|
||||
User Message → Model Planning → Tool Execution → Enhanced Prompt → LLM Response
|
||||
```
|
||||
|
||||
**Example:**
|
||||
|
||||
```typescript
|
||||
// 1. Analyze intent
|
||||
const toolCalls = await IntentAnalyzer.analyzeIntent(message);
|
||||
// 1. Plan tools using model
|
||||
const { toolCalls, salientTerms } = await this.planToolCalls(message, chatModel);
|
||||
|
||||
// 2. Execute tools
|
||||
// 2. Process @commands (add localSearch, webSearch, etc. if needed)
|
||||
toolCalls = await this.processAtCommands(message, toolCalls, { salientTerms });
|
||||
|
||||
// 3. Execute tools
|
||||
const toolOutputs = await this.executeToolCalls(toolCalls);
|
||||
|
||||
// 3. Enhance message with context
|
||||
const enhancedMessage = this.prepareEnhancedUserMessage(message, toolOutputs);
|
||||
|
||||
// 4. Send to LLM
|
||||
const response = await this.streamMultimodalResponse(enhancedMessage, ...);
|
||||
const response = await this.streamMultimodalResponse(message, toolOutputs, ...);
|
||||
```
|
||||
|
||||
**Tools Available:**
|
||||
|
|
|
|||
|
|
@ -1,154 +0,0 @@
|
|||
import ProjectManager from "@/LLMProviders/projectManager";
|
||||
import { isProjectMode } from "@/aiParams";
|
||||
import { createGetFileTreeTool } from "@/tools/FileTreeTools";
|
||||
import { indexTool, localSearchTool, webSearchTool } from "@/tools/SearchTools";
|
||||
import {
|
||||
convertTimeBetweenTimezonesTool,
|
||||
getCurrentTimeTool,
|
||||
getTimeInfoByEpochTool,
|
||||
getTimeRangeMsTool,
|
||||
TimeInfo,
|
||||
} from "@/tools/TimeTools";
|
||||
import { ToolManager } from "@/tools/toolManager";
|
||||
import { extractChatHistory } from "@/utils";
|
||||
import { Vault } from "obsidian";
|
||||
import { BrevilabsClient } from "./brevilabsClient";
|
||||
import { updateMemoryTool } from "@/tools/memoryTools";
|
||||
import { AVAILABLE_TOOLS } from "@/components/chat-components/constants/tools";
|
||||
|
||||
type ToolCall = {
|
||||
tool: any;
|
||||
args: any;
|
||||
};
|
||||
|
||||
export class IntentAnalyzer {
|
||||
private static tools: any[] = [];
|
||||
|
||||
static initTools(vault: Vault) {
|
||||
if (this.tools.length === 0) {
|
||||
this.tools = [
|
||||
getCurrentTimeTool,
|
||||
convertTimeBetweenTimezonesTool,
|
||||
getTimeInfoByEpochTool,
|
||||
getTimeRangeMsTool,
|
||||
localSearchTool,
|
||||
indexTool,
|
||||
webSearchTool,
|
||||
createGetFileTreeTool(vault.getRoot()),
|
||||
];
|
||||
}
|
||||
}
|
||||
|
||||
static async analyzeIntent(originalMessage: string): Promise<ToolCall[]> {
|
||||
try {
|
||||
const brocaResponse = await BrevilabsClient.getInstance().broca(
|
||||
originalMessage,
|
||||
isProjectMode()
|
||||
);
|
||||
|
||||
// Check if the response is successful and has the expected structure
|
||||
if (!brocaResponse?.response) {
|
||||
throw new Error(brocaResponse?.detail || "Broca API call failed");
|
||||
}
|
||||
|
||||
const brocaToolCalls = brocaResponse.response.tool_calls;
|
||||
const salientTerms = brocaResponse.response.salience_terms;
|
||||
|
||||
const processedToolCalls: ToolCall[] = [];
|
||||
let timeRange: { startTime: TimeInfo; endTime: TimeInfo } | undefined;
|
||||
|
||||
// Process tool calls from broca
|
||||
for (const brocaToolCall of brocaToolCalls) {
|
||||
const tool = this.tools.find((t) => t.name === brocaToolCall.tool);
|
||||
if (tool) {
|
||||
const args = brocaToolCall.args || {};
|
||||
|
||||
if (tool.name === "getTimeRangeMs") {
|
||||
timeRange = await ToolManager.callTool(tool, args);
|
||||
}
|
||||
if (tool.name == "getFileTree" && isProjectMode()) {
|
||||
// Skip file tree tool call in project mode so when user asks "what files do I have?",
|
||||
// we return files in the project context instead of the vault.
|
||||
continue;
|
||||
}
|
||||
|
||||
processedToolCalls.push({ tool, args });
|
||||
}
|
||||
}
|
||||
|
||||
// Process @ commands from original message only
|
||||
await this.processAtCommands(originalMessage, processedToolCalls, {
|
||||
timeRange,
|
||||
salientTerms,
|
||||
});
|
||||
|
||||
return processedToolCalls;
|
||||
} catch (error) {
|
||||
console.error("Error in intent analysis:", error);
|
||||
throw error; // Re-throw the error to be caught by CopilotPlusChainRunner
|
||||
}
|
||||
}
|
||||
|
||||
private static async processAtCommands(
|
||||
originalMessage: string,
|
||||
processedToolCalls: ToolCall[],
|
||||
context: {
|
||||
timeRange?: { startTime: TimeInfo; endTime: TimeInfo };
|
||||
salientTerms: string[];
|
||||
}
|
||||
): Promise<void> {
|
||||
const message = originalMessage.toLowerCase();
|
||||
const { timeRange, salientTerms } = context;
|
||||
|
||||
// Handle @vault command
|
||||
if (message.includes("@vault")) {
|
||||
// Remove all @commands from the query
|
||||
const cleanQuery = this.removeAtCommands(originalMessage);
|
||||
|
||||
processedToolCalls.push({
|
||||
tool: localSearchTool,
|
||||
args: {
|
||||
timeRange: timeRange || undefined,
|
||||
query: cleanQuery,
|
||||
salientTerms,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
// Handle @websearch command and also support @web for backward compatibility
|
||||
if (message.includes("@websearch") || message.includes("@web")) {
|
||||
const cleanQuery = this.removeAtCommands(originalMessage);
|
||||
const memory = ProjectManager.instance.getCurrentChainManager().memoryManager.getMemory();
|
||||
const memoryVariables = await memory.loadMemoryVariables({});
|
||||
const chatHistory = extractChatHistory(memoryVariables);
|
||||
|
||||
processedToolCalls.push({
|
||||
tool: webSearchTool,
|
||||
args: {
|
||||
query: cleanQuery,
|
||||
chatHistory,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
// Handle @memory command
|
||||
if (message.includes("@memory")) {
|
||||
const cleanQuery = this.removeAtCommands(originalMessage);
|
||||
|
||||
processedToolCalls.push({
|
||||
tool: updateMemoryTool,
|
||||
args: {
|
||||
statement: cleanQuery,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private static removeAtCommands(message: string): string {
|
||||
return message
|
||||
.split(" ")
|
||||
.filter((word) => !AVAILABLE_TOOLS.includes(word.toLowerCase()))
|
||||
.join(" ")
|
||||
.trim();
|
||||
}
|
||||
}
|
||||
|
|
@ -45,7 +45,6 @@ import {
|
|||
TFolder,
|
||||
WorkspaceLeaf,
|
||||
} from "obsidian";
|
||||
import { IntentAnalyzer } from "./LLMProviders/intentAnalyzer";
|
||||
import { ChatHistoryItem } from "@/components/chat-components/ChatHistoryPopover";
|
||||
import { extractChatTitle, extractChatDate } from "@/utils/chatHistoryUtils";
|
||||
import { v4 as uuidv4 } from "uuid";
|
||||
|
|
@ -135,7 +134,7 @@ export default class CopilotPlugin extends Plugin {
|
|||
}
|
||||
});
|
||||
|
||||
IntentAnalyzer.initTools(this.app.vault);
|
||||
// Tool initialization is now handled automatically in CopilotPlusChainRunner and AutonomousAgentChainRunner
|
||||
|
||||
this.registerEvent(
|
||||
this.app.workspace.on("editor-menu", (menu: Menu) => {
|
||||
|
|
|
|||
Loading…
Reference in a new issue