logancyang_obsidian-copilot/designdocs/TOOLS.md

Tool System Documentation

Overview

The Copilot tool system uses a centralized registry pattern that makes it easy to add new tools, including future MCP (Model Context Protocol) tools. All tools are managed through a singleton ToolRegistry that provides a unified interface for tool discovery, configuration, and execution.

Tool Prompt Architecture

How Tool Instructions Flow to the LLM

The system uses native tool calling via LangChain's bindTools() for tool invocation. Tool results are formatted as context in a layered approach:

1. Tool Schema Descriptions (Zod schemas in tool implementations)

   • Defines parameter formats, rules, and validation
   • Provided to LLM via bindTools() for native tool calling

2. Custom Prompt Instructions (in builtinTools.ts)

   • Behavioral guidance for when and how to use tools
   • Special requirements (e.g., "always provide salientTerms")

3. Model-Specific Adaptations (in modelAdapter.ts)

   • Last resort for model-specific quirks

Layered Prompt Integration

• ContextManager promotes user-attached artifacts from earlier turns into L2 (Context Library), so every chain runner starts with the same cacheable system prefix.
• When a tool executes during the current turn, its result payload is prepended to the user message (L3 + L5) using renderCiCMessage(...). Nothing is injected into the system message, keeping L1/L2 stable.
• LayerToMessagesConverter.convert(envelope, { includeSystemMessage: true, mergeUserContent: true }) materializes the base messages; runners then append tool results before sending to the model.
• promptPayloadRecorder inspects the final payload and highlights tool blocks in its layered view, making it easy to debug the L1-L5 structure.

Why Two Layers: Schema vs Custom Instructions

Key Difference: Schema descriptions document parameters via Zod, while custom instructions provide behavioral guidance.

1. Clear Separation

   • Schema: Parameter documentation (types, formats, rules) via Zod - used by bindTools()
   • Custom Instructions: Behavioral guidance on when and how to use the tool

2. MCP Compatibility

   • External tools provide immutable schemas
   • We add custom instructions without modifying their code

3. Example

   // Schema (provided to LLM via bindTools())
   const searchSchema = z.object({
     query: z.string().min(1).describe("The search query"),
     salientTerms: z.array(z.string()).describe("Keywords to find in notes"),
   });
   
   // Custom Instructions (behavioral guidance)
   customPromptInstructions: `
   When searching notes:
   - Always provide salientTerms extracted from the user's query
   - Use getTimeRangeMs first for time-based queries
   - Examine relevance scores before using results
   `;
   

Best Practices

1. Schema Descriptions: Parameter Documentation Only

   • Document parameter types, formats, and validation rules via Zod
   • Schemas are automatically provided to LLM via bindTools()
   • Focus on the data contract

2. Custom Instructions: Behavioral Guidance

   • Explain when to use this tool vs alternatives
   • Show common usage patterns and requirements
   • Include tips for better results (e.g., "use getTimeRangeMs before localSearch")

3. Model Adapters: Model-Specific Fixes

   • Only for persistent model-specific failures
   • Keep minimal and targeted

localSearch CiC Prompting Flow

• CiC: Corpus in Context https://arxiv.org/pdf/2406.13121
• Instruction First: CopilotPlusChainRunner now assembles the localSearch payload via buildLocalSearchInnerContent, ensuring citation guidance (e.g., <guidance> rules) tops the XML block before any documents.
• Documents Next: Search hits are serialized once through formatSearchResultsForLLM; the helper simply appends them after guidance, keeping the documents section untouched but clearly separated.
• Question Last: renderCiCMessage formats the final prompt so any context precedes the user's original query; this matches the CiC recommendation for instruction → context → query ordering.
  • CopilotPlus: Uses LayerToMessagesConverter which adds [User query]: label when merging L3+L5 content from envelope
  • AutonomousAgent: Uses ensureCiCOrderingWithQuestion which adds [User query]: label to clearly separate tool results from original query in iterative loop
  • Consistency: Both chains use the same [User query]: label format for uniform prompting across the codebase
• Reusable Wrapping: wrapLocalSearchPayload centralizes the <localSearch> tag creation (including optional timeRange), making the layout reusable for future chains without copying string glue.

Current Implementation

Core Files

• src/tools/ToolRegistry.ts - Central registry for all tools
• src/tools/builtinTools.ts - Built-in tool definitions and initialization
• src/LLMProviders/chainRunner/AutonomousAgentChainRunner.ts - Tool execution in the agent
• src/settings/v2/components/ToolSettingsSection.tsx - Settings UI for tool configuration

Tool Registry Pattern

The ToolRegistry is a singleton that manages all tools:

class ToolRegistry {
  static getInstance(): ToolRegistry;
  register(definition: ToolDefinition): void;
  registerAll(definitions: ToolDefinition[]): void;
  getAllTools(): ToolDefinition[];
  getEnabledTools(enabledToolIds: Set<string>, vaultAvailable: boolean): SimpleTool<any, any>[];
  getToolsByCategory(): Map<string, ToolDefinition[]>;
  getConfigurableTools(): ToolDefinition[];
  getToolMetadata(id: string): ToolMetadata | undefined;
  clear(): void;
}

Tool Definition Structure

interface ToolDefinition {
  tool: SimpleTool<any, any>; // The actual tool implementation
  metadata: ToolMetadata; // UI and configuration metadata
}

interface ToolMetadata {
  id: string; // Unique identifier
  displayName: string; // Shown in UI
  description: string; // Help text
  category: "search" | "time" | "file" | "media" | "mcp" | "custom";
  isAlwaysEnabled?: boolean; // If true, not configurable (e.g., time tools)
  requiresVault?: boolean; // Needs vault access
  customPromptInstructions?: string; // Tool-specific prompts
}

Adding a New Built-in Tool

1. Implement the Tool

Create your tool following the SimpleTool interface:

// Example: New built-in tool
import { z } from "zod";
import { SimpleTool } from "./SimpleTool";

export const myNewTool: SimpleTool<{ input: string }, { result: string }> = {
  name: "myNewTool",
  description: "Description for the LLM to understand when to use this tool",
  schema: z.object({
    input: z.string().describe("The input parameter description"),
  }),
  func: async (params) => {
    // Tool implementation
    const result = await performOperation(params.input);
    return { result };
  },
};

2. Add to Built-in Tools

Update src/tools/builtinTools.ts:

export const BUILTIN_TOOLS: ToolDefinition[] = [
  // ... existing tools ...
  {
    tool: myNewTool,
    metadata: {
      id: "myNewTool",
      displayName: "My New Tool",
      description: "User-friendly description for settings UI",
      category: "custom", // Choose appropriate category
      // Optional flags:
      isAlwaysEnabled: false, // Set true if tool should always be available
      requiresVault: true, // Set true if tool needs vault access
      customPromptInstructions: "Special instructions for the AI when using this tool",
    },
  },
];

3. Update Default Settings (if configurable)

If the tool is configurable (not always-enabled), add its ID to the default enabled tools in src/constants.ts:

autonomousAgentEnabledToolIds: [
  "localSearch",
  "webSearch",
  "pomodoro",
  "youtubeTranscription",
  "writeToFile",
  "myNewTool"  // Add your tool ID here
],

Adding MCP Tools (Future Implementation)

1. MCP Tool Wrapper

Create a wrapper to convert MCP tools to the SimpleTool interface:

function createMcpToolWrapper(serverName: string, mcpTool: McpTool): SimpleTool<any, any> {
  return {
    name: `${serverName}_${mcpTool.name}`,
    description: mcpTool.description || `MCP tool from ${serverName}`,
    schema: convertMcpSchemaToZod(mcpTool.inputSchema),
    func: async (params) => {
      // Call the MCP server
      const result = await mcpHub.callTool(serverName, mcpTool.name, params);

      // Convert MCP response to expected format
      return {
        result: formatMcpResponse(result),
      };
    },
  };
}

2. Dynamic MCP Tool Registration

Register MCP tools when servers connect:

// In your MCP initialization code
export async function registerMcpServerTools(serverName: string, mcpTools: McpTool[]) {
  const registry = ToolRegistry.getInstance();

  for (const mcpTool of mcpTools) {
    registry.register({
      tool: createMcpToolWrapper(serverName, mcpTool),
      metadata: {
        id: `mcp_${serverName}_${mcpTool.name}`,
        displayName: mcpTool.displayName || mcpTool.name,
        description: mcpTool.description || `MCP tool from ${serverName}`,
        category: "mcp",
        // MCP tools are user-configurable by default
        isAlwaysEnabled: false,
        // Add any MCP-specific prompt instructions
        customPromptInstructions: mcpTool.systemPrompt,
      },
    });
  }
}

// When MCP server disconnects
export function unregisterMcpServerTools(serverName: string) {
  const registry = ToolRegistry.getInstance();
  const allTools = registry.getAllTools();

  // Remove tools from this server
  const toolsToKeep = allTools.filter((t) => !t.metadata.id.startsWith(`mcp_${serverName}_`));

  registry.clear();
  registry.registerAll(toolsToKeep);

  // Re-initialize built-in tools
  initializeBuiltinTools(app.vault);
}

3. Schema Conversion Helper

Convert MCP JSON Schema to Zod schema:

function convertMcpSchemaToZod(jsonSchema: any): z.ZodSchema {
  // Basic implementation - extend as needed
  if (jsonSchema.type === "object") {
    const shape: any = {};

    for (const [key, prop] of Object.entries(jsonSchema.properties || {})) {
      const propSchema = prop as any;

      if (propSchema.type === "string") {
        shape[key] = z.string();
        if (propSchema.description) {
          shape[key] = shape[key].describe(propSchema.description);
        }
      } else if (propSchema.type === "number") {
        shape[key] = z.number();
      } else if (propSchema.type === "boolean") {
        shape[key] = z.boolean();
      } else if (propSchema.type === "array") {
        shape[key] = z.array(z.any()); // Simplification
      } else if (propSchema.type === "object") {
        shape[key] = z.object({});
      }

      // Handle optional properties
      if (!jsonSchema.required?.includes(key)) {
        shape[key] = shape[key].optional();
      }
    }

    return z.object(shape);
  }

  // Fallback for other types
  return z.any();
}

4. Settings Storage for MCP Tools

MCP tool preferences are stored in the same array as built-in tools:

// When enabling/disabling MCP tools:
function updateMcpToolSetting(toolId: string, enabled: boolean) {
  const settings = getSettings();
  const enabledIds = new Set(settings.autonomousAgentEnabledToolIds || []);

  if (enabled) {
    enabledIds.add(toolId);
  } else {
    enabledIds.delete(toolId);
  }

  updateSetting("autonomousAgentEnabledToolIds", Array.from(enabledIds));
}

How the System Works

Tool Discovery Flow

1. Initialization: initializeBuiltinTools() registers all built-in tools
2. MCP Connection: When MCP servers connect, their tools are dynamically registered
3. Settings UI: ToolSettingsSection component reads from the registry to generate UI
4. Tool Execution: AutonomousAgentChainRunner.getAvailableTools() filters tools based on settings

Tool Call Rendering Roots

React invariant #409 surfaced when tool-call banners attempted to render into React roots that had already been unmounted. To prevent this regression, the plugin routes all banner rendering through the shared manager in src/components/chat-components/toolCallRootManager.tsx.

Manager Responsibilities

• Tracks { root, isUnmounting } per message/tool call via window.__copilotToolCallRoots.
• ensureToolCallRoot finalises pending disposals and creates a new createRoot when needed.
• renderToolCallBanner renders <ToolCallBanner /> into the managed root; components never call root.render directly.
• removeToolCallRoot and cleanupMessageToolCallRoots schedule unmounts on the next tick and drop entries only after disposal completes.
• cleanupStaleToolCallRoots purges message IDs older than one hour to avoid leaking historical roots.

Integration Notes

ChatSingleMessage keeps const rootsRef = useRef(getMessageToolCallRoots(messageId)), which provides a stable registry for each message. The component delegates all lifecycle calls to the manager and snapshots rootsRef.current inside effect cleanup to satisfy react-hooks/exhaustive-deps.

Verification

Run the focused test to cover the streaming behaviour and tool-call integration:

npm test -- src/components/chat-components/ChatSingleMessage.test.tsx