logancyang_obsidian-copilot/src/main.ts
Logan Yang df3efaecc0
Implement vault search v3 (#1703)
* Add LexicalEngine and related interfaces for enhanced search functionality

- Implement LexicalEngine class utilizing FlexSearch for efficient document indexing and searching.
- Create Hit, SearchResult, SearchOptions, and RetrieverEngine interfaces to standardize search operations.
- Update package.json and package-lock.json to include new dependencies: flexsearch and p-queue.

* Add QueryExpander class and tests for enhanced search query expansion

* Add README for Tiered Note-Level Lexical Retrieval with multilingual support and enhanced search pipeline

* Implement v3 tiered search with GrepScanner, FullTextEngine, and GraphExpander

- Add GrepScanner for fast substring search (L0)
- Implement FullTextEngine with ephemeral FlexSearch index (L1)
- Add GraphExpander for link-based candidate expansion
- Create MemoryManager with platform-aware limits
- Implement weighted RRF fusion for result combination
- Add SemanticReranker placeholder for future integration
- Include comprehensive tests for core components
- Simplify code based on review: use TextEncoder, extract methods, streamline RRF
- Add clear logging for debugging search pipeline

The architecture follows a tiered approach:
1. Grep scan for initial candidates
2. Graph expansion to increase recall
3. Full-text search on expanded set
4. Optional semantic reranking
5. RRF fusion to combine signals

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Enhance tiered retrieval recall with both rewritten queries and the expanded salient terms

- Added detailed example of end-to-end query processing in README.md
- Updated TieredRetriever to log salient terms during query expansion
- Modified FullTextEngine to index link basenames for improved searchability
- Adjusted NoteDoc interface to clarify link handling and indexing

* Refactor QueryExpander configuration and caching logic for improved clarity and performance; enhance GraphExpander documentation and testing; remove unused fields from NoteDoc interface.

* Refactor search retrieval system to wire in TieredLexicalRetriever

- Replaced HybridRetriever with TieredLexicalRetriever in ChainManager, VaultQAChainRunner, and main.ts for improved multi-stage retrieval.
- Updated README.md to reflect new integration tasks and performance benchmarks.
- Introduced TieredLexicalRetriever class to handle on-demand indexing and retrieval.
- Enhanced GrepScanner to support inclusion/exclusion patterns for file indexing.
- Modified TieredRetriever to combine expanded salient terms and improved logging for final results.
- Removed legacy index management in favor of ephemeral indexing with TieredLexicalRetriever.
- Updated interfaces and utility functions to support new retrieval logic.

* feat: Implement first working TieredLexicalRetriever and refactor search scoring

- Added comprehensive tests for TieredLexicalRetriever, covering folder boosting and result combination.
- Refactored TieredLexicalRetriever to utilize SearchCore instead of TieredRetriever.
- Enhanced FullTextEngine with improved scoring mechanisms, including field weighting and multi-field match bonuses.
- Introduced FuzzyMatcher utility for fuzzy matching capabilities, including Levenshtein distance and variant generation.
- Updated GrepScanner to prioritize path matching for faster search results.
- Implemented normalized scoring in RRF for better ranking consistency.

* feat: Update vault search result display to show snippet of content instead of full text

* feat: Enhance FullTextEngine to index frontmatter property values and improve search scoring

* feat: Improve scoring and logging

* feat: Refactor TieredLexicalRetriever to support time-based queries and improve document retrieval logic

* Remove TODO.md from tracking and add to .gitignore

- TODO.md is now a local-only development session tracker
- Prevents accidental commits of work-in-progress task lists
- Each developer can maintain their own TODO.md without conflicts

* feat: Update dependencies and enhance search functionality

- Upgraded axios to version 1.11.0 and electron to version 27.3.11 for improved performance and security.
- Refactored QueryExpander to limit queries to the original for strict fallback tests.
- Enhanced SearchCore to rank grep hits by evidence quality before fusion, improving retrieval accuracy.
- Implemented a new method in SearchCore to rank grep hits based on evidence strength.
- Updated TieredLexicalRetriever tests to ensure proper integration with mocked SearchCore.
- Improved FuzzyMatcher to include plural/singular normalization for better fuzzy matching.

* feat: Add tool call marker encoding/decoding

- Introduced new utility functions for encoding and decoding tool call markers to ensure safe embedding in HTML comments.
- Updated `updateChatMemory` to handle encoded tool call markers, preserving their integrity during memory storage.
- Enhanced logging to decode tool marker results for better readability while maintaining encoded formats for storage.
- Added comprehensive tests for tool call marker functionality to ensure correct encoding and decoding behavior.
- Refactored related components to integrate the new encoding/decoding logic seamlessly.

* feat: Enhance FullTextEngine search to support low-weight terms and improve scoring

- Updated FullTextEngine to accept low-weight terms in the search method, allowing for better handling of salient terms.
- Implemented downweighting for boolean and numeric tokens in the properties field to reduce noise in search results.
- Added a new test case to validate the downweighting behavior for boolean and numeric queries in FullTextEngine.
- Improved logging for full-text search results to provide clearer insights into the search process.

* feat: Update GraphExpander to enhance search recall with adaptive hop logic

- Implemented guardrails in GraphExpander to adjust hop depth based on the number of grep hits: allows +1 hop for small sets (<5) and limits to 1 hop for large sets (≥50).
- Updated README.md to reflect new guardrail logic and scoring normalization in search results.
- Enhanced test cases to validate the new behavior of skipping co-citations for large result sets and allowing additional hops for smaller sets.

* feat: Filter out background tools during streaming to enhance user experience

- Added logic to determine and filter out background tools, preventing their names from appearing in the tool call display during streaming.
- Implemented a mechanism to include partial tool names only if they meet a specified length threshold, improving clarity in tool call presentations.

* feat: Enhance search tools to support time-based queries and display modified time

- Updated localSearchTool to ensure a healthy cap on max source chunks for time-based queries, improving recall.
- Modified ToolResultFormatter to display actual modified time for time-filtered results, enhancing clarity in search results.
- Adjusted output formatting to differentiate between recency and relevance scores based on query type.

* feat: Introduce semantic search capabilities with Memory Index support

- Added `enableSemanticSearchV3` setting to control the new semantic search feature.
- Implemented `MemoryIndexManager` for managing in-memory vector indexing with JSONL persistence.
- Enhanced `SearchCore` to utilize semantic retrieval based on the new memory index.
- Introduced command to build the semantic memory index, improving search accuracy and retrieval efficiency.
- Updated relevant components to integrate semantic search functionality seamlessly.

* feat: Add unit tests for MemoryIndexManager to validate functionality

- Introduced comprehensive tests for the MemoryIndexManager, covering scenarios such as loading from a file, building a vector store, and indexing vault contents.
- Implemented a mock embeddings API to facilitate testing without external dependencies.
- Added a utility function to reset the MemoryIndexManager state between tests, ensuring isolation and reliability of test outcomes.

* feat: Enhance MemoryIndexManager and related components for improved semantic indexing

- Introduced incremental indexing capabilities in MemoryIndexManager to update the JSONL index with new or modified files, enhancing efficiency.
- Updated commands to utilize the new incremental indexing method, providing users with real-time updates to the semantic memory index.
- Refactored existing indexing logic to support partitioned writes, improving performance and manageability of large datasets.
- Enhanced user notifications during indexing processes to provide better feedback on progress and status.
- Added a setting to enable semantic search, allowing users to blend semantic similarity into search results seamlessly.

* feat: Refine scoring display and enhance search result handling

- Updated SourcesModal to display relevance scores with four decimal places for consistency with SearchCore logs.
- Enhanced TieredLexicalRetriever to include a new `rerank_score` field for better score management and consistency across search results.
- Modified the combineResults method to ensure that title matches retain their original score while incorporating a fused score as `rerank_score`.
- Adjusted formatting in SearchTools to ensure both `score` and `rerank_score` reflect the same final score when present.
- Updated ToolResultFormatter to display scores with four decimal places, improving clarity in search result presentations.

* refactor: Simplify relevant notes retrieval by removing VectorStoreManager dependency

- Removed the use of VectorStoreManager in the relevant notes fetching logic, streamlining the process.
- Integrated MemoryIndexManager for improved handling of in-memory indexing and note retrieval.
- Enhanced error handling to ensure relevant notes are only fetched when the embedding index is available.
- Updated related functions to utilize the new memory index approach, improving performance and reliability.

* refactor: Remove VectorStoreManager dependency and streamline indexing logic

- Eliminated the VectorStoreManager from various components, transitioning to MemoryIndexManager for indexing and retrieval.
- Updated project and chain managers to remove unnecessary dependencies, enhancing code clarity and maintainability.
- Improved error handling and indexing conditions, particularly for mobile users, ensuring a more robust initialization process.
- Refactored settings components to utilize the new memory indexing approach, simplifying the user experience and reducing legacy code.

* chore: Mark legacy components as deprecated in preparation for v3 transition

- Annotated various files including DebugSearchModal, OramaSearchModal, chunkedStorage, dbOperations, hybridRetriever, and vectorStoreManager as deprecated, indicating they are obsolete in v3.
- Updated README.md to reflect the current implementation status and migration notes, emphasizing the removal of Orama-based modules and the transition to MemoryIndexManager for indexing and retrieval.

* feat: Implement file tracking and reindexing for modified files

- Introduced a new FileTrackingState interface to manage the last active file and its modification time.
- Enhanced the CopilotPlugin to opportunistically reindex the previous active file if it was modified while active, contingent on semantic search settings.
- Updated MemoryIndexManager to support reindexing of single modified files, improving efficiency in handling changes.
- Added unit tests for reindexSingleFileIfModified to ensure correct functionality and performance.

* feat: Add graph hops setting for enhanced search result expansion

- Introduced a new `graphHops` setting in `CopilotSettings` to control the number of hops for graph expansion during search, with a default value of 1 and a range of 1-3.
- Updated the `sanitizeSettings` function to validate the `graphHops` value.
- Enhanced the `SearchCore` and `TieredLexicalRetriever` classes to utilize the `graphHops` setting for improved search result relevance.
- Updated documentation in `README.md` to reflect the new feature and its security optimizations.

* refactor: Improve error handling and indexing logic in MemoryIndexManager and related components

- Replaced console error logging with structured logging using logError and logWarn for better error tracking.
- Enhanced the refresh and reindexing functions to ensure proper user notifications and error handling.
- Updated the logic for managing indexed files, including handling exclusions and ensuring accurate reporting of indexed and unindexed files.
- Implemented rate limiting in the MemoryIndexManager to optimize embedding requests and prevent overloading the service.
- Refactored chunk processing to improve efficiency and clarity in the indexing workflow.

* chore: Update README.md to reflect final session completion and key fixes

- Expanded the final session summary with a date and detailed list of completed features and fixes, including UI enhancements and improved logging practices.
- Clarified the status of deferred features and provided migration notes for better user guidance.
- Ensured documentation aligns with the latest implementation changes and optimizations.

* chore: Update memory limits in README.md and MemoryManager.ts for improved performance

- Adjusted memory limits for mobile and desktop platforms in both README.md and MemoryManager.ts to reflect increased capacity (20MB mobile, 100MB desktop).
- Ensured documentation aligns with the latest implementation changes for better clarity on resource management.

* feat: Integrate HyDE document generation into SearchCore for enhanced semantic search

- Implemented a new method to generate hypothetical documents (HyDE) to improve semantic search capabilities.
- Added timeout handling for HyDE generation to ensure graceful error management.
- Updated SearchCore to utilize HyDE documents in search queries, enhancing the relevance of results when semantic search is enabled.
- Introduced unit tests to validate the integration and functionality of HyDE generation within the search process.

* refactor: Enhance MemoryIndexManager with public methods for better indexing management

- Introduced public methods in MemoryIndexManager to streamline access to indexed file paths, check if a file is indexed, retrieve embeddings, and clear the index.
- Updated related components to utilize these new methods, improving code clarity and reducing direct access to internal properties.
- Enhanced error handling and user notifications during memory index operations.

* feat: Implement indexing notification and progress management

- Introduced the IndexingNotificationManager to handle UI notifications during indexing operations, allowing users to pause or stop the process.
- Added the IndexingProgressTracker to track progress across multiple files, enhancing user feedback during indexing.
- Developed the IndexingPipeline to manage the chunking and processing of files into embeddings, improving the overall indexing workflow.
- Created the IndexPersistenceManager for managing the persistence of indexed data, ensuring efficient storage and retrieval of index records.
- Refactored MemoryIndexManager to utilize the new components, streamlining the indexing process and improving code organization.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-08-11 16:58:04 -07:00

492 lines
17 KiB
TypeScript

import { BrevilabsClient } from "@/LLMProviders/brevilabsClient";
import ProjectManager from "@/LLMProviders/projectManager";
import { CustomModel, getCurrentProject } from "@/aiParams";
import { AutocompleteService } from "@/autocomplete/autocompleteService";
import { parseChatContent } from "@/chatUtils";
import { registerCommands } from "@/commands";
import CopilotView from "@/components/CopilotView";
import { APPLY_VIEW_TYPE, ApplyView } from "@/components/composer/ApplyView";
import { LoadChatHistoryModal } from "@/components/modals/LoadChatHistoryModal";
import { QUICK_COMMAND_CODE_BLOCK } from "@/commands/constants";
import { registerContextMenu } from "@/commands/contextMenu";
import { CustomCommandRegister } from "@/commands/customCommandRegister";
import { migrateCommands, suggestDefaultCommands } from "@/commands/migrator";
import { createQuickCommandContainer } from "@/components/QuickCommand";
import {
ABORT_REASON,
CHAT_VIEWTYPE,
DEFAULT_OPEN_AREA,
EVENT_NAMES,
VAULT_VECTOR_STORE_STRATEGY,
} from "@/constants";
import { ChatManager } from "@/core/ChatManager";
import { MessageRepository } from "@/core/MessageRepository";
import { encryptAllKeys } from "@/encryptionService";
import { logInfo, logWarn } from "@/logger";
import { checkIsPlusUser } from "@/plusUtils";
import { MemoryIndexManager } from "@/search/v3/MemoryIndexManager";
import { TieredLexicalRetriever } from "@/search/v3/TieredLexicalRetriever";
import { CopilotSettingTab } from "@/settings/SettingsPage";
import {
getModelKeyFromModel,
getSettings,
sanitizeSettings,
setSettings,
subscribeToSettingsChange,
} from "@/settings/model";
import { ChatUIState } from "@/state/ChatUIState";
import { FileParserManager } from "@/tools/FileParserManager";
import { initializeBuiltinTools } from "@/tools/builtinTools";
import {
Editor,
MarkdownView,
Menu,
Notice,
Plugin,
TFile,
TFolder,
WorkspaceLeaf,
} from "obsidian";
import { IntentAnalyzer } from "./LLMProviders/intentAnalyzer";
interface FileTrackingState {
lastActiveFile: TFile | null;
lastActiveMtime: number | null;
}
export default class CopilotPlugin extends Plugin {
// Plugin components
projectManager: ProjectManager;
brevilabsClient: BrevilabsClient;
userMessageHistory: string[] = [];
fileParserManager: FileParserManager;
customCommandRegister: CustomCommandRegister;
settingsUnsubscriber?: () => void;
private autocompleteService: AutocompleteService;
chatUIState: ChatUIState;
private fileTracker: FileTrackingState = { lastActiveFile: null, lastActiveMtime: null };
async onload(): Promise<void> {
await this.loadSettings();
this.settingsUnsubscriber = subscribeToSettingsChange(async (prev, next) => {
if (next.enableEncryption) {
await this.saveData(await encryptAllKeys(next));
} else {
await this.saveData(next);
}
registerCommands(this, prev, next);
});
this.addSettingTab(new CopilotSettingTab(this.app, this));
// Core plugin initialization
// Initialize built-in tools with vault access
initializeBuiltinTools(this.app.vault);
// Initialize BrevilabsClient
this.brevilabsClient = BrevilabsClient.getInstance();
this.brevilabsClient.setPluginVersion(this.manifest.version);
checkIsPlusUser();
// Initialize ProjectManager
this.projectManager = ProjectManager.getInstance(this.app, this);
// Initialize FileParserManager early with other core services
this.fileParserManager = new FileParserManager(this.brevilabsClient, this.app.vault);
// Initialize ChatUIState with new architecture
const messageRepo = new MessageRepository();
const chainManager = this.projectManager.getCurrentChainManager();
const chatManager = new ChatManager(messageRepo, chainManager, this.fileParserManager, this);
this.chatUIState = new ChatUIState(chatManager);
this.registerView(CHAT_VIEWTYPE, (leaf: WorkspaceLeaf) => new CopilotView(leaf, this));
this.registerView(APPLY_VIEW_TYPE, (leaf: WorkspaceLeaf) => new ApplyView(leaf));
this.initActiveLeafChangeHandler();
this.addRibbonIcon("message-square", "Open Copilot Chat", (evt: MouseEvent) => {
this.activateView();
});
registerCommands(this, undefined, getSettings());
this.registerMarkdownCodeBlockProcessor(QUICK_COMMAND_CODE_BLOCK, (_, el) => {
createQuickCommandContainer({
plugin: this,
element: el,
});
// Remove parent element class names to clear default code block styling
if (el.parentElement) {
el.parentElement.className = "";
}
});
IntentAnalyzer.initTools(this.app.vault);
// Auto-index per strategy when semantic toggle is enabled
try {
const settings = getSettings();
const semanticOn = settings.enableSemanticSearchV3;
if (semanticOn) {
const strategy = settings.indexVaultToVectorStore;
const isMobileDisabled = settings.disableIndexOnMobile && (this.app as any).isMobile;
if (!isMobileDisabled && strategy === VAULT_VECTOR_STORE_STRATEGY.ON_STARTUP) {
await MemoryIndexManager.getInstance(this.app).indexVaultIncremental();
await MemoryIndexManager.getInstance(this.app).ensureLoaded();
} else {
const loaded = isMobileDisabled
? false
: await MemoryIndexManager.getInstance(this.app).loadIfExists();
if (!loaded) {
logWarn("MemoryIndex: embedding index not found; falling back to full-text only");
new Notice("embedding index doesn't exist, fall back to full-text search");
}
}
} else {
// If semantic is off, we still try to load index for features that depend on it
if (!(settings.disableIndexOnMobile && (this.app as any).isMobile)) {
await MemoryIndexManager.getInstance(this.app).loadIfExists();
}
}
} catch {
// Swallow errors to avoid disrupting startup
}
this.registerEvent(
this.app.workspace.on("editor-menu", (menu: Menu) => {
return registerContextMenu(menu);
})
);
this.registerEvent(
this.app.workspace.on("active-leaf-change", (leaf) => {
if (leaf && leaf.view instanceof MarkdownView) {
const file = leaf.view.file;
if (file) {
// On switching to a new file, opportunistically re-index the previous active file
// if semantic search v3 is enabled and file was modified while active
try {
const settings = getSettings();
if (settings.enableSemanticSearchV3) {
const { lastActiveFile, lastActiveMtime } = this.fileTracker;
if (
lastActiveFile &&
typeof lastActiveMtime === "number" &&
lastActiveFile.extension === "md"
) {
if (lastActiveFile.stat?.mtime && lastActiveFile.stat.mtime > lastActiveMtime) {
// Reindex only the last active file that changed
void MemoryIndexManager.getInstance(this.app).reindexSingleFileIfModified(
lastActiveFile,
lastActiveMtime
);
}
}
// update trackers
this.fileTracker = {
lastActiveFile: file,
lastActiveMtime: file.stat?.mtime ?? null,
};
}
} catch {
// non-fatal: ignore indexing errors during active file switch
}
const activeCopilotView = this.app.workspace
.getLeavesOfType(CHAT_VIEWTYPE)
.find((leaf) => leaf.view instanceof CopilotView)?.view as CopilotView;
if (activeCopilotView) {
const event = new CustomEvent(EVENT_NAMES.ACTIVE_LEAF_CHANGE);
activeCopilotView.eventTarget.dispatchEvent(event);
}
}
}
})
);
// Initialize autocomplete service
this.autocompleteService = AutocompleteService.getInstance(this);
this.customCommandRegister = new CustomCommandRegister(this, this.app.vault);
this.app.workspace.onLayoutReady(() => {
this.customCommandRegister.initialize().then(migrateCommands).then(suggestDefaultCommands);
});
}
async onunload() {
if (this.projectManager) {
this.projectManager.onunload();
}
this.customCommandRegister.cleanup();
this.settingsUnsubscriber?.();
this.autocompleteService?.destroy();
logInfo("Copilot plugin unloaded");
}
updateUserMessageHistory(newMessage: string) {
this.userMessageHistory = [...this.userMessageHistory, newMessage];
}
async autosaveCurrentChat() {
if (getSettings().autosaveChat) {
const chatView = this.app.workspace.getLeavesOfType(CHAT_VIEWTYPE)[0]?.view as CopilotView;
if (chatView) {
await chatView.saveChat();
}
}
}
async processText(
editor: Editor,
eventType: string,
eventSubtype?: string,
checkSelectedText = true
) {
const selectedText = await editor.getSelection();
const isChatWindowActive = this.app.workspace.getLeavesOfType(CHAT_VIEWTYPE).length > 0;
if (!isChatWindowActive) {
await this.activateView();
}
// Without the timeout, the view is not yet active
setTimeout(() => {
const activeCopilotView = this.app.workspace
.getLeavesOfType(CHAT_VIEWTYPE)
.find((leaf) => leaf.view instanceof CopilotView)?.view as CopilotView;
if (activeCopilotView && (!checkSelectedText || selectedText)) {
const event = new CustomEvent(eventType, { detail: { selectedText, eventSubtype } });
activeCopilotView.eventTarget.dispatchEvent(event);
}
}, 0);
}
processSelection(editor: Editor, eventType: string, eventSubtype?: string) {
this.processText(editor, eventType, eventSubtype);
}
emitChatIsVisible() {
const activeCopilotView = this.app.workspace
.getLeavesOfType(CHAT_VIEWTYPE)
.find((leaf) => leaf.view instanceof CopilotView)?.view as CopilotView;
if (activeCopilotView) {
const event = new CustomEvent(EVENT_NAMES.CHAT_IS_VISIBLE);
activeCopilotView.eventTarget.dispatchEvent(event);
}
}
initActiveLeafChangeHandler() {
this.registerEvent(
this.app.workspace.on("active-leaf-change", (leaf) => {
if (!leaf) {
return;
}
if (leaf.getViewState().type === CHAT_VIEWTYPE) {
this.emitChatIsVisible();
}
})
);
}
private getCurrentEditorOrDummy(): Editor {
const activeView = this.app.workspace.getActiveViewOfType(MarkdownView);
return {
getSelection: () => {
const selection = activeView?.editor?.getSelection();
if (selection) return selection;
// Default to the entire active file if no selection
const activeFile = this.app.workspace.getActiveFile();
return activeFile ? this.app.vault.cachedRead(activeFile) : "";
},
replaceSelection: activeView?.editor?.replaceSelection.bind(activeView.editor) || (() => {}),
} as Partial<Editor> as Editor;
}
processCustomPrompt(eventType: string, customPrompt: string) {
const editor = this.getCurrentEditorOrDummy();
this.processText(editor, eventType, customPrompt, false);
}
toggleView() {
const leaves = this.app.workspace.getLeavesOfType(CHAT_VIEWTYPE);
if (leaves.length > 0) {
this.deactivateView();
} else {
this.activateView();
}
}
async activateView(): Promise<void> {
const leaves = this.app.workspace.getLeavesOfType(CHAT_VIEWTYPE);
if (leaves.length === 0) {
if (getSettings().defaultOpenArea === DEFAULT_OPEN_AREA.VIEW) {
await this.app.workspace.getRightLeaf(false).setViewState({
type: CHAT_VIEWTYPE,
active: true,
});
} else {
await this.app.workspace.getLeaf(true).setViewState({
type: CHAT_VIEWTYPE,
active: true,
});
}
} else {
this.app.workspace.revealLeaf(leaves[0]);
}
this.emitChatIsVisible();
}
async deactivateView() {
this.app.workspace.detachLeavesOfType(CHAT_VIEWTYPE);
}
async loadSettings() {
const savedSettings = await this.loadData();
const sanitizedSettings = sanitizeSettings(savedSettings);
setSettings(sanitizedSettings);
}
mergeActiveModels(
existingActiveModels: CustomModel[],
builtInModels: CustomModel[]
): CustomModel[] {
const modelMap = new Map<string, CustomModel>();
// Create a unique key for each model, it's model (name + provider)
// Add or update existing models in the map
existingActiveModels.forEach((model) => {
const key = getModelKeyFromModel(model);
const existingModel = modelMap.get(key);
if (existingModel) {
// If it's a built-in model, preserve the built-in status
modelMap.set(key, {
...model,
isBuiltIn: existingModel.isBuiltIn || model.isBuiltIn,
});
} else {
modelMap.set(key, model);
}
});
return Array.from(modelMap.values());
}
async loadCopilotChatHistory() {
const chatFiles = await this.getChatHistoryFiles();
if (chatFiles.length === 0) {
new Notice("No chat history found.");
return;
}
new LoadChatHistoryModal(this.app, chatFiles, this.loadChatHistory.bind(this)).open();
}
async getChatHistoryFiles(): Promise<TFile[]> {
const folder = this.app.vault.getAbstractFileByPath(getSettings().defaultSaveFolder);
if (!(folder instanceof TFolder)) {
return [];
}
const files = await this.app.vault.getMarkdownFiles();
const folderFiles = files.filter((file) => file.path.startsWith(folder.path));
// Get current project ID if in a project
const currentProject = getCurrentProject();
const currentProjectId = currentProject?.id;
if (currentProjectId) {
// In project mode: return only files with this project's ID prefix
const projectPrefix = `${currentProjectId}__`;
return folderFiles.filter((file) => file.basename.startsWith(projectPrefix));
} else {
// In non-project mode: return only files without any project ID prefix
// This assumes project IDs always use the format projectId__ as prefix
return folderFiles.filter((file) => {
// Check if the filename has any projectId__ prefix pattern
return !file.basename.match(/^[a-zA-Z0-9-]+__/);
});
}
}
async loadChatHistory(file: TFile) {
// First autosave the current chat if the setting is enabled
await this.autosaveCurrentChat();
const content = await this.app.vault.read(file);
const messages = parseChatContent(content);
// Check if the Copilot view is already active
const existingView = this.app.workspace.getLeavesOfType(CHAT_VIEWTYPE)[0];
if (!existingView) {
// Only activate the view if it's not already open
this.activateView();
}
// Load messages into ChatUIState (which now handles memory updates)
await this.chatUIState.loadMessages(messages);
// Update the view
const copilotView = (existingView || this.app.workspace.getLeavesOfType(CHAT_VIEWTYPE)[0])
?.view as CopilotView;
if (copilotView) {
copilotView.updateView();
}
}
async handleNewChat() {
// First autosave the current chat if the setting is enabled
await this.autosaveCurrentChat();
// Abort any ongoing streams before clearing chat
const existingView = this.app.workspace.getLeavesOfType(CHAT_VIEWTYPE)[0];
if (existingView) {
const copilotView = existingView.view as CopilotView;
// Dispatch abort event to stop any ongoing streams
const abortEvent = new CustomEvent(EVENT_NAMES.ABORT_STREAM, {
detail: { reason: ABORT_REASON.NEW_CHAT },
});
copilotView.eventTarget.dispatchEvent(abortEvent);
}
// Clear messages through ChatUIState (which also clears chain memory)
this.chatUIState.clearMessages();
// Update view if it exists
if (existingView) {
const copilotView = existingView.view as CopilotView;
copilotView.updateView();
} else {
// If view doesn't exist, open it
await this.activateView();
}
// Note: UI-specific state like includeActiveNote setting is handled in the Chat component
// This ensures proper separation of concerns between plugin logic and UI state
}
async newChat() {
// Just delegate to the shared method
await this.handleNewChat();
}
async customSearchDB(query: string, salientTerms: string[], textWeight: number): Promise<any[]> {
const retriever = new TieredLexicalRetriever(app, {
minSimilarityScore: 0.3,
maxK: 20,
salientTerms: salientTerms,
textWeight: textWeight,
});
const results = await retriever.getRelevantDocuments(query);
return results.map((doc) => ({
content: doc.pageContent,
metadata: doc.metadata,
}));
}
}