logancyang_obsidian-copilot/src/error.ts
Logan Yang 18f6405c76
Search v3 improvement 2 (#1733)
* feat: implement chunk-based search architecture and enhance retrieval consistency

- Introduced a new `ChunkManager` for managing note chunking, enabling efficient handling of large documents.
- Added comprehensive tests for `ChunkManager`, `SearchCore`, and `TieredLexicalRetriever` to ensure consistent chunk retrieval and scoring.
- Updated `SearchCore` to utilize chunk-based retrieval, improving performance and relevance in search results.
- Enhanced `FullTextEngine` to support chunk indexing, optimizing memory usage and search accuracy.
- Refactored scoring mechanisms to maintain chunk order and consistency across search results.
- Updated documentation to reflect changes in search architecture and chunking strategies.

feat: integrate ChunkManager into SearchCore and related components

- Added ChunkManager to SearchCore for improved chunk-based retrieval.
- Updated FullTextEngine to accept ChunkManager as an optional parameter, enhancing memory management.
- Refactored TieredLexicalRetriever to utilize the shared ChunkManager instance from SearchCore, ensuring consistent chunk handling.
- Enhanced tests for TieredLexicalRetriever to mock ChunkManager functionality, validating chunk retrieval behavior.

* feat: add enableLexicalBoosts flag to enhance search capabilities. Fix expanded term in ranking bug.

- Introduced `enableLexicalBoosts` setting in `CopilotSettings` to control the application of lexical boosts during search operations.
- Updated `SearchCore` to utilize the new setting, allowing for conditional application of folder and graph-based relevance boosts.
- Enhanced `QueryExpander` and related interfaces to include `expandedTerms`, capturing LLM-generated terms for improved recall.
- Added tests to validate the behavior of lexical boosts based on the new setting, ensuring expected functionality in various scenarios.
- Updated documentation to reflect the new feature and its implications for search performance and relevance.

* feat: enhance SearchCore to conditionally skip lexical and semantic searches based on semantic weight

- Updated SearchCore to skip lexical search when semantic weight is 100% and skip semantic search when weight is 0%, improving performance.
- Added logging to indicate when searches are skipped, enhancing traceability.
- Introduced new tests to validate the behavior of search execution under different semantic weight scenarios, ensuring expected functionality.

* feat: enhance ChunkManager and SearchCore with input validation and error handling

- Improved input validation in `ChunkManager` to ensure valid note paths and handle edge cases, including empty and excessively long inputs.
- Enhanced error handling in `SearchCore` to gracefully manage failures during chunk retrieval and fallback to simple grep results when necessary.
- Updated `QueryExpander` to utilize a race condition approach for timeout handling, improving reliability during query expansion.
- Removed deprecated tests and added new tests to validate the behavior of input validation and error handling, ensuring robustness in various scenarios.
- Updated documentation to reflect changes in input handling and error management strategies.

* fix: update semanticWeight assignment to use nullish coalescing operator

- Changed the assignment of `semanticWeight` in `TieredLexicalRetriever` to utilize the nullish coalescing operator (`??`) for better handling of undefined values, ensuring a default value of 0.6 is applied when no weight is provided.

* feat: enhance CopilotPlusChainRunner and MemoryIndexManager for improved source handling

- Added deduplication of sources in CopilotPlusChainRunner to ensure unique results.
- Refactored MemoryIndexManager to aggregate scores by chunk ID instead of note path, improving consistency in scoring.
- Introduced a new method for aggregating scores per chunk, enhancing the retrieval process.
- Updated tests to validate the new behavior in source handling and scoring aggregation.

* feat: standardize chunk ID formatting and enhance ChunkManager functionality

- Updated chunk ID format to consistently use zero-padding for indices across all components, ensuring uniformity in chunk identification.
- Refactored ChunkManager to include automatic cache validation and regeneration for chunk retrieval, improving reliability and performance.
- Introduced content hash for chunks to validate content integrity and prevent stale data usage.
- Enhanced tests to validate new chunk ID formatting and ensure consistent behavior across various scenarios.
- Updated documentation to reflect changes in chunk ID handling and retrieval processes.

* feat: streamline index clearing process in registerCommands

- Removed redundant file removal logic for clearing local Copilot semantic index.
- Introduced IndexPersistenceManager to handle index file clearing, improving code clarity and maintainability.
- Updated user notification to reflect the new index clearing process.

* feat: add comprehensive tests for FullTextEngine's clear method

- Introduced multiple test cases for the clear method in FullTextEngine to ensure safe operation under various conditions, including uninitialized index, multiple clear calls, and handling of index cleanup methods.
- Enhanced the clear method to handle scenarios where the index may throw errors during cleanup, ensuring robust state management.
- Verified memory management reset during the clear operation, improving overall reliability of the FullTextEngine.

* feat: implement TimeoutError and withTimeout utility for enhanced error handling

- Introduced a new TimeoutError class for consistent timeout error management across operations.
- Added withTimeout utility function to execute operations with a specified timeout, utilizing AbortController for proper cancellation.
- Updated QueryExpander and SearchCore to leverage withTimeout for handling LLM requests, improving reliability during long-running operations.
- Enhanced tests for withTimeout and TimeoutError to ensure correct behavior under various scenarios, including timeout handling and error propagation.
- Updated documentation to reflect the new timeout handling features and their usage.

* fix: OOM error for single file index update

- Introduced memory safety checks in IndexPersistenceManager to prevent out-of-memory (OOM) errors during record processing.
- Implemented batch processing for writing records and updating file records, improving memory efficiency.
- Added methods for estimating memory usage and checking memory safety thresholds.
- Enhanced MemoryIndexManager to utilize incremental updates for modified files, reducing memory load during reindexing.
- Updated tests to validate the new memory-safe operations and ensure correct behavior under high memory usage scenarios.

* fix: have semantic search completely independent of grep and lexical search

* refactor: update chunk ID formatting to non-padded indices

- Changed chunk ID format in ChunkManager and related components to use non-padded indices, improving consistency and simplifying ID generation.
- Updated tests to reflect the new ID format and ensure correct behavior across various scenarios.
- Removed legacy checks for old padded formats in tests, streamlining the codebase.

* feat: add SemanticSearchToggleModal for enabling/disabling semantic search

- Introduced SemanticSearchToggleModal to confirm user actions for enabling or disabling semantic search.
- Updated QASettings to utilize the new modal for handling semantic search state changes, providing user feedback and guidance on index management.

* refactor: streamline embedding model handling in BasicSettings and QASettings

- Removed the embedding model selection from BasicSettings, consolidating its functionality within QASettings.
- Introduced a new handler for setting the default embedding model in QASettings, ensuring user confirmation via RebuildIndexConfirmModal.
- Updated the UI components in QASettings to provide detailed descriptions and tooltips for embedding model selection, enhancing user experience and clarity.

* refactor: remove unused components and streamline memory management

- Deleted the SemanticReranker and VaultPathValidator classes to simplify the codebase and remove unused functionality.
- Removed the index.ts file for v3 search, consolidating exports and reducing clutter.
- Updated MemoryIndexManager to utilize a simplified in-memory approach for reindexing, enhancing performance and memory efficiency.
- Adjusted tests to reflect changes in the MemoryIndexManager's reindexing strategy, ensuring consistent behavior and improved reliability.
2025-08-19 19:00:15 -07:00

22 lines
672 B
TypeScript

export class CustomError extends Error {
public code?: string;
constructor(message: string, code?: string) {
super(message);
this.code = code;
// This is needed in TypeScript when extending built-in classes
Object.setPrototypeOf(this, CustomError.prototype);
}
}
/**
* TimeoutError class for consistent timeout error handling
*/
export class TimeoutError extends Error {
constructor(operation: string, timeoutMs: number) {
super(`${operation} timed out after ${timeoutMs}ms`);
this.name = "TimeoutError";
// This is needed in TypeScript when extending built-in classes
Object.setPrototypeOf(this, TimeoutError.prototype);
}
}