mirror of
https://github.com/logancyang/obsidian-copilot.git
synced 2026-07-22 07:50:24 +00:00
df3efaecc0
* Add LexicalEngine and related interfaces for enhanced search functionality - Implement LexicalEngine class utilizing FlexSearch for efficient document indexing and searching. - Create Hit, SearchResult, SearchOptions, and RetrieverEngine interfaces to standardize search operations. - Update package.json and package-lock.json to include new dependencies: flexsearch and p-queue. * Add QueryExpander class and tests for enhanced search query expansion * Add README for Tiered Note-Level Lexical Retrieval with multilingual support and enhanced search pipeline * Implement v3 tiered search with GrepScanner, FullTextEngine, and GraphExpander - Add GrepScanner for fast substring search (L0) - Implement FullTextEngine with ephemeral FlexSearch index (L1) - Add GraphExpander for link-based candidate expansion - Create MemoryManager with platform-aware limits - Implement weighted RRF fusion for result combination - Add SemanticReranker placeholder for future integration - Include comprehensive tests for core components - Simplify code based on review: use TextEncoder, extract methods, streamline RRF - Add clear logging for debugging search pipeline The architecture follows a tiered approach: 1. Grep scan for initial candidates 2. Graph expansion to increase recall 3. Full-text search on expanded set 4. Optional semantic reranking 5. RRF fusion to combine signals 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Enhance tiered retrieval recall with both rewritten queries and the expanded salient terms - Added detailed example of end-to-end query processing in README.md - Updated TieredRetriever to log salient terms during query expansion - Modified FullTextEngine to index link basenames for improved searchability - Adjusted NoteDoc interface to clarify link handling and indexing * Refactor QueryExpander configuration and caching logic for improved clarity and performance; enhance GraphExpander documentation and testing; remove unused fields from NoteDoc interface. * Refactor search retrieval system to wire in TieredLexicalRetriever - Replaced HybridRetriever with TieredLexicalRetriever in ChainManager, VaultQAChainRunner, and main.ts for improved multi-stage retrieval. - Updated README.md to reflect new integration tasks and performance benchmarks. - Introduced TieredLexicalRetriever class to handle on-demand indexing and retrieval. - Enhanced GrepScanner to support inclusion/exclusion patterns for file indexing. - Modified TieredRetriever to combine expanded salient terms and improved logging for final results. - Removed legacy index management in favor of ephemeral indexing with TieredLexicalRetriever. - Updated interfaces and utility functions to support new retrieval logic. * feat: Implement first working TieredLexicalRetriever and refactor search scoring - Added comprehensive tests for TieredLexicalRetriever, covering folder boosting and result combination. - Refactored TieredLexicalRetriever to utilize SearchCore instead of TieredRetriever. - Enhanced FullTextEngine with improved scoring mechanisms, including field weighting and multi-field match bonuses. - Introduced FuzzyMatcher utility for fuzzy matching capabilities, including Levenshtein distance and variant generation. - Updated GrepScanner to prioritize path matching for faster search results. - Implemented normalized scoring in RRF for better ranking consistency. * feat: Update vault search result display to show snippet of content instead of full text * feat: Enhance FullTextEngine to index frontmatter property values and improve search scoring * feat: Improve scoring and logging * feat: Refactor TieredLexicalRetriever to support time-based queries and improve document retrieval logic * Remove TODO.md from tracking and add to .gitignore - TODO.md is now a local-only development session tracker - Prevents accidental commits of work-in-progress task lists - Each developer can maintain their own TODO.md without conflicts * feat: Update dependencies and enhance search functionality - Upgraded axios to version 1.11.0 and electron to version 27.3.11 for improved performance and security. - Refactored QueryExpander to limit queries to the original for strict fallback tests. - Enhanced SearchCore to rank grep hits by evidence quality before fusion, improving retrieval accuracy. - Implemented a new method in SearchCore to rank grep hits based on evidence strength. - Updated TieredLexicalRetriever tests to ensure proper integration with mocked SearchCore. - Improved FuzzyMatcher to include plural/singular normalization for better fuzzy matching. * feat: Add tool call marker encoding/decoding - Introduced new utility functions for encoding and decoding tool call markers to ensure safe embedding in HTML comments. - Updated `updateChatMemory` to handle encoded tool call markers, preserving their integrity during memory storage. - Enhanced logging to decode tool marker results for better readability while maintaining encoded formats for storage. - Added comprehensive tests for tool call marker functionality to ensure correct encoding and decoding behavior. - Refactored related components to integrate the new encoding/decoding logic seamlessly. * feat: Enhance FullTextEngine search to support low-weight terms and improve scoring - Updated FullTextEngine to accept low-weight terms in the search method, allowing for better handling of salient terms. - Implemented downweighting for boolean and numeric tokens in the properties field to reduce noise in search results. - Added a new test case to validate the downweighting behavior for boolean and numeric queries in FullTextEngine. - Improved logging for full-text search results to provide clearer insights into the search process. * feat: Update GraphExpander to enhance search recall with adaptive hop logic - Implemented guardrails in GraphExpander to adjust hop depth based on the number of grep hits: allows +1 hop for small sets (<5) and limits to 1 hop for large sets (≥50). - Updated README.md to reflect new guardrail logic and scoring normalization in search results. - Enhanced test cases to validate the new behavior of skipping co-citations for large result sets and allowing additional hops for smaller sets. * feat: Filter out background tools during streaming to enhance user experience - Added logic to determine and filter out background tools, preventing their names from appearing in the tool call display during streaming. - Implemented a mechanism to include partial tool names only if they meet a specified length threshold, improving clarity in tool call presentations. * feat: Enhance search tools to support time-based queries and display modified time - Updated localSearchTool to ensure a healthy cap on max source chunks for time-based queries, improving recall. - Modified ToolResultFormatter to display actual modified time for time-filtered results, enhancing clarity in search results. - Adjusted output formatting to differentiate between recency and relevance scores based on query type. * feat: Introduce semantic search capabilities with Memory Index support - Added `enableSemanticSearchV3` setting to control the new semantic search feature. - Implemented `MemoryIndexManager` for managing in-memory vector indexing with JSONL persistence. - Enhanced `SearchCore` to utilize semantic retrieval based on the new memory index. - Introduced command to build the semantic memory index, improving search accuracy and retrieval efficiency. - Updated relevant components to integrate semantic search functionality seamlessly. * feat: Add unit tests for MemoryIndexManager to validate functionality - Introduced comprehensive tests for the MemoryIndexManager, covering scenarios such as loading from a file, building a vector store, and indexing vault contents. - Implemented a mock embeddings API to facilitate testing without external dependencies. - Added a utility function to reset the MemoryIndexManager state between tests, ensuring isolation and reliability of test outcomes. * feat: Enhance MemoryIndexManager and related components for improved semantic indexing - Introduced incremental indexing capabilities in MemoryIndexManager to update the JSONL index with new or modified files, enhancing efficiency. - Updated commands to utilize the new incremental indexing method, providing users with real-time updates to the semantic memory index. - Refactored existing indexing logic to support partitioned writes, improving performance and manageability of large datasets. - Enhanced user notifications during indexing processes to provide better feedback on progress and status. - Added a setting to enable semantic search, allowing users to blend semantic similarity into search results seamlessly. * feat: Refine scoring display and enhance search result handling - Updated SourcesModal to display relevance scores with four decimal places for consistency with SearchCore logs. - Enhanced TieredLexicalRetriever to include a new `rerank_score` field for better score management and consistency across search results. - Modified the combineResults method to ensure that title matches retain their original score while incorporating a fused score as `rerank_score`. - Adjusted formatting in SearchTools to ensure both `score` and `rerank_score` reflect the same final score when present. - Updated ToolResultFormatter to display scores with four decimal places, improving clarity in search result presentations. * refactor: Simplify relevant notes retrieval by removing VectorStoreManager dependency - Removed the use of VectorStoreManager in the relevant notes fetching logic, streamlining the process. - Integrated MemoryIndexManager for improved handling of in-memory indexing and note retrieval. - Enhanced error handling to ensure relevant notes are only fetched when the embedding index is available. - Updated related functions to utilize the new memory index approach, improving performance and reliability. * refactor: Remove VectorStoreManager dependency and streamline indexing logic - Eliminated the VectorStoreManager from various components, transitioning to MemoryIndexManager for indexing and retrieval. - Updated project and chain managers to remove unnecessary dependencies, enhancing code clarity and maintainability. - Improved error handling and indexing conditions, particularly for mobile users, ensuring a more robust initialization process. - Refactored settings components to utilize the new memory indexing approach, simplifying the user experience and reducing legacy code. * chore: Mark legacy components as deprecated in preparation for v3 transition - Annotated various files including DebugSearchModal, OramaSearchModal, chunkedStorage, dbOperations, hybridRetriever, and vectorStoreManager as deprecated, indicating they are obsolete in v3. - Updated README.md to reflect the current implementation status and migration notes, emphasizing the removal of Orama-based modules and the transition to MemoryIndexManager for indexing and retrieval. * feat: Implement file tracking and reindexing for modified files - Introduced a new FileTrackingState interface to manage the last active file and its modification time. - Enhanced the CopilotPlugin to opportunistically reindex the previous active file if it was modified while active, contingent on semantic search settings. - Updated MemoryIndexManager to support reindexing of single modified files, improving efficiency in handling changes. - Added unit tests for reindexSingleFileIfModified to ensure correct functionality and performance. * feat: Add graph hops setting for enhanced search result expansion - Introduced a new `graphHops` setting in `CopilotSettings` to control the number of hops for graph expansion during search, with a default value of 1 and a range of 1-3. - Updated the `sanitizeSettings` function to validate the `graphHops` value. - Enhanced the `SearchCore` and `TieredLexicalRetriever` classes to utilize the `graphHops` setting for improved search result relevance. - Updated documentation in `README.md` to reflect the new feature and its security optimizations. * refactor: Improve error handling and indexing logic in MemoryIndexManager and related components - Replaced console error logging with structured logging using logError and logWarn for better error tracking. - Enhanced the refresh and reindexing functions to ensure proper user notifications and error handling. - Updated the logic for managing indexed files, including handling exclusions and ensuring accurate reporting of indexed and unindexed files. - Implemented rate limiting in the MemoryIndexManager to optimize embedding requests and prevent overloading the service. - Refactored chunk processing to improve efficiency and clarity in the indexing workflow. * chore: Update README.md to reflect final session completion and key fixes - Expanded the final session summary with a date and detailed list of completed features and fixes, including UI enhancements and improved logging practices. - Clarified the status of deferred features and provided migration notes for better user guidance. - Ensured documentation aligns with the latest implementation changes and optimizations. * chore: Update memory limits in README.md and MemoryManager.ts for improved performance - Adjusted memory limits for mobile and desktop platforms in both README.md and MemoryManager.ts to reflect increased capacity (20MB mobile, 100MB desktop). - Ensured documentation aligns with the latest implementation changes for better clarity on resource management. * feat: Integrate HyDE document generation into SearchCore for enhanced semantic search - Implemented a new method to generate hypothetical documents (HyDE) to improve semantic search capabilities. - Added timeout handling for HyDE generation to ensure graceful error management. - Updated SearchCore to utilize HyDE documents in search queries, enhancing the relevance of results when semantic search is enabled. - Introduced unit tests to validate the integration and functionality of HyDE generation within the search process. * refactor: Enhance MemoryIndexManager with public methods for better indexing management - Introduced public methods in MemoryIndexManager to streamline access to indexed file paths, check if a file is indexed, retrieve embeddings, and clear the index. - Updated related components to utilize these new methods, improving code clarity and reducing direct access to internal properties. - Enhanced error handling and user notifications during memory index operations. * feat: Implement indexing notification and progress management - Introduced the IndexingNotificationManager to handle UI notifications during indexing operations, allowing users to pause or stop the process. - Added the IndexingProgressTracker to track progress across multiple files, enhancing user feedback during indexing. - Developed the IndexingPipeline to manage the chunking and processing of files into embeddings, improving the overall indexing workflow. - Created the IndexPersistenceManager for managing the persistence of indexed data, ensuring efficient storage and retrieval of index records. - Refactored MemoryIndexManager to utilize the new components, streamlining the indexing process and improving code organization. --------- Co-authored-by: Claude <noreply@anthropic.com>
32 lines
458 B
Text
32 lines
458 B
Text
# vscode
|
|
.vscode
|
|
|
|
# Intellij
|
|
*.iml
|
|
.idea
|
|
|
|
# npm
|
|
node_modules
|
|
|
|
# Don't include the compiled main.js file in the repo.
|
|
# They should be uploaded to GitHub releases instead.
|
|
main.js
|
|
styles.css
|
|
|
|
# Exclude sourcemaps
|
|
*.map
|
|
|
|
# obsidian
|
|
data.json
|
|
|
|
# Exclude macOS Finder (System Explorer) View States
|
|
.DS_Store
|
|
|
|
# Test Environment
|
|
.env.test
|
|
|
|
# Claude configuration
|
|
.claude/settings.local.json
|
|
|
|
# Development session tracking
|
|
TODO.md
|