* Add LexicalEngine and related interfaces for enhanced search functionality - Implement LexicalEngine class utilizing FlexSearch for efficient document indexing and searching. - Create Hit, SearchResult, SearchOptions, and RetrieverEngine interfaces to standardize search operations. - Update package.json and package-lock.json to include new dependencies: flexsearch and p-queue. * Add QueryExpander class and tests for enhanced search query expansion * Add README for Tiered Note-Level Lexical Retrieval with multilingual support and enhanced search pipeline * Implement v3 tiered search with GrepScanner, FullTextEngine, and GraphExpander - Add GrepScanner for fast substring search (L0) - Implement FullTextEngine with ephemeral FlexSearch index (L1) - Add GraphExpander for link-based candidate expansion - Create MemoryManager with platform-aware limits - Implement weighted RRF fusion for result combination - Add SemanticReranker placeholder for future integration - Include comprehensive tests for core components - Simplify code based on review: use TextEncoder, extract methods, streamline RRF - Add clear logging for debugging search pipeline The architecture follows a tiered approach: 1. Grep scan for initial candidates 2. Graph expansion to increase recall 3. Full-text search on expanded set 4. Optional semantic reranking 5. RRF fusion to combine signals 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Enhance tiered retrieval recall with both rewritten queries and the expanded salient terms - Added detailed example of end-to-end query processing in README.md - Updated TieredRetriever to log salient terms during query expansion - Modified FullTextEngine to index link basenames for improved searchability - Adjusted NoteDoc interface to clarify link handling and indexing * Refactor QueryExpander configuration and caching logic for improved clarity and performance; enhance GraphExpander documentation and testing; remove unused fields from NoteDoc interface. * Refactor search retrieval system to wire in TieredLexicalRetriever - Replaced HybridRetriever with TieredLexicalRetriever in ChainManager, VaultQAChainRunner, and main.ts for improved multi-stage retrieval. - Updated README.md to reflect new integration tasks and performance benchmarks. - Introduced TieredLexicalRetriever class to handle on-demand indexing and retrieval. - Enhanced GrepScanner to support inclusion/exclusion patterns for file indexing. - Modified TieredRetriever to combine expanded salient terms and improved logging for final results. - Removed legacy index management in favor of ephemeral indexing with TieredLexicalRetriever. - Updated interfaces and utility functions to support new retrieval logic. * feat: Implement first working TieredLexicalRetriever and refactor search scoring - Added comprehensive tests for TieredLexicalRetriever, covering folder boosting and result combination. - Refactored TieredLexicalRetriever to utilize SearchCore instead of TieredRetriever. - Enhanced FullTextEngine with improved scoring mechanisms, including field weighting and multi-field match bonuses. - Introduced FuzzyMatcher utility for fuzzy matching capabilities, including Levenshtein distance and variant generation. - Updated GrepScanner to prioritize path matching for faster search results. - Implemented normalized scoring in RRF for better ranking consistency. * feat: Update vault search result display to show snippet of content instead of full text * feat: Enhance FullTextEngine to index frontmatter property values and improve search scoring * feat: Improve scoring and logging * feat: Refactor TieredLexicalRetriever to support time-based queries and improve document retrieval logic * Remove TODO.md from tracking and add to .gitignore - TODO.md is now a local-only development session tracker - Prevents accidental commits of work-in-progress task lists - Each developer can maintain their own TODO.md without conflicts * feat: Update dependencies and enhance search functionality - Upgraded axios to version 1.11.0 and electron to version 27.3.11 for improved performance and security. - Refactored QueryExpander to limit queries to the original for strict fallback tests. - Enhanced SearchCore to rank grep hits by evidence quality before fusion, improving retrieval accuracy. - Implemented a new method in SearchCore to rank grep hits based on evidence strength. - Updated TieredLexicalRetriever tests to ensure proper integration with mocked SearchCore. - Improved FuzzyMatcher to include plural/singular normalization for better fuzzy matching. * feat: Add tool call marker encoding/decoding - Introduced new utility functions for encoding and decoding tool call markers to ensure safe embedding in HTML comments. - Updated `updateChatMemory` to handle encoded tool call markers, preserving their integrity during memory storage. - Enhanced logging to decode tool marker results for better readability while maintaining encoded formats for storage. - Added comprehensive tests for tool call marker functionality to ensure correct encoding and decoding behavior. - Refactored related components to integrate the new encoding/decoding logic seamlessly. * feat: Enhance FullTextEngine search to support low-weight terms and improve scoring - Updated FullTextEngine to accept low-weight terms in the search method, allowing for better handling of salient terms. - Implemented downweighting for boolean and numeric tokens in the properties field to reduce noise in search results. - Added a new test case to validate the downweighting behavior for boolean and numeric queries in FullTextEngine. - Improved logging for full-text search results to provide clearer insights into the search process. * feat: Update GraphExpander to enhance search recall with adaptive hop logic - Implemented guardrails in GraphExpander to adjust hop depth based on the number of grep hits: allows +1 hop for small sets (<5) and limits to 1 hop for large sets (≥50). - Updated README.md to reflect new guardrail logic and scoring normalization in search results. - Enhanced test cases to validate the new behavior of skipping co-citations for large result sets and allowing additional hops for smaller sets. * feat: Filter out background tools during streaming to enhance user experience - Added logic to determine and filter out background tools, preventing their names from appearing in the tool call display during streaming. - Implemented a mechanism to include partial tool names only if they meet a specified length threshold, improving clarity in tool call presentations. * feat: Enhance search tools to support time-based queries and display modified time - Updated localSearchTool to ensure a healthy cap on max source chunks for time-based queries, improving recall. - Modified ToolResultFormatter to display actual modified time for time-filtered results, enhancing clarity in search results. - Adjusted output formatting to differentiate between recency and relevance scores based on query type. * feat: Introduce semantic search capabilities with Memory Index support - Added `enableSemanticSearchV3` setting to control the new semantic search feature. - Implemented `MemoryIndexManager` for managing in-memory vector indexing with JSONL persistence. - Enhanced `SearchCore` to utilize semantic retrieval based on the new memory index. - Introduced command to build the semantic memory index, improving search accuracy and retrieval efficiency. - Updated relevant components to integrate semantic search functionality seamlessly. * feat: Add unit tests for MemoryIndexManager to validate functionality - Introduced comprehensive tests for the MemoryIndexManager, covering scenarios such as loading from a file, building a vector store, and indexing vault contents. - Implemented a mock embeddings API to facilitate testing without external dependencies. - Added a utility function to reset the MemoryIndexManager state between tests, ensuring isolation and reliability of test outcomes. * feat: Enhance MemoryIndexManager and related components for improved semantic indexing - Introduced incremental indexing capabilities in MemoryIndexManager to update the JSONL index with new or modified files, enhancing efficiency. - Updated commands to utilize the new incremental indexing method, providing users with real-time updates to the semantic memory index. - Refactored existing indexing logic to support partitioned writes, improving performance and manageability of large datasets. - Enhanced user notifications during indexing processes to provide better feedback on progress and status. - Added a setting to enable semantic search, allowing users to blend semantic similarity into search results seamlessly. * feat: Refine scoring display and enhance search result handling - Updated SourcesModal to display relevance scores with four decimal places for consistency with SearchCore logs. - Enhanced TieredLexicalRetriever to include a new `rerank_score` field for better score management and consistency across search results. - Modified the combineResults method to ensure that title matches retain their original score while incorporating a fused score as `rerank_score`. - Adjusted formatting in SearchTools to ensure both `score` and `rerank_score` reflect the same final score when present. - Updated ToolResultFormatter to display scores with four decimal places, improving clarity in search result presentations. * refactor: Simplify relevant notes retrieval by removing VectorStoreManager dependency - Removed the use of VectorStoreManager in the relevant notes fetching logic, streamlining the process. - Integrated MemoryIndexManager for improved handling of in-memory indexing and note retrieval. - Enhanced error handling to ensure relevant notes are only fetched when the embedding index is available. - Updated related functions to utilize the new memory index approach, improving performance and reliability. * refactor: Remove VectorStoreManager dependency and streamline indexing logic - Eliminated the VectorStoreManager from various components, transitioning to MemoryIndexManager for indexing and retrieval. - Updated project and chain managers to remove unnecessary dependencies, enhancing code clarity and maintainability. - Improved error handling and indexing conditions, particularly for mobile users, ensuring a more robust initialization process. - Refactored settings components to utilize the new memory indexing approach, simplifying the user experience and reducing legacy code. * chore: Mark legacy components as deprecated in preparation for v3 transition - Annotated various files including DebugSearchModal, OramaSearchModal, chunkedStorage, dbOperations, hybridRetriever, and vectorStoreManager as deprecated, indicating they are obsolete in v3. - Updated README.md to reflect the current implementation status and migration notes, emphasizing the removal of Orama-based modules and the transition to MemoryIndexManager for indexing and retrieval. * feat: Implement file tracking and reindexing for modified files - Introduced a new FileTrackingState interface to manage the last active file and its modification time. - Enhanced the CopilotPlugin to opportunistically reindex the previous active file if it was modified while active, contingent on semantic search settings. - Updated MemoryIndexManager to support reindexing of single modified files, improving efficiency in handling changes. - Added unit tests for reindexSingleFileIfModified to ensure correct functionality and performance. * feat: Add graph hops setting for enhanced search result expansion - Introduced a new `graphHops` setting in `CopilotSettings` to control the number of hops for graph expansion during search, with a default value of 1 and a range of 1-3. - Updated the `sanitizeSettings` function to validate the `graphHops` value. - Enhanced the `SearchCore` and `TieredLexicalRetriever` classes to utilize the `graphHops` setting for improved search result relevance. - Updated documentation in `README.md` to reflect the new feature and its security optimizations. * refactor: Improve error handling and indexing logic in MemoryIndexManager and related components - Replaced console error logging with structured logging using logError and logWarn for better error tracking. - Enhanced the refresh and reindexing functions to ensure proper user notifications and error handling. - Updated the logic for managing indexed files, including handling exclusions and ensuring accurate reporting of indexed and unindexed files. - Implemented rate limiting in the MemoryIndexManager to optimize embedding requests and prevent overloading the service. - Refactored chunk processing to improve efficiency and clarity in the indexing workflow. * chore: Update README.md to reflect final session completion and key fixes - Expanded the final session summary with a date and detailed list of completed features and fixes, including UI enhancements and improved logging practices. - Clarified the status of deferred features and provided migration notes for better user guidance. - Ensured documentation aligns with the latest implementation changes and optimizations. * chore: Update memory limits in README.md and MemoryManager.ts for improved performance - Adjusted memory limits for mobile and desktop platforms in both README.md and MemoryManager.ts to reflect increased capacity (20MB mobile, 100MB desktop). - Ensured documentation aligns with the latest implementation changes for better clarity on resource management. * feat: Integrate HyDE document generation into SearchCore for enhanced semantic search - Implemented a new method to generate hypothetical documents (HyDE) to improve semantic search capabilities. - Added timeout handling for HyDE generation to ensure graceful error management. - Updated SearchCore to utilize HyDE documents in search queries, enhancing the relevance of results when semantic search is enabled. - Introduced unit tests to validate the integration and functionality of HyDE generation within the search process. * refactor: Enhance MemoryIndexManager with public methods for better indexing management - Introduced public methods in MemoryIndexManager to streamline access to indexed file paths, check if a file is indexed, retrieve embeddings, and clear the index. - Updated related components to utilize these new methods, improving code clarity and reducing direct access to internal properties. - Enhanced error handling and user notifications during memory index operations. * feat: Implement indexing notification and progress management - Introduced the IndexingNotificationManager to handle UI notifications during indexing operations, allowing users to pause or stop the process. - Added the IndexingProgressTracker to track progress across multiple files, enhancing user feedback during indexing. - Developed the IndexingPipeline to manage the chunking and processing of files into embeddings, improving the overall indexing workflow. - Created the IndexPersistenceManager for managing the persistence of indexed data, ensuring efficient storage and retrieval of index records. - Refactored MemoryIndexManager to utilize the new components, streamlining the indexing process and improving code organization. --------- Co-authored-by: Claude <noreply@anthropic.com>
11 KiB
CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
Overview
Copilot for Obsidian is an AI-powered assistant plugin that integrates various LLM providers (OpenAI, Anthropic, Google, etc.) with Obsidian. It provides chat interfaces, autocomplete, semantic search, and various AI-powered commands for note-taking and knowledge management.
Development Commands
Build & Development
- NEVER RUN
npm run dev- The user will handle all builds manually npm run build- Production build (TypeScript check + minified output)
Code Quality
npm run lint- Run ESLint checksnpm run lint:fix- Auto-fix ESLint issuesnpm run format- Format code with Prettiernpm run format:check- Check formatting without changing files- Before PR: Always run
npm run format && npm run lint
Testing
npm run test- Run unit tests (excludes integration tests)npm run test:integration- Run integration tests (requires API keys)- Run single test:
npm test -- -t "test name"
High-Level Architecture
Core Systems
-
LLM Provider System (
src/LLMProviders/)- Provider implementations for OpenAI, Anthropic, Google, Azure, local models
LLMProviderManagerhandles provider lifecycle and switching- Stream-based responses with error handling and rate limiting
- Custom model configuration support
-
Chain Factory Pattern (
src/chainFactory.ts)- Different chain types for various AI operations (chat, copilot, adhoc prompts)
- LangChain integration for complex workflows
- Memory management for conversation context
- Tool integration (search, file operations, time queries)
-
Vector Store & Search (
src/search/)VectorStoreManagermanages embeddings and semantic searchChunkedStoragefor efficient large document handling- Event-driven index updates via
IndexManager - Multiple embedding providers support
-
UI Component System (
src/components/)- React functional components with Radix UI primitives
- Tailwind CSS with class variance authority (CVA)
- Modal system for user interactions
- Chat interface with streaming support
- Settings UI with versioned components
-
Message Management Architecture (
src/core/,src/state/)- MessageRepository (
src/core/MessageRepository.ts): Single source of truth for all messages- Stores each message once with both
displayTextandprocessedText - Provides computed views for UI display and LLM processing
- No complex dual-array synchronization
- Stores each message once with both
- ChatManager (
src/core/ChatManager.ts): Central business logic coordinator- Orchestrates MessageRepository, ContextManager, and LLM operations
- Handles message sending, editing, regeneration, and deletion
- Manages context processing and chain memory synchronization
- Project Chat Isolation: Maintains separate MessageRepository per project
- Automatically detects project switches via
getCurrentMessageRepo() - Each project has its own isolated message history
- Non-project chats use
defaultProjectKeyrepository
- Automatically detects project switches via
- ChatUIState (
src/state/ChatUIState.ts): Clean UI-only state manager- Delegates all business logic to ChatManager
- Provides React integration with subscription mechanism
- Replaces legacy SharedState with minimal, focused approach
- ContextManager (
src/core/ContextManager.ts): Handles context processing- Processes message context (notes, URLs, selected text)
- Reprocesses context when messages are edited
- MessageRepository (
-
Settings Management
- Jotai for atomic settings state management
- React contexts for feature-specific state
-
Plugin Integration
- Main entry:
src/main.tsextends Obsidian Plugin - Command registration system
- Event handling for Obsidian lifecycle
- Settings persistence and migration
- Chat history loading via pending message mechanism
- Main entry:
Key Patterns
- Single Source of Truth: MessageRepository stores each message once with computed views
- Clean Architecture: Repository → Manager → UIState → React Components
- Context Reprocessing: Automatic context updates when messages are edited
- Computed Views: Display messages for UI, LLM messages for AI processing
- Project Isolation: Each project maintains its own MessageRepository instance
- Error Handling: Custom error types with detailed interfaces
- Async Operations: Consistent async/await pattern with proper error boundaries
- Caching: Multi-layer caching for files, PDFs, and API responses
- Streaming: Real-time streaming for LLM responses
- Testing: Unit tests adjacent to implementation, integration tests for API calls
Message Management Architecture
For detailed architecture diagrams and documentation, see MESSAGE_ARCHITECTURE.md.
Core Classes and Flow
-
MessageRepository (
src/core/MessageRepository.ts)- Single source of truth for all messages
- Stores
StoredMessageobjects with bothdisplayTextandprocessedText - Provides computed views via
getDisplayMessages()andgetLLMMessages() - No complex dual-array synchronization or ID matching
-
ChatManager (
src/core/ChatManager.ts)- Central business logic coordinator
- Orchestrates MessageRepository, ContextManager, and LLM operations
- Handles all message CRUD operations with proper error handling
- Synchronizes with chain memory for conversation history
- Project Chat Isolation Implementation:
- Maintains
projectMessageRepos: Map<string, MessageRepository>for project-specific storage getCurrentMessageRepo()automatically detects current project and returns correct repository- Seamlessly switches between project repositories when project changes
- Creates new empty repository for each project (no message caching)
- Maintains
-
ChatUIState (
src/state/ChatUIState.ts)- Clean UI-only state manager
- Delegates all business logic to ChatManager
- Provides React integration with subscription mechanism
- Replaces legacy SharedState with minimal, focused approach
-
ContextManager (
src/core/ContextManager.ts)- Handles context processing (notes, URLs, selected text)
- Reprocesses context when messages are edited
- Ensures fresh context for LLM processing
-
ChatPersistenceManager (
src/core/ChatPersistenceManager.ts)- Handles saving and loading chat history to/from markdown files
- Project-aware file naming (prefixes with project ID)
- Parses and formats chat content for storage
- Integrated with ChatManager for seamless persistence
Code Style Guidelines
MAJOR PRINCIPLES
- ALWAYS WRITE GENERALIZABLE SOLUTIONS: Never add edge-case handling or hardcoded logic for specific scenarios (like "piano notes" or "daily notes"). Solutions must work for all cases.
- Avoid hardcoding: No hardcoded folder names, file patterns, or special-case logic
- Configuration over convention: If behavior needs to vary, make it configurable, not hardcoded
- Universal patterns: Solutions should work equally well for any folder structure, naming convention, or content type
TypeScript
- Strict mode enabled (no implicit any, strict null checks)
- Use absolute imports with
@/prefix:import { ChainType } from "@/chainFactory" - Prefer const assertions and type inference where appropriate
- Use interface for object shapes, type for unions/aliases
React
- Functional components only (no class components)
- Custom hooks for reusable logic
- Props interfaces defined above components
- Avoid inline styles, use Tailwind classes
General
- File naming: PascalCase for components, camelCase for utilities
- Async/await over promises
- Early returns for error conditions
- Always add JSDoc comments for all functions and methods
- Organize imports: React → external → internal
- Avoid language-specific lists (like stopwords or action verbs) - use language-agnostic approaches instead
Logging
- NEVER use console.log - Use the logging utilities instead:
logInfo()for informational messageslogWarn()for warningslogError()for errors
- Import from logger:
import { logInfo, logWarn, logError } from "@/logger"
Testing Guidelines
- Unit tests use Jest with TypeScript support
- Mock Obsidian API for plugin testing
- Integration tests require API keys in
.env.test - Test files adjacent to implementation (
.test.ts) - Use
@testing-library/reactfor component testing
Development Session Planning
Using TODO.md for Session Management
IMPORTANT: When working on a development session, maintain a comprehensive TODO.md file that serves as the central plan and tracker:
- Session Goal: Define the high-level objective at the start
- Task Tracking:
- List all completed tasks with [x] checkboxes
- Track pending tasks with [ ] checkboxes
- Group related tasks into logical sections
- Architecture Decisions: Document key design choices and rationale
- Progress Updates: Keep the TODO.md updated as tasks complete
- Testing Checklist: Include verification steps for the session
The TODO.md should be:
- The single source of truth for session progress
- Updated frequently as work progresses
- Clear enough that another developer can understand what was done
- Comprehensive enough to serve as a migration guide
Structure Example:
# Development Session TODO
## Session Goal
[Clear statement of what this session aims to achieve]
## Completed Tasks ✅
- [x] Task description with key details
- [x] Another completed task
## Pending Tasks 📋
- [ ] Next task to work on
- [ ] Future enhancement
## Architecture Summary
[Key design decisions and rationale]
## Testing Checklist
- [ ] Functionality verification
- [ ] Performance checks
Important Notes
- The plugin supports multiple LLM providers with custom endpoints
- Vector store requires rebuilding when switching embedding providers
- Settings are versioned - migrations may be needed
- Local model support available via Ollama/LM Studio
- Rate limiting is implemented for all API calls
- For technical debt and known issues, see
TECHDEBT.md - For current development session planning, see
TODO.md
Obsidian Plugin Environment
- Global
appvariable: In Obsidian plugins,appis a globally available variable that provides access to the Obsidian API. It's automatically available in all files without needing to import or declare it.
Architecture Migration Notes
- SharedState Removed: The legacy
src/sharedState.tshas been completely removed - Clean Architecture: New architecture follows Repository → Manager → UIState → UI pattern
- Single Source of Truth: All messages stored once in MessageRepository with computed views
- Context Always Fresh: Context is reprocessed when messages are edited to ensure accuracy
- Chat History Loading: Uses pending message mechanism through CopilotView → Chat component props
- Project Chat Isolation: Each project now has completely isolated chat history
- Automatic detection of project switches via
ProjectManager.getCurrentProjectId() - Separate MessageRepository instances per project ID
- Non-project chats stored in default repository
- Backwards compatible - loads existing messages from ProjectManager cache
- Zero configuration required - works automatically
- Automatic detection of project switches via