logancyang_obsidian-copilot/CLAUDE.md
Logan Yang df3efaecc0
Implement vault search v3 (#1703)
* Add LexicalEngine and related interfaces for enhanced search functionality

- Implement LexicalEngine class utilizing FlexSearch for efficient document indexing and searching.
- Create Hit, SearchResult, SearchOptions, and RetrieverEngine interfaces to standardize search operations.
- Update package.json and package-lock.json to include new dependencies: flexsearch and p-queue.

* Add QueryExpander class and tests for enhanced search query expansion

* Add README for Tiered Note-Level Lexical Retrieval with multilingual support and enhanced search pipeline

* Implement v3 tiered search with GrepScanner, FullTextEngine, and GraphExpander

- Add GrepScanner for fast substring search (L0)
- Implement FullTextEngine with ephemeral FlexSearch index (L1)
- Add GraphExpander for link-based candidate expansion
- Create MemoryManager with platform-aware limits
- Implement weighted RRF fusion for result combination
- Add SemanticReranker placeholder for future integration
- Include comprehensive tests for core components
- Simplify code based on review: use TextEncoder, extract methods, streamline RRF
- Add clear logging for debugging search pipeline

The architecture follows a tiered approach:
1. Grep scan for initial candidates
2. Graph expansion to increase recall
3. Full-text search on expanded set
4. Optional semantic reranking
5. RRF fusion to combine signals

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Enhance tiered retrieval recall with both rewritten queries and the expanded salient terms

- Added detailed example of end-to-end query processing in README.md
- Updated TieredRetriever to log salient terms during query expansion
- Modified FullTextEngine to index link basenames for improved searchability
- Adjusted NoteDoc interface to clarify link handling and indexing

* Refactor QueryExpander configuration and caching logic for improved clarity and performance; enhance GraphExpander documentation and testing; remove unused fields from NoteDoc interface.

* Refactor search retrieval system to wire in TieredLexicalRetriever

- Replaced HybridRetriever with TieredLexicalRetriever in ChainManager, VaultQAChainRunner, and main.ts for improved multi-stage retrieval.
- Updated README.md to reflect new integration tasks and performance benchmarks.
- Introduced TieredLexicalRetriever class to handle on-demand indexing and retrieval.
- Enhanced GrepScanner to support inclusion/exclusion patterns for file indexing.
- Modified TieredRetriever to combine expanded salient terms and improved logging for final results.
- Removed legacy index management in favor of ephemeral indexing with TieredLexicalRetriever.
- Updated interfaces and utility functions to support new retrieval logic.

* feat: Implement first working TieredLexicalRetriever and refactor search scoring

- Added comprehensive tests for TieredLexicalRetriever, covering folder boosting and result combination.
- Refactored TieredLexicalRetriever to utilize SearchCore instead of TieredRetriever.
- Enhanced FullTextEngine with improved scoring mechanisms, including field weighting and multi-field match bonuses.
- Introduced FuzzyMatcher utility for fuzzy matching capabilities, including Levenshtein distance and variant generation.
- Updated GrepScanner to prioritize path matching for faster search results.
- Implemented normalized scoring in RRF for better ranking consistency.

* feat: Update vault search result display to show snippet of content instead of full text

* feat: Enhance FullTextEngine to index frontmatter property values and improve search scoring

* feat: Improve scoring and logging

* feat: Refactor TieredLexicalRetriever to support time-based queries and improve document retrieval logic

* Remove TODO.md from tracking and add to .gitignore

- TODO.md is now a local-only development session tracker
- Prevents accidental commits of work-in-progress task lists
- Each developer can maintain their own TODO.md without conflicts

* feat: Update dependencies and enhance search functionality

- Upgraded axios to version 1.11.0 and electron to version 27.3.11 for improved performance and security.
- Refactored QueryExpander to limit queries to the original for strict fallback tests.
- Enhanced SearchCore to rank grep hits by evidence quality before fusion, improving retrieval accuracy.
- Implemented a new method in SearchCore to rank grep hits based on evidence strength.
- Updated TieredLexicalRetriever tests to ensure proper integration with mocked SearchCore.
- Improved FuzzyMatcher to include plural/singular normalization for better fuzzy matching.

* feat: Add tool call marker encoding/decoding

- Introduced new utility functions for encoding and decoding tool call markers to ensure safe embedding in HTML comments.
- Updated `updateChatMemory` to handle encoded tool call markers, preserving their integrity during memory storage.
- Enhanced logging to decode tool marker results for better readability while maintaining encoded formats for storage.
- Added comprehensive tests for tool call marker functionality to ensure correct encoding and decoding behavior.
- Refactored related components to integrate the new encoding/decoding logic seamlessly.

* feat: Enhance FullTextEngine search to support low-weight terms and improve scoring

- Updated FullTextEngine to accept low-weight terms in the search method, allowing for better handling of salient terms.
- Implemented downweighting for boolean and numeric tokens in the properties field to reduce noise in search results.
- Added a new test case to validate the downweighting behavior for boolean and numeric queries in FullTextEngine.
- Improved logging for full-text search results to provide clearer insights into the search process.

* feat: Update GraphExpander to enhance search recall with adaptive hop logic

- Implemented guardrails in GraphExpander to adjust hop depth based on the number of grep hits: allows +1 hop for small sets (<5) and limits to 1 hop for large sets (≥50).
- Updated README.md to reflect new guardrail logic and scoring normalization in search results.
- Enhanced test cases to validate the new behavior of skipping co-citations for large result sets and allowing additional hops for smaller sets.

* feat: Filter out background tools during streaming to enhance user experience

- Added logic to determine and filter out background tools, preventing their names from appearing in the tool call display during streaming.
- Implemented a mechanism to include partial tool names only if they meet a specified length threshold, improving clarity in tool call presentations.

* feat: Enhance search tools to support time-based queries and display modified time

- Updated localSearchTool to ensure a healthy cap on max source chunks for time-based queries, improving recall.
- Modified ToolResultFormatter to display actual modified time for time-filtered results, enhancing clarity in search results.
- Adjusted output formatting to differentiate between recency and relevance scores based on query type.

* feat: Introduce semantic search capabilities with Memory Index support

- Added `enableSemanticSearchV3` setting to control the new semantic search feature.
- Implemented `MemoryIndexManager` for managing in-memory vector indexing with JSONL persistence.
- Enhanced `SearchCore` to utilize semantic retrieval based on the new memory index.
- Introduced command to build the semantic memory index, improving search accuracy and retrieval efficiency.
- Updated relevant components to integrate semantic search functionality seamlessly.

* feat: Add unit tests for MemoryIndexManager to validate functionality

- Introduced comprehensive tests for the MemoryIndexManager, covering scenarios such as loading from a file, building a vector store, and indexing vault contents.
- Implemented a mock embeddings API to facilitate testing without external dependencies.
- Added a utility function to reset the MemoryIndexManager state between tests, ensuring isolation and reliability of test outcomes.

* feat: Enhance MemoryIndexManager and related components for improved semantic indexing

- Introduced incremental indexing capabilities in MemoryIndexManager to update the JSONL index with new or modified files, enhancing efficiency.
- Updated commands to utilize the new incremental indexing method, providing users with real-time updates to the semantic memory index.
- Refactored existing indexing logic to support partitioned writes, improving performance and manageability of large datasets.
- Enhanced user notifications during indexing processes to provide better feedback on progress and status.
- Added a setting to enable semantic search, allowing users to blend semantic similarity into search results seamlessly.

* feat: Refine scoring display and enhance search result handling

- Updated SourcesModal to display relevance scores with four decimal places for consistency with SearchCore logs.
- Enhanced TieredLexicalRetriever to include a new `rerank_score` field for better score management and consistency across search results.
- Modified the combineResults method to ensure that title matches retain their original score while incorporating a fused score as `rerank_score`.
- Adjusted formatting in SearchTools to ensure both `score` and `rerank_score` reflect the same final score when present.
- Updated ToolResultFormatter to display scores with four decimal places, improving clarity in search result presentations.

* refactor: Simplify relevant notes retrieval by removing VectorStoreManager dependency

- Removed the use of VectorStoreManager in the relevant notes fetching logic, streamlining the process.
- Integrated MemoryIndexManager for improved handling of in-memory indexing and note retrieval.
- Enhanced error handling to ensure relevant notes are only fetched when the embedding index is available.
- Updated related functions to utilize the new memory index approach, improving performance and reliability.

* refactor: Remove VectorStoreManager dependency and streamline indexing logic

- Eliminated the VectorStoreManager from various components, transitioning to MemoryIndexManager for indexing and retrieval.
- Updated project and chain managers to remove unnecessary dependencies, enhancing code clarity and maintainability.
- Improved error handling and indexing conditions, particularly for mobile users, ensuring a more robust initialization process.
- Refactored settings components to utilize the new memory indexing approach, simplifying the user experience and reducing legacy code.

* chore: Mark legacy components as deprecated in preparation for v3 transition

- Annotated various files including DebugSearchModal, OramaSearchModal, chunkedStorage, dbOperations, hybridRetriever, and vectorStoreManager as deprecated, indicating they are obsolete in v3.
- Updated README.md to reflect the current implementation status and migration notes, emphasizing the removal of Orama-based modules and the transition to MemoryIndexManager for indexing and retrieval.

* feat: Implement file tracking and reindexing for modified files

- Introduced a new FileTrackingState interface to manage the last active file and its modification time.
- Enhanced the CopilotPlugin to opportunistically reindex the previous active file if it was modified while active, contingent on semantic search settings.
- Updated MemoryIndexManager to support reindexing of single modified files, improving efficiency in handling changes.
- Added unit tests for reindexSingleFileIfModified to ensure correct functionality and performance.

* feat: Add graph hops setting for enhanced search result expansion

- Introduced a new `graphHops` setting in `CopilotSettings` to control the number of hops for graph expansion during search, with a default value of 1 and a range of 1-3.
- Updated the `sanitizeSettings` function to validate the `graphHops` value.
- Enhanced the `SearchCore` and `TieredLexicalRetriever` classes to utilize the `graphHops` setting for improved search result relevance.
- Updated documentation in `README.md` to reflect the new feature and its security optimizations.

* refactor: Improve error handling and indexing logic in MemoryIndexManager and related components

- Replaced console error logging with structured logging using logError and logWarn for better error tracking.
- Enhanced the refresh and reindexing functions to ensure proper user notifications and error handling.
- Updated the logic for managing indexed files, including handling exclusions and ensuring accurate reporting of indexed and unindexed files.
- Implemented rate limiting in the MemoryIndexManager to optimize embedding requests and prevent overloading the service.
- Refactored chunk processing to improve efficiency and clarity in the indexing workflow.

* chore: Update README.md to reflect final session completion and key fixes

- Expanded the final session summary with a date and detailed list of completed features and fixes, including UI enhancements and improved logging practices.
- Clarified the status of deferred features and provided migration notes for better user guidance.
- Ensured documentation aligns with the latest implementation changes and optimizations.

* chore: Update memory limits in README.md and MemoryManager.ts for improved performance

- Adjusted memory limits for mobile and desktop platforms in both README.md and MemoryManager.ts to reflect increased capacity (20MB mobile, 100MB desktop).
- Ensured documentation aligns with the latest implementation changes for better clarity on resource management.

* feat: Integrate HyDE document generation into SearchCore for enhanced semantic search

- Implemented a new method to generate hypothetical documents (HyDE) to improve semantic search capabilities.
- Added timeout handling for HyDE generation to ensure graceful error management.
- Updated SearchCore to utilize HyDE documents in search queries, enhancing the relevance of results when semantic search is enabled.
- Introduced unit tests to validate the integration and functionality of HyDE generation within the search process.

* refactor: Enhance MemoryIndexManager with public methods for better indexing management

- Introduced public methods in MemoryIndexManager to streamline access to indexed file paths, check if a file is indexed, retrieve embeddings, and clear the index.
- Updated related components to utilize these new methods, improving code clarity and reducing direct access to internal properties.
- Enhanced error handling and user notifications during memory index operations.

* feat: Implement indexing notification and progress management

- Introduced the IndexingNotificationManager to handle UI notifications during indexing operations, allowing users to pause or stop the process.
- Added the IndexingProgressTracker to track progress across multiple files, enhancing user feedback during indexing.
- Developed the IndexingPipeline to manage the chunking and processing of files into embeddings, improving the overall indexing workflow.
- Created the IndexPersistenceManager for managing the persistence of indexed data, ensuring efficient storage and retrieval of index records.
- Refactored MemoryIndexManager to utilize the new components, streamlining the indexing process and improving code organization.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-08-11 16:58:04 -07:00

11 KiB

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Overview

Copilot for Obsidian is an AI-powered assistant plugin that integrates various LLM providers (OpenAI, Anthropic, Google, etc.) with Obsidian. It provides chat interfaces, autocomplete, semantic search, and various AI-powered commands for note-taking and knowledge management.

Development Commands

Build & Development

  • NEVER RUN npm run dev - The user will handle all builds manually
  • npm run build - Production build (TypeScript check + minified output)

Code Quality

  • npm run lint - Run ESLint checks
  • npm run lint:fix - Auto-fix ESLint issues
  • npm run format - Format code with Prettier
  • npm run format:check - Check formatting without changing files
  • Before PR: Always run npm run format && npm run lint

Testing

  • npm run test - Run unit tests (excludes integration tests)
  • npm run test:integration - Run integration tests (requires API keys)
  • Run single test: npm test -- -t "test name"

High-Level Architecture

Core Systems

  1. LLM Provider System (src/LLMProviders/)

    • Provider implementations for OpenAI, Anthropic, Google, Azure, local models
    • LLMProviderManager handles provider lifecycle and switching
    • Stream-based responses with error handling and rate limiting
    • Custom model configuration support
  2. Chain Factory Pattern (src/chainFactory.ts)

    • Different chain types for various AI operations (chat, copilot, adhoc prompts)
    • LangChain integration for complex workflows
    • Memory management for conversation context
    • Tool integration (search, file operations, time queries)
  3. Vector Store & Search (src/search/)

    • VectorStoreManager manages embeddings and semantic search
    • ChunkedStorage for efficient large document handling
    • Event-driven index updates via IndexManager
    • Multiple embedding providers support
  4. UI Component System (src/components/)

    • React functional components with Radix UI primitives
    • Tailwind CSS with class variance authority (CVA)
    • Modal system for user interactions
    • Chat interface with streaming support
    • Settings UI with versioned components
  5. Message Management Architecture (src/core/, src/state/)

    • MessageRepository (src/core/MessageRepository.ts): Single source of truth for all messages
      • Stores each message once with both displayText and processedText
      • Provides computed views for UI display and LLM processing
      • No complex dual-array synchronization
    • ChatManager (src/core/ChatManager.ts): Central business logic coordinator
      • Orchestrates MessageRepository, ContextManager, and LLM operations
      • Handles message sending, editing, regeneration, and deletion
      • Manages context processing and chain memory synchronization
      • Project Chat Isolation: Maintains separate MessageRepository per project
        • Automatically detects project switches via getCurrentMessageRepo()
        • Each project has its own isolated message history
        • Non-project chats use defaultProjectKey repository
    • ChatUIState (src/state/ChatUIState.ts): Clean UI-only state manager
      • Delegates all business logic to ChatManager
      • Provides React integration with subscription mechanism
      • Replaces legacy SharedState with minimal, focused approach
    • ContextManager (src/core/ContextManager.ts): Handles context processing
      • Processes message context (notes, URLs, selected text)
      • Reprocesses context when messages are edited
  6. Settings Management

    • Jotai for atomic settings state management
    • React contexts for feature-specific state
  7. Plugin Integration

    • Main entry: src/main.ts extends Obsidian Plugin
    • Command registration system
    • Event handling for Obsidian lifecycle
    • Settings persistence and migration
    • Chat history loading via pending message mechanism

Key Patterns

  • Single Source of Truth: MessageRepository stores each message once with computed views
  • Clean Architecture: Repository → Manager → UIState → React Components
  • Context Reprocessing: Automatic context updates when messages are edited
  • Computed Views: Display messages for UI, LLM messages for AI processing
  • Project Isolation: Each project maintains its own MessageRepository instance
  • Error Handling: Custom error types with detailed interfaces
  • Async Operations: Consistent async/await pattern with proper error boundaries
  • Caching: Multi-layer caching for files, PDFs, and API responses
  • Streaming: Real-time streaming for LLM responses
  • Testing: Unit tests adjacent to implementation, integration tests for API calls

Message Management Architecture

For detailed architecture diagrams and documentation, see MESSAGE_ARCHITECTURE.md.

Core Classes and Flow

  1. MessageRepository (src/core/MessageRepository.ts)

    • Single source of truth for all messages
    • Stores StoredMessage objects with both displayText and processedText
    • Provides computed views via getDisplayMessages() and getLLMMessages()
    • No complex dual-array synchronization or ID matching
  2. ChatManager (src/core/ChatManager.ts)

    • Central business logic coordinator
    • Orchestrates MessageRepository, ContextManager, and LLM operations
    • Handles all message CRUD operations with proper error handling
    • Synchronizes with chain memory for conversation history
    • Project Chat Isolation Implementation:
      • Maintains projectMessageRepos: Map<string, MessageRepository> for project-specific storage
      • getCurrentMessageRepo() automatically detects current project and returns correct repository
      • Seamlessly switches between project repositories when project changes
      • Creates new empty repository for each project (no message caching)
  3. ChatUIState (src/state/ChatUIState.ts)

    • Clean UI-only state manager
    • Delegates all business logic to ChatManager
    • Provides React integration with subscription mechanism
    • Replaces legacy SharedState with minimal, focused approach
  4. ContextManager (src/core/ContextManager.ts)

    • Handles context processing (notes, URLs, selected text)
    • Reprocesses context when messages are edited
    • Ensures fresh context for LLM processing
  5. ChatPersistenceManager (src/core/ChatPersistenceManager.ts)

    • Handles saving and loading chat history to/from markdown files
    • Project-aware file naming (prefixes with project ID)
    • Parses and formats chat content for storage
    • Integrated with ChatManager for seamless persistence

Code Style Guidelines

MAJOR PRINCIPLES

  • ALWAYS WRITE GENERALIZABLE SOLUTIONS: Never add edge-case handling or hardcoded logic for specific scenarios (like "piano notes" or "daily notes"). Solutions must work for all cases.
  • Avoid hardcoding: No hardcoded folder names, file patterns, or special-case logic
  • Configuration over convention: If behavior needs to vary, make it configurable, not hardcoded
  • Universal patterns: Solutions should work equally well for any folder structure, naming convention, or content type

TypeScript

  • Strict mode enabled (no implicit any, strict null checks)
  • Use absolute imports with @/ prefix: import { ChainType } from "@/chainFactory"
  • Prefer const assertions and type inference where appropriate
  • Use interface for object shapes, type for unions/aliases

React

  • Functional components only (no class components)
  • Custom hooks for reusable logic
  • Props interfaces defined above components
  • Avoid inline styles, use Tailwind classes

General

  • File naming: PascalCase for components, camelCase for utilities
  • Async/await over promises
  • Early returns for error conditions
  • Always add JSDoc comments for all functions and methods
  • Organize imports: React → external → internal
  • Avoid language-specific lists (like stopwords or action verbs) - use language-agnostic approaches instead

Logging

  • NEVER use console.log - Use the logging utilities instead:
    • logInfo() for informational messages
    • logWarn() for warnings
    • logError() for errors
  • Import from logger: import { logInfo, logWarn, logError } from "@/logger"

Testing Guidelines

  • Unit tests use Jest with TypeScript support
  • Mock Obsidian API for plugin testing
  • Integration tests require API keys in .env.test
  • Test files adjacent to implementation (.test.ts)
  • Use @testing-library/react for component testing

Development Session Planning

Using TODO.md for Session Management

IMPORTANT: When working on a development session, maintain a comprehensive TODO.md file that serves as the central plan and tracker:

  1. Session Goal: Define the high-level objective at the start
  2. Task Tracking:
    • List all completed tasks with [x] checkboxes
    • Track pending tasks with [ ] checkboxes
    • Group related tasks into logical sections
  3. Architecture Decisions: Document key design choices and rationale
  4. Progress Updates: Keep the TODO.md updated as tasks complete
  5. Testing Checklist: Include verification steps for the session

The TODO.md should be:

  • The single source of truth for session progress
  • Updated frequently as work progresses
  • Clear enough that another developer can understand what was done
  • Comprehensive enough to serve as a migration guide

Structure Example:

# Development Session TODO

## Session Goal

[Clear statement of what this session aims to achieve]

## Completed Tasks ✅

- [x] Task description with key details
- [x] Another completed task

## Pending Tasks 📋

- [ ] Next task to work on
- [ ] Future enhancement

## Architecture Summary

[Key design decisions and rationale]

## Testing Checklist

- [ ] Functionality verification
- [ ] Performance checks

Important Notes

  • The plugin supports multiple LLM providers with custom endpoints
  • Vector store requires rebuilding when switching embedding providers
  • Settings are versioned - migrations may be needed
  • Local model support available via Ollama/LM Studio
  • Rate limiting is implemented for all API calls
  • For technical debt and known issues, see TECHDEBT.md
  • For current development session planning, see TODO.md

Obsidian Plugin Environment

  • Global app variable: In Obsidian plugins, app is a globally available variable that provides access to the Obsidian API. It's automatically available in all files without needing to import or declare it.

Architecture Migration Notes

  • SharedState Removed: The legacy src/sharedState.ts has been completely removed
  • Clean Architecture: New architecture follows Repository → Manager → UIState → UI pattern
  • Single Source of Truth: All messages stored once in MessageRepository with computed views
  • Context Always Fresh: Context is reprocessed when messages are edited to ensure accuracy
  • Chat History Loading: Uses pending message mechanism through CopilotView → Chat component props
  • Project Chat Isolation: Each project now has completely isolated chat history
    • Automatic detection of project switches via ProjectManager.getCurrentProjectId()
    • Separate MessageRepository instances per project ID
    • Non-project chats stored in default repository
    • Backwards compatible - loads existing messages from ProjectManager cache
    • Zero configuration required - works automatically