obsidian-plugin/logancyang_obsidian-copilot
1
0
Fork 0
mirror of https://github.com/logancyang/obsidian-copilot.git synced 2026-07-22 07:50:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
logancyang_obsidian-copilot/TECHDEBT.md
Logan Yang df3efaecc0
Implement vault search v3 (#1703) 
* Add LexicalEngine and related interfaces for enhanced search functionality

- Implement LexicalEngine class utilizing FlexSearch for efficient document indexing and searching.
- Create Hit, SearchResult, SearchOptions, and RetrieverEngine interfaces to standardize search operations.
- Update package.json and package-lock.json to include new dependencies: flexsearch and p-queue.

* Add QueryExpander class and tests for enhanced search query expansion

* Add README for Tiered Note-Level Lexical Retrieval with multilingual support and enhanced search pipeline

* Implement v3 tiered search with GrepScanner, FullTextEngine, and GraphExpander

- Add GrepScanner for fast substring search (L0)
- Implement FullTextEngine with ephemeral FlexSearch index (L1)
- Add GraphExpander for link-based candidate expansion
- Create MemoryManager with platform-aware limits
- Implement weighted RRF fusion for result combination
- Add SemanticReranker placeholder for future integration
- Include comprehensive tests for core components
- Simplify code based on review: use TextEncoder, extract methods, streamline RRF
- Add clear logging for debugging search pipeline

The architecture follows a tiered approach:
1. Grep scan for initial candidates
2. Graph expansion to increase recall
3. Full-text search on expanded set
4. Optional semantic reranking
5. RRF fusion to combine signals

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Enhance tiered retrieval recall with both rewritten queries and the expanded salient terms

- Added detailed example of end-to-end query processing in README.md
- Updated TieredRetriever to log salient terms during query expansion
- Modified FullTextEngine to index link basenames for improved searchability
- Adjusted NoteDoc interface to clarify link handling and indexing

* Refactor QueryExpander configuration and caching logic for improved clarity and performance; enhance GraphExpander documentation and testing; remove unused fields from NoteDoc interface.

* Refactor search retrieval system to wire in TieredLexicalRetriever

- Replaced HybridRetriever with TieredLexicalRetriever in ChainManager, VaultQAChainRunner, and main.ts for improved multi-stage retrieval.
- Updated README.md to reflect new integration tasks and performance benchmarks.
- Introduced TieredLexicalRetriever class to handle on-demand indexing and retrieval.
- Enhanced GrepScanner to support inclusion/exclusion patterns for file indexing.
- Modified TieredRetriever to combine expanded salient terms and improved logging for final results.
- Removed legacy index management in favor of ephemeral indexing with TieredLexicalRetriever.
- Updated interfaces and utility functions to support new retrieval logic.

* feat: Implement first working TieredLexicalRetriever and refactor search scoring

- Added comprehensive tests for TieredLexicalRetriever, covering folder boosting and result combination.
- Refactored TieredLexicalRetriever to utilize SearchCore instead of TieredRetriever.
- Enhanced FullTextEngine with improved scoring mechanisms, including field weighting and multi-field match bonuses.
- Introduced FuzzyMatcher utility for fuzzy matching capabilities, including Levenshtein distance and variant generation.
- Updated GrepScanner to prioritize path matching for faster search results.
- Implemented normalized scoring in RRF for better ranking consistency.

* feat: Update vault search result display to show snippet of content instead of full text

* feat: Enhance FullTextEngine to index frontmatter property values and improve search scoring

* feat: Improve scoring and logging

* feat: Refactor TieredLexicalRetriever to support time-based queries and improve document retrieval logic

* Remove TODO.md from tracking and add to .gitignore

- TODO.md is now a local-only development session tracker
- Prevents accidental commits of work-in-progress task lists
- Each developer can maintain their own TODO.md without conflicts

* feat: Update dependencies and enhance search functionality

- Upgraded axios to version 1.11.0 and electron to version 27.3.11 for improved performance and security.
- Refactored QueryExpander to limit queries to the original for strict fallback tests.
- Enhanced SearchCore to rank grep hits by evidence quality before fusion, improving retrieval accuracy.
- Implemented a new method in SearchCore to rank grep hits based on evidence strength.
- Updated TieredLexicalRetriever tests to ensure proper integration with mocked SearchCore.
- Improved FuzzyMatcher to include plural/singular normalization for better fuzzy matching.

* feat: Add tool call marker encoding/decoding

- Introduced new utility functions for encoding and decoding tool call markers to ensure safe embedding in HTML comments.
- Updated `updateChatMemory` to handle encoded tool call markers, preserving their integrity during memory storage.
- Enhanced logging to decode tool marker results for better readability while maintaining encoded formats for storage.
- Added comprehensive tests for tool call marker functionality to ensure correct encoding and decoding behavior.
- Refactored related components to integrate the new encoding/decoding logic seamlessly.

* feat: Enhance FullTextEngine search to support low-weight terms and improve scoring

- Updated FullTextEngine to accept low-weight terms in the search method, allowing for better handling of salient terms.
- Implemented downweighting for boolean and numeric tokens in the properties field to reduce noise in search results.
- Added a new test case to validate the downweighting behavior for boolean and numeric queries in FullTextEngine.
- Improved logging for full-text search results to provide clearer insights into the search process.

* feat: Update GraphExpander to enhance search recall with adaptive hop logic

- Implemented guardrails in GraphExpander to adjust hop depth based on the number of grep hits: allows +1 hop for small sets (<5) and limits to 1 hop for large sets (≥50).
- Updated README.md to reflect new guardrail logic and scoring normalization in search results.
- Enhanced test cases to validate the new behavior of skipping co-citations for large result sets and allowing additional hops for smaller sets.

* feat: Filter out background tools during streaming to enhance user experience

- Added logic to determine and filter out background tools, preventing their names from appearing in the tool call display during streaming.
- Implemented a mechanism to include partial tool names only if they meet a specified length threshold, improving clarity in tool call presentations.

* feat: Enhance search tools to support time-based queries and display modified time

- Updated localSearchTool to ensure a healthy cap on max source chunks for time-based queries, improving recall.
- Modified ToolResultFormatter to display actual modified time for time-filtered results, enhancing clarity in search results.
- Adjusted output formatting to differentiate between recency and relevance scores based on query type.

* feat: Introduce semantic search capabilities with Memory Index support

- Added `enableSemanticSearchV3` setting to control the new semantic search feature.
- Implemented `MemoryIndexManager` for managing in-memory vector indexing with JSONL persistence.
- Enhanced `SearchCore` to utilize semantic retrieval based on the new memory index.
- Introduced command to build the semantic memory index, improving search accuracy and retrieval efficiency.
- Updated relevant components to integrate semantic search functionality seamlessly.

* feat: Add unit tests for MemoryIndexManager to validate functionality

- Introduced comprehensive tests for the MemoryIndexManager, covering scenarios such as loading from a file, building a vector store, and indexing vault contents.
- Implemented a mock embeddings API to facilitate testing without external dependencies.
- Added a utility function to reset the MemoryIndexManager state between tests, ensuring isolation and reliability of test outcomes.

* feat: Enhance MemoryIndexManager and related components for improved semantic indexing

- Introduced incremental indexing capabilities in MemoryIndexManager to update the JSONL index with new or modified files, enhancing efficiency.
- Updated commands to utilize the new incremental indexing method, providing users with real-time updates to the semantic memory index.
- Refactored existing indexing logic to support partitioned writes, improving performance and manageability of large datasets.
- Enhanced user notifications during indexing processes to provide better feedback on progress and status.
- Added a setting to enable semantic search, allowing users to blend semantic similarity into search results seamlessly.

* feat: Refine scoring display and enhance search result handling

- Updated SourcesModal to display relevance scores with four decimal places for consistency with SearchCore logs.
- Enhanced TieredLexicalRetriever to include a new `rerank_score` field for better score management and consistency across search results.
- Modified the combineResults method to ensure that title matches retain their original score while incorporating a fused score as `rerank_score`.
- Adjusted formatting in SearchTools to ensure both `score` and `rerank_score` reflect the same final score when present.
- Updated ToolResultFormatter to display scores with four decimal places, improving clarity in search result presentations.

* refactor: Simplify relevant notes retrieval by removing VectorStoreManager dependency

- Removed the use of VectorStoreManager in the relevant notes fetching logic, streamlining the process.
- Integrated MemoryIndexManager for improved handling of in-memory indexing and note retrieval.
- Enhanced error handling to ensure relevant notes are only fetched when the embedding index is available.
- Updated related functions to utilize the new memory index approach, improving performance and reliability.

* refactor: Remove VectorStoreManager dependency and streamline indexing logic

- Eliminated the VectorStoreManager from various components, transitioning to MemoryIndexManager for indexing and retrieval.
- Updated project and chain managers to remove unnecessary dependencies, enhancing code clarity and maintainability.
- Improved error handling and indexing conditions, particularly for mobile users, ensuring a more robust initialization process.
- Refactored settings components to utilize the new memory indexing approach, simplifying the user experience and reducing legacy code.

* chore: Mark legacy components as deprecated in preparation for v3 transition

- Annotated various files including DebugSearchModal, OramaSearchModal, chunkedStorage, dbOperations, hybridRetriever, and vectorStoreManager as deprecated, indicating they are obsolete in v3.
- Updated README.md to reflect the current implementation status and migration notes, emphasizing the removal of Orama-based modules and the transition to MemoryIndexManager for indexing and retrieval.

* feat: Implement file tracking and reindexing for modified files

- Introduced a new FileTrackingState interface to manage the last active file and its modification time.
- Enhanced the CopilotPlugin to opportunistically reindex the previous active file if it was modified while active, contingent on semantic search settings.
- Updated MemoryIndexManager to support reindexing of single modified files, improving efficiency in handling changes.
- Added unit tests for reindexSingleFileIfModified to ensure correct functionality and performance.

* feat: Add graph hops setting for enhanced search result expansion

- Introduced a new `graphHops` setting in `CopilotSettings` to control the number of hops for graph expansion during search, with a default value of 1 and a range of 1-3.
- Updated the `sanitizeSettings` function to validate the `graphHops` value.
- Enhanced the `SearchCore` and `TieredLexicalRetriever` classes to utilize the `graphHops` setting for improved search result relevance.
- Updated documentation in `README.md` to reflect the new feature and its security optimizations.

* refactor: Improve error handling and indexing logic in MemoryIndexManager and related components

- Replaced console error logging with structured logging using logError and logWarn for better error tracking.
- Enhanced the refresh and reindexing functions to ensure proper user notifications and error handling.
- Updated the logic for managing indexed files, including handling exclusions and ensuring accurate reporting of indexed and unindexed files.
- Implemented rate limiting in the MemoryIndexManager to optimize embedding requests and prevent overloading the service.
- Refactored chunk processing to improve efficiency and clarity in the indexing workflow.

* chore: Update README.md to reflect final session completion and key fixes

- Expanded the final session summary with a date and detailed list of completed features and fixes, including UI enhancements and improved logging practices.
- Clarified the status of deferred features and provided migration notes for better user guidance.
- Ensured documentation aligns with the latest implementation changes and optimizations.

* chore: Update memory limits in README.md and MemoryManager.ts for improved performance

- Adjusted memory limits for mobile and desktop platforms in both README.md and MemoryManager.ts to reflect increased capacity (20MB mobile, 100MB desktop).
- Ensured documentation aligns with the latest implementation changes for better clarity on resource management.

* feat: Integrate HyDE document generation into SearchCore for enhanced semantic search

- Implemented a new method to generate hypothetical documents (HyDE) to improve semantic search capabilities.
- Added timeout handling for HyDE generation to ensure graceful error management.
- Updated SearchCore to utilize HyDE documents in search queries, enhancing the relevance of results when semantic search is enabled.
- Introduced unit tests to validate the integration and functionality of HyDE generation within the search process.

* refactor: Enhance MemoryIndexManager with public methods for better indexing management

- Introduced public methods in MemoryIndexManager to streamline access to indexed file paths, check if a file is indexed, retrieve embeddings, and clear the index.
- Updated related components to utilize these new methods, improving code clarity and reducing direct access to internal properties.
- Enhanced error handling and user notifications during memory index operations.

* feat: Implement indexing notification and progress management

- Introduced the IndexingNotificationManager to handle UI notifications during indexing operations, allowing users to pause or stop the process.
- Added the IndexingProgressTracker to track progress across multiple files, enhancing user feedback during indexing.
- Developed the IndexingPipeline to manage the chunking and processing of files into embeddings, improving the overall indexing workflow.
- Created the IndexPersistenceManager for managing the persistence of indexed data, ensuring efficient storage and retrieval of index records.
- Refactored MemoryIndexManager to utilize the new components, streamlining the indexing process and improving code organization.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-08-11 16:58:04 -07:00

2.3 KiB
Raw Permalink Blame History

TODO - Technical Debt & Future Improvements

This document tracks technical debt items and improvements that need to be addressed in the future.

1. Docs4LLM SSL Error in Projects Mode

Issue Description

Document parsing for projects mode is failing with an SSL error (net::ERR_SSL_BAD_RECORD_MAC_ALERT) when trying to upload files to the docs4llm API endpoint.

Technical Details

  • Error Location: src/LLMProviders/brevilabsClient.ts:185 in makeFormDataRequest method
  • Root Cause: The method uses native fetch API instead of Obsidian's requestUrl API
  • Context: While regular JSON requests were migrated to use safeFetch (which uses requestUrl) in commit e49aafa to fix CORS issues, the makeFormDataRequest method was not updated

Why Current Approaches Won't Work

  1. Backend Constraint: The /docs4llm endpoint only accepts multipart/form-data format with files: List[UploadFile]
  2. Obsidian Limitation: The existing safeFetch function is hardcoded for application/json content type
  3. No JSON Alternative: Unlike pdf4llm which accepts base64 JSON, there's no JSON endpoint for docs4llm

Recommended Solution

Create a new safeFetchFormData function that:

  1. Uses Obsidian's requestUrl API with proper multipart/form-data configuration
  2. Handles FormData objects correctly
  3. Bypasses CORS and SSL restrictions like safeFetch does for JSON

Alternative Solutions

  1. Backend Modification: Add a new /docs4llm-base64 endpoint that accepts base64-encoded JSON payloads
  2. Research Obsidian API: Investigate if newer versions of Obsidian's requestUrl support multipart/form-data
  3. SSL Certificate Fix: Address the underlying SSL certificate issue (temporary workaround)

Impact

  • Users cannot parse non-markdown files (PDFs, Word docs, etc.) in projects mode
  • This affects the core functionality of project context loading
  • Workaround: Users must ensure their projects only contain markdown files

References

Last updated: 2025-07-18