mirror of
https://github.com/logancyang/obsidian-copilot.git
synced 2026-07-22 07:50:24 +00:00
92d6f98af2
* Implement inline citation for vault search for both vault QA and Plus chains refactor: enhance citation utilities and improve source handling in chain runners - Introduced new utility functions for citation management, including sanitization and formatting of source catalogs. - Updated CopilotPlusChainRunner and VaultQAChainRunner to utilize the new citation utilities for improved citation handling. - Enhanced ChatSingleMessage component to consolidate duplicate sources and ensure accurate citation numbering. - Added comprehensive tests for citation utilities to ensure functionality and edge case handling. refactor: dedupe chunks for unique titles in search tool sources refactor: enhance citation handling in CopilotPlus and VaultQA chain runners - Introduced a sanitization function to remove pre-existing citation markers from document content, preventing number leakage. - Updated the formatting of retrieved documents in VaultQA to utilize the new sanitization function for cleaner output. - Improved footnote renumbering logic in ChatSingleMessage to prioritize first-mention order, ensuring accurate citation mapping. refactor: streamline source citation handling in ChatSingleMessage component and enhance web search tool instructions - Simplified the processing of source links in ChatSingleMessage to use a more efficient mapping approach. - Updated the web search tool's instruction to clarify the use of footnote-style citations, ensuring compatibility with Markdown formatting. * feat: add inline citation setting toggle and refactor - Introduced a new setting to enable inline citations in AI-generated responses, allowing for footnote-style references within the text. - Updated citation utilities to manage fallback sources and improve citation instructions based on user settings. - Enhanced CopilotPlusChainRunner and VaultQAChainRunner to utilize the new inline citation feature, ensuring accurate source representation. - Modified ChatSingleMessage component to process inline citations based on the new setting, improving the display of sources in chat messages. - Added tests for new citation functionalities to ensure reliability and correctness. * feat: enhance citation utilities and add tests for normalization - Introduced the `normalizeCitations` function to improve citation formatting by removing periods after citations and handling footnote references. - Updated `processInlineCitations` to ensure correct mapping of non-sequential citations and consolidate duplicate sources. - Added comprehensive tests for the new citation functionalities, ensuring accurate citation processing and representation in AI-generated responses. - Modified QASettings description for clarity regarding the inline citations feature. * fix: address critical bug in citation processing for consecutive citations - Enhanced the `normalizeCitations` function to correctly handle consecutive citations without spaces, ensuring accurate mapping and representation in output. - Added a test case to replicate and validate the fix for the identified bug, confirming that both citations are processed and displayed correctly. - Updated existing citation handling logic to improve robustness and prevent unconverted citations in the output. * refactor: improve citation handling and enhance citation utilities - Refactored `CopilotPlusChainRunner` to streamline citation source management, replacing `lastCitationLines` with `lastCitationSources` for better clarity and structure. - Updated the `addFallbackSources` function to handle citation formatting more effectively, ensuring proper display of titles and paths. - Enhanced the `normalizeCitations` function to support multiple citations in brackets, ensuring correct sorting and mapping of citation numbers. - Added new test cases to validate the handling of alternate headings, HTML summaries, and multiple citations, improving overall robustness of citation processing. - Introduced `getWebSearchCitationInstructions` for web-specific citation guidance, enhancing user experience with clear instructions for inline citations. * fix: move documentations * refactor: normalize footnote rendering in ChatSingleMessage component - Introduced `normalizeFootnoteRendering` function to streamline footnote display by removing separators and backreferences, and fixing numbering artifacts. - Updated `ChatSingleMessage` to utilize the new normalization function, enhancing the clarity and presentation of footnotes in chat messages. - Added unit tests for `normalizeFootnoteRendering` to ensure proper functionality and handling of various footnote scenarios. * test: enhance ChatSingleMessage tests and add deduplication tests for search tools - Updated `ChatSingleMessage.test.tsx` to include `TooltipProvider` for improved rendering context. - Refactored mock implementation for `obsidian` to streamline testing setup. - Introduced new test file `SearchTools.dedupe.test.ts` to validate the deduplication logic in search tool sources, ensuring correct handling of duplicate entries while preserving the highest score. - Added tests to verify deduplication behavior for various scenarios, enhancing overall test coverage and reliability.
51 lines
2.3 KiB
Markdown
51 lines
2.3 KiB
Markdown
# TODO - Technical Debt & Future Improvements
|
|
|
|
This document tracks technical debt items and improvements that need to be addressed in the future.
|
|
|
|
## 1. Docs4LLM SSL Error in Projects Mode
|
|
|
|
### Issue Description
|
|
|
|
Document parsing for projects mode is failing with an SSL error (`net::ERR_SSL_BAD_RECORD_MAC_ALERT`) when trying to upload files to the docs4llm API endpoint.
|
|
|
|
### Technical Details
|
|
|
|
- **Error Location**: `src/LLMProviders/brevilabsClient.ts:185` in `makeFormDataRequest` method
|
|
- **Root Cause**: The method uses native `fetch` API instead of Obsidian's `requestUrl` API
|
|
- **Context**: While regular JSON requests were migrated to use `safeFetch` (which uses `requestUrl`) in commit e49aafa to fix CORS issues, the `makeFormDataRequest` method was not updated
|
|
|
|
### Why Current Approaches Won't Work
|
|
|
|
1. **Backend Constraint**: The `/docs4llm` endpoint only accepts multipart/form-data format with `files: List[UploadFile]`
|
|
2. **Obsidian Limitation**: The existing `safeFetch` function is hardcoded for `application/json` content type
|
|
3. **No JSON Alternative**: Unlike `pdf4llm` which accepts base64 JSON, there's no JSON endpoint for `docs4llm`
|
|
|
|
### Recommended Solution
|
|
|
|
Create a new `safeFetchFormData` function that:
|
|
|
|
1. Uses Obsidian's `requestUrl` API with proper multipart/form-data configuration
|
|
2. Handles FormData objects correctly
|
|
3. Bypasses CORS and SSL restrictions like `safeFetch` does for JSON
|
|
|
|
### Alternative Solutions
|
|
|
|
1. **Backend Modification**: Add a new `/docs4llm-base64` endpoint that accepts base64-encoded JSON payloads
|
|
2. **Research Obsidian API**: Investigate if newer versions of Obsidian's `requestUrl` support multipart/form-data
|
|
3. **SSL Certificate Fix**: Address the underlying SSL certificate issue (temporary workaround)
|
|
|
|
### Impact
|
|
|
|
- Users cannot parse non-markdown files (PDFs, Word docs, etc.) in projects mode
|
|
- This affects the core functionality of project context loading
|
|
- Workaround: Users must ensure their projects only contain markdown files
|
|
|
|
### References
|
|
|
|
- Related commit: e49aafa (Brevilabs CORS issue #918)
|
|
- Forum discussion: https://forum.obsidian.md/t/holo-how-to-add-a-png-image-or-file-to-formdata-in-obsidian-like-below-this-help/73420
|
|
- Backend implementation: `/Users/chaoyang/webapps/brevilabs-api/app/main.py:1039`
|
|
|
|
---
|
|
|
|
_Last updated: 2025-07-18_
|