obsidian-plugin/logancyang_obsidian-copilot
1
0
Fork 0
mirror of https://github.com/logancyang/obsidian-copilot.git synced 2026-07-22 07:50:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
logancyang_obsidian-copilot/docs/TECHDEBT.md
Logan Yang 92d6f98af2
Implement inline citation (#1821) 
* Implement inline citation for vault search for both vault QA and Plus chains

refactor: enhance citation utilities and improve source handling in chain runners

- Introduced new utility functions for citation management, including sanitization and formatting of source catalogs.
- Updated CopilotPlusChainRunner and VaultQAChainRunner to utilize the new citation utilities for improved citation handling.
- Enhanced ChatSingleMessage component to consolidate duplicate sources and ensure accurate citation numbering.
- Added comprehensive tests for citation utilities to ensure functionality and edge case handling.

refactor: dedupe chunks for unique titles in search tool sources

refactor: enhance citation handling in CopilotPlus and VaultQA chain runners

- Introduced a sanitization function to remove pre-existing citation markers from document content, preventing number leakage.
- Updated the formatting of retrieved documents in VaultQA to utilize the new sanitization function for cleaner output.
- Improved footnote renumbering logic in ChatSingleMessage to prioritize first-mention order, ensuring accurate citation mapping.

refactor: streamline source citation handling in ChatSingleMessage component and enhance web search tool instructions

- Simplified the processing of source links in ChatSingleMessage to use a more efficient mapping approach.
- Updated the web search tool's instruction to clarify the use of footnote-style citations, ensuring compatibility with Markdown formatting.

* feat: add inline citation setting toggle and refactor

- Introduced a new setting to enable inline citations in AI-generated responses, allowing for footnote-style references within the text.
- Updated citation utilities to manage fallback sources and improve citation instructions based on user settings.
- Enhanced CopilotPlusChainRunner and VaultQAChainRunner to utilize the new inline citation feature, ensuring accurate source representation.
- Modified ChatSingleMessage component to process inline citations based on the new setting, improving the display of sources in chat messages.
- Added tests for new citation functionalities to ensure reliability and correctness.

* feat: enhance citation utilities and add tests for normalization

- Introduced the `normalizeCitations` function to improve citation formatting by removing periods after citations and handling footnote references.
- Updated `processInlineCitations` to ensure correct mapping of non-sequential citations and consolidate duplicate sources.
- Added comprehensive tests for the new citation functionalities, ensuring accurate citation processing and representation in AI-generated responses.
- Modified QASettings description for clarity regarding the inline citations feature.

* fix: address critical bug in citation processing for consecutive citations

- Enhanced the `normalizeCitations` function to correctly handle consecutive citations without spaces, ensuring accurate mapping and representation in output.
- Added a test case to replicate and validate the fix for the identified bug, confirming that both citations are processed and displayed correctly.
- Updated existing citation handling logic to improve robustness and prevent unconverted citations in the output.

* refactor: improve citation handling and enhance citation utilities

- Refactored `CopilotPlusChainRunner` to streamline citation source management, replacing `lastCitationLines` with `lastCitationSources` for better clarity and structure.
- Updated the `addFallbackSources` function to handle citation formatting more effectively, ensuring proper display of titles and paths.
- Enhanced the `normalizeCitations` function to support multiple citations in brackets, ensuring correct sorting and mapping of citation numbers.
- Added new test cases to validate the handling of alternate headings, HTML summaries, and multiple citations, improving overall robustness of citation processing.
- Introduced `getWebSearchCitationInstructions` for web-specific citation guidance, enhancing user experience with clear instructions for inline citations.

* fix: move documentations

* refactor: normalize footnote rendering in ChatSingleMessage component

- Introduced `normalizeFootnoteRendering` function to streamline footnote display by removing separators and backreferences, and fixing numbering artifacts.
- Updated `ChatSingleMessage` to utilize the new normalization function, enhancing the clarity and presentation of footnotes in chat messages.
- Added unit tests for `normalizeFootnoteRendering` to ensure proper functionality and handling of various footnote scenarios.

* test: enhance ChatSingleMessage tests and add deduplication tests for search tools

- Updated `ChatSingleMessage.test.tsx` to include `TooltipProvider` for improved rendering context.
- Refactored mock implementation for `obsidian` to streamline testing setup.
- Introduced new test file `SearchTools.dedupe.test.ts` to validate the deduplication logic in search tool sources, ensuring correct handling of duplicate entries while preserving the highest score.
- Added tests to verify deduplication behavior for various scenarios, enhancing overall test coverage and reliability.
2025-09-21 20:03:09 -07:00

2.3 KiB
Raw Blame History

TODO - Technical Debt & Future Improvements

This document tracks technical debt items and improvements that need to be addressed in the future.

1. Docs4LLM SSL Error in Projects Mode

Issue Description

Document parsing for projects mode is failing with an SSL error (net::ERR_SSL_BAD_RECORD_MAC_ALERT) when trying to upload files to the docs4llm API endpoint.

Technical Details

  • Error Location: src/LLMProviders/brevilabsClient.ts:185 in makeFormDataRequest method
  • Root Cause: The method uses native fetch API instead of Obsidian's requestUrl API
  • Context: While regular JSON requests were migrated to use safeFetch (which uses requestUrl) in commit e49aafa to fix CORS issues, the makeFormDataRequest method was not updated

Why Current Approaches Won't Work

  1. Backend Constraint: The /docs4llm endpoint only accepts multipart/form-data format with files: List[UploadFile]
  2. Obsidian Limitation: The existing safeFetch function is hardcoded for application/json content type
  3. No JSON Alternative: Unlike pdf4llm which accepts base64 JSON, there's no JSON endpoint for docs4llm

Recommended Solution

Create a new safeFetchFormData function that:

  1. Uses Obsidian's requestUrl API with proper multipart/form-data configuration
  2. Handles FormData objects correctly
  3. Bypasses CORS and SSL restrictions like safeFetch does for JSON

Alternative Solutions

  1. Backend Modification: Add a new /docs4llm-base64 endpoint that accepts base64-encoded JSON payloads
  2. Research Obsidian API: Investigate if newer versions of Obsidian's requestUrl support multipart/form-data
  3. SSL Certificate Fix: Address the underlying SSL certificate issue (temporary workaround)

Impact

  • Users cannot parse non-markdown files (PDFs, Word docs, etc.) in projects mode
  • This affects the core functionality of project context loading
  • Workaround: Users must ensure their projects only contain markdown files

References

Last updated: 2025-07-18