obsidian-plugin/logancyang_obsidian-copilot

mirror of https://github.com/logancyang/obsidian-copilot.git synced 2026-07-22 07:50:24 +00:00

* Implement inline citation for vault search for both vault QA and Plus chains

refactor: enhance citation utilities and improve source handling in chain runners

- Introduced new utility functions for citation management, including sanitization and formatting of source catalogs.
- Updated CopilotPlusChainRunner and VaultQAChainRunner to utilize the new citation utilities for improved citation handling.
- Enhanced ChatSingleMessage component to consolidate duplicate sources and ensure accurate citation numbering.
- Added comprehensive tests for citation utilities to ensure functionality and edge case handling.

refactor: dedupe chunks for unique titles in search tool sources

refactor: enhance citation handling in CopilotPlus and VaultQA chain runners

- Introduced a sanitization function to remove pre-existing citation markers from document content, preventing number leakage.
- Updated the formatting of retrieved documents in VaultQA to utilize the new sanitization function for cleaner output.
- Improved footnote renumbering logic in ChatSingleMessage to prioritize first-mention order, ensuring accurate citation mapping.

refactor: streamline source citation handling in ChatSingleMessage component and enhance web search tool instructions

- Simplified the processing of source links in ChatSingleMessage to use a more efficient mapping approach.
- Updated the web search tool's instruction to clarify the use of footnote-style citations, ensuring compatibility with Markdown formatting.

* feat: add inline citation setting toggle and refactor

- Introduced a new setting to enable inline citations in AI-generated responses, allowing for footnote-style references within the text.
- Updated citation utilities to manage fallback sources and improve citation instructions based on user settings.
- Enhanced CopilotPlusChainRunner and VaultQAChainRunner to utilize the new inline citation feature, ensuring accurate source representation.
- Modified ChatSingleMessage component to process inline citations based on the new setting, improving the display of sources in chat messages.
- Added tests for new citation functionalities to ensure reliability and correctness.

* feat: enhance citation utilities and add tests for normalization

- Introduced the `normalizeCitations` function to improve citation formatting by removing periods after citations and handling footnote references.
- Updated `processInlineCitations` to ensure correct mapping of non-sequential citations and consolidate duplicate sources.
- Added comprehensive tests for the new citation functionalities, ensuring accurate citation processing and representation in AI-generated responses.
- Modified QASettings description for clarity regarding the inline citations feature.

* fix: address critical bug in citation processing for consecutive citations

- Enhanced the `normalizeCitations` function to correctly handle consecutive citations without spaces, ensuring accurate mapping and representation in output.
- Added a test case to replicate and validate the fix for the identified bug, confirming that both citations are processed and displayed correctly.
- Updated existing citation handling logic to improve robustness and prevent unconverted citations in the output.

* refactor: improve citation handling and enhance citation utilities

- Refactored `CopilotPlusChainRunner` to streamline citation source management, replacing `lastCitationLines` with `lastCitationSources` for better clarity and structure.
- Updated the `addFallbackSources` function to handle citation formatting more effectively, ensuring proper display of titles and paths.
- Enhanced the `normalizeCitations` function to support multiple citations in brackets, ensuring correct sorting and mapping of citation numbers.
- Added new test cases to validate the handling of alternate headings, HTML summaries, and multiple citations, improving overall robustness of citation processing.
- Introduced `getWebSearchCitationInstructions` for web-specific citation guidance, enhancing user experience with clear instructions for inline citations.

* fix: move documentations

* refactor: normalize footnote rendering in ChatSingleMessage component

- Introduced `normalizeFootnoteRendering` function to streamline footnote display by removing separators and backreferences, and fixing numbering artifacts.
- Updated `ChatSingleMessage` to utilize the new normalization function, enhancing the clarity and presentation of footnotes in chat messages.
- Added unit tests for `normalizeFootnoteRendering` to ensure proper functionality and handling of various footnote scenarios.

* test: enhance ChatSingleMessage tests and add deduplication tests for search tools

- Updated `ChatSingleMessage.test.tsx` to include `TooltipProvider` for improved rendering context.
- Refactored mock implementation for `obsidian` to streamline testing setup.
- Introduced new test file `SearchTools.dedupe.test.ts` to validate the deduplication logic in search tool sources, ensuring correct handling of duplicate entries while preserving the highest score.
- Added tests to verify deduplication behavior for various scenarios, enhancing overall test coverage and reliability.

2025-09-21 20:03:09 -07:00

4.1 KiB

Raw Blame History

Inline Citation System

This guide explains how inline citations are produced across Copilot Plus, Vault QA, and web search, and how the feature is exercised by automated tests.

Feature Toggle & Surface Area

enableInlineCitations (default true) lives in src/settings/model.ts and is exposed in the QA settings UI (src/settings/v2/components/QASettings.tsx).
The toggle gates prompt instructions, fallback post-processing, and chat rendering. When disabled the system falls back to a collapsible sources list without inline markers.

Pipeline Overview

Retrieval Conditioning
- Both CopilotPlusChainRunner.prepareLocalSearchResult and VaultQAChainRunner sanitize note content with sanitizeContentForCitations to strip stray [^n]/[n] markers before prompting.
- Retrieved notes receive stable __sourceId values and are serialized with formatSearchResultsForLLM; deduplicateSources keeps the highest-scoring entry per path/title.
- A compact source catalog is built via formatSourceCatalog, and Copilot Plus caches the first 20 entries in lastCitationSources for fallback footnotes.
Prompt Assembly
- CITATION_RULES and WEB_CITATION_RULES live in src/LLMProviders/chainRunner/utils/citationUtils.ts.
- getCitationInstructions (Copilot Plus) and getQACitationInstructionsConditional (Vault QA) append guidance and a source catalog only when inline citations are enabled.
- Web search calls getWebSearchCitationInstructions so external sources emit [title](url) definitions while vault answers stay on [[Note]] links.
Response Safeguards
- addFallbackSources appends a #### Sources block when the model produced inline markers but no definitions. Detection relies on hasExistingCitations, which now accepts alternate headings (e.g., ## Sources, Sources -) and <summary>Sources</summary> blocks.
- Copilot Plus passes structured lastCitationSources into the fallback helper; Vault QA derives titles from the retriever output.
Chat Rendering
- src/components/chat-components/ChatSingleMessage.tsx always pipes assistant messages through processInlineCitations.
- The helper extracts the trailing sources section, builds a first-mention map with buildCitationMap, normalizes references (normalizeCitations) so constructs like [^7][^8] become [1][2], and converts definitions (convertFootnoteDefinitions) into clickable wiki links or Markdown anchors.
- Duplicate definitions collapse via consolidateDuplicateSources + updateCitationsForConsolidation, keeping numbering stable. When the sources block is not footnote formatted or citations are disabled, the renderer falls back to a simple <details> list.

Testing

src/LLMProviders/chainRunner/utils/citationUtils.test.ts
- Sanitization, catalog formatting, and fallback insertion.
- hasExistingCitations coverage for markdown headings, plain Sources labels, and <summary> wrappers.
- Regression suites for non-sequential citations, duplicate source consolidation, and consecutive markers ([^7][^8]).
src/LLMProviders/chainRunner/utils/searchResultUtils.test.ts
- Ensures retrieved documents are serialized with stable IDs and filtered for includeInContext before prompting.
src/tools/ToolResultFormatter.test.ts
- Verifies the local search tool emits JSON with the { type: "local_search", documents: [...] } shape expected by the chain runners.

Manual QA Checklist

Vault QA turn using only local search: confirm inline [1] markers and numbered sources render without duplication.
Mixed Copilot Plus turn (local search + another tool): ensure fallback still works if the model omits the sources block.
Web search answer: verify footnote definitions render as [title](url) links when citations are enabled.

Watchlist

sanitizeContentForCitations intentionally strips bracketed numbers; keep an eye on domains (math, law) where literal [1990] values might be desirable.
Inline citations remain model-dependent. addFallbackSources guarantees a sources list, but the UI still reflects whatever inline markers the provider returns.

4.1 KiB Raw Blame History