mirror of
https://github.com/logancyang/obsidian-copilot.git
synced 2026-07-22 07:50:24 +00:00
92d6f98af2
* Implement inline citation for vault search for both vault QA and Plus chains refactor: enhance citation utilities and improve source handling in chain runners - Introduced new utility functions for citation management, including sanitization and formatting of source catalogs. - Updated CopilotPlusChainRunner and VaultQAChainRunner to utilize the new citation utilities for improved citation handling. - Enhanced ChatSingleMessage component to consolidate duplicate sources and ensure accurate citation numbering. - Added comprehensive tests for citation utilities to ensure functionality and edge case handling. refactor: dedupe chunks for unique titles in search tool sources refactor: enhance citation handling in CopilotPlus and VaultQA chain runners - Introduced a sanitization function to remove pre-existing citation markers from document content, preventing number leakage. - Updated the formatting of retrieved documents in VaultQA to utilize the new sanitization function for cleaner output. - Improved footnote renumbering logic in ChatSingleMessage to prioritize first-mention order, ensuring accurate citation mapping. refactor: streamline source citation handling in ChatSingleMessage component and enhance web search tool instructions - Simplified the processing of source links in ChatSingleMessage to use a more efficient mapping approach. - Updated the web search tool's instruction to clarify the use of footnote-style citations, ensuring compatibility with Markdown formatting. * feat: add inline citation setting toggle and refactor - Introduced a new setting to enable inline citations in AI-generated responses, allowing for footnote-style references within the text. - Updated citation utilities to manage fallback sources and improve citation instructions based on user settings. - Enhanced CopilotPlusChainRunner and VaultQAChainRunner to utilize the new inline citation feature, ensuring accurate source representation. - Modified ChatSingleMessage component to process inline citations based on the new setting, improving the display of sources in chat messages. - Added tests for new citation functionalities to ensure reliability and correctness. * feat: enhance citation utilities and add tests for normalization - Introduced the `normalizeCitations` function to improve citation formatting by removing periods after citations and handling footnote references. - Updated `processInlineCitations` to ensure correct mapping of non-sequential citations and consolidate duplicate sources. - Added comprehensive tests for the new citation functionalities, ensuring accurate citation processing and representation in AI-generated responses. - Modified QASettings description for clarity regarding the inline citations feature. * fix: address critical bug in citation processing for consecutive citations - Enhanced the `normalizeCitations` function to correctly handle consecutive citations without spaces, ensuring accurate mapping and representation in output. - Added a test case to replicate and validate the fix for the identified bug, confirming that both citations are processed and displayed correctly. - Updated existing citation handling logic to improve robustness and prevent unconverted citations in the output. * refactor: improve citation handling and enhance citation utilities - Refactored `CopilotPlusChainRunner` to streamline citation source management, replacing `lastCitationLines` with `lastCitationSources` for better clarity and structure. - Updated the `addFallbackSources` function to handle citation formatting more effectively, ensuring proper display of titles and paths. - Enhanced the `normalizeCitations` function to support multiple citations in brackets, ensuring correct sorting and mapping of citation numbers. - Added new test cases to validate the handling of alternate headings, HTML summaries, and multiple citations, improving overall robustness of citation processing. - Introduced `getWebSearchCitationInstructions` for web-specific citation guidance, enhancing user experience with clear instructions for inline citations. * fix: move documentations * refactor: normalize footnote rendering in ChatSingleMessage component - Introduced `normalizeFootnoteRendering` function to streamline footnote display by removing separators and backreferences, and fixing numbering artifacts. - Updated `ChatSingleMessage` to utilize the new normalization function, enhancing the clarity and presentation of footnotes in chat messages. - Added unit tests for `normalizeFootnoteRendering` to ensure proper functionality and handling of various footnote scenarios. * test: enhance ChatSingleMessage tests and add deduplication tests for search tools - Updated `ChatSingleMessage.test.tsx` to include `TooltipProvider` for improved rendering context. - Refactored mock implementation for `obsidian` to streamline testing setup. - Introduced new test file `SearchTools.dedupe.test.ts` to validate the deduplication logic in search tool sources, ensuring correct handling of duplicate entries while preserving the highest score. - Added tests to verify deduplication behavior for various scenarios, enhancing overall test coverage and reliability.
48 lines
4.1 KiB
Markdown
48 lines
4.1 KiB
Markdown
# Inline Citation System
|
|
|
|
This guide explains how inline citations are produced across Copilot Plus, Vault QA, and web search, and how the feature is exercised by automated tests.
|
|
|
|
## Feature Toggle & Surface Area
|
|
|
|
- `enableInlineCitations` (default `true`) lives in `src/settings/model.ts` and is exposed in the QA settings UI (`src/settings/v2/components/QASettings.tsx`).
|
|
- The toggle gates prompt instructions, fallback post-processing, and chat rendering. When disabled the system falls back to a collapsible sources list without inline markers.
|
|
|
|
## Pipeline Overview
|
|
|
|
1. **Retrieval Conditioning**
|
|
- Both `CopilotPlusChainRunner.prepareLocalSearchResult` and `VaultQAChainRunner` sanitize note content with `sanitizeContentForCitations` to strip stray `[^n]`/`[n]` markers before prompting.
|
|
- Retrieved notes receive stable `__sourceId` values and are serialized with `formatSearchResultsForLLM`; `deduplicateSources` keeps the highest-scoring entry per path/title.
|
|
- A compact source catalog is built via `formatSourceCatalog`, and Copilot Plus caches the first 20 entries in `lastCitationSources` for fallback footnotes.
|
|
2. **Prompt Assembly**
|
|
- `CITATION_RULES` and `WEB_CITATION_RULES` live in `src/LLMProviders/chainRunner/utils/citationUtils.ts`.
|
|
- `getCitationInstructions` (Copilot Plus) and `getQACitationInstructionsConditional` (Vault QA) append guidance and a source catalog only when inline citations are enabled.
|
|
- Web search calls `getWebSearchCitationInstructions` so external sources emit `[title](url)` definitions while vault answers stay on `[[Note]]` links.
|
|
3. **Response Safeguards**
|
|
- `addFallbackSources` appends a `#### Sources` block when the model produced inline markers but no definitions. Detection relies on `hasExistingCitations`, which now accepts alternate headings (e.g., `## Sources`, `Sources -`) and `<summary>Sources</summary>` blocks.
|
|
- Copilot Plus passes structured `lastCitationSources` into the fallback helper; Vault QA derives titles from the retriever output.
|
|
4. **Chat Rendering**
|
|
- `src/components/chat-components/ChatSingleMessage.tsx` always pipes assistant messages through `processInlineCitations`.
|
|
- The helper extracts the trailing sources section, builds a first-mention map with `buildCitationMap`, normalizes references (`normalizeCitations`) so constructs like `[^7][^8]` become `[1][2]`, and converts definitions (`convertFootnoteDefinitions`) into clickable wiki links or Markdown anchors.
|
|
- Duplicate definitions collapse via `consolidateDuplicateSources` + `updateCitationsForConsolidation`, keeping numbering stable. When the sources block is not footnote formatted or citations are disabled, the renderer falls back to a simple `<details>` list.
|
|
|
|
## Testing
|
|
|
|
- `src/LLMProviders/chainRunner/utils/citationUtils.test.ts`
|
|
- Sanitization, catalog formatting, and fallback insertion.
|
|
- `hasExistingCitations` coverage for markdown headings, plain `Sources` labels, and `<summary>` wrappers.
|
|
- Regression suites for non-sequential citations, duplicate source consolidation, and consecutive markers (`[^7][^8]`).
|
|
- `src/LLMProviders/chainRunner/utils/searchResultUtils.test.ts`
|
|
- Ensures retrieved documents are serialized with stable IDs and filtered for `includeInContext` before prompting.
|
|
- `src/tools/ToolResultFormatter.test.ts`
|
|
- Verifies the local search tool emits JSON with the `{ type: "local_search", documents: [...] }` shape expected by the chain runners.
|
|
|
|
## Manual QA Checklist
|
|
|
|
- Vault QA turn using only local search: confirm inline `[1]` markers and numbered sources render without duplication.
|
|
- Mixed Copilot Plus turn (local search + another tool): ensure fallback still works if the model omits the sources block.
|
|
- Web search answer: verify footnote definitions render as `[title](url)` links when citations are enabled.
|
|
|
|
## Watchlist
|
|
|
|
- `sanitizeContentForCitations` intentionally strips bracketed numbers; keep an eye on domains (math, law) where literal `[1990]` values might be desirable.
|
|
- Inline citations remain model-dependent. `addFallbackSources` guarantees a sources list, but the UI still reflects whatever inline markers the provider returns.
|