logancyang_obsidian-copilot

847 commits 55 branches 115 tags 260 MiB

Author	SHA1	Message	Date
Logan Yang	05beb941eb	Context revamp (#1971 ) * Implement context engine for a layered context * Wire in new context library to LLM chain - Updated the context engine to maintain a cumulative library of all context items (L2) without deduplication, enhancing cache stability. - Introduced smart referencing in the user message (L3) to reference existing items in L2 by ID and include full content for new items. - Improved the LayerToMessagesConverter to format and handle layered context envelopes for better debugging and clarity. - Fixed critical bugs related to context deduplication that previously broke cache stability. - Enhanced tests to validate the new context handling and ensure proper functionality across various scenarios. * Enhance VaultQAChainRunner with tag extraction and inline citation detection - Integrated tag extraction from user queries to improve context-aware retrieval. - Implemented `hasInlineCitations` function to detect inline citation markers in responses. - Updated message construction logic to utilize envelope-based context, enhancing the quality of AI interactions. - Added comprehensive tests for the new inline citation detection functionality, ensuring robustness and accuracy. - Improved logging for better debugging and traceability of the message processing flow. Refactor VaultQAChainRunner to enhance context handling and citation instructions - Integrated LayerToMessagesConverter for improved message construction, ensuring smart referencing and context preservation. - Updated logging to reflect the use of the new converter in envelope-based context construction. - Revised comments for clarity on context preparation and citation instructions. - Minor adjustment in citation utility to improve regex handling for citation updates. * Complete migration to envelope-based context in CopilotPlus and LLM chain runners - Removed all legacy fallback paths, ensuring that context envelopes are now mandatory for operations in CopilotPlusChainRunner and LLMChainRunner. - Enhanced image extraction to only pull from the active note, preventing unintended leaks from attached context notes. - Fixed critical issues with context-in-context (CiC) formatting and added early envelope guards in VaultQAChainRunner to prevent silent failures. - Updated message construction logic to utilize LayerToMessagesConverter for improved smart referencing and context preservation. - Comprehensive tests added to validate the new envelope-based context handling and ensure robustness across all chain runners. * Integrate Plus chain with context envelope * Enhance AutonomousAgentChainRunner with envelope-based context integration - Introduced `LayerToMessagesConverter` for improved message construction, ensuring system and user messages are derived from the context envelope. - Implemented envelope validation to ensure context is available before processing. - Updated message preparation logic to maintain consistency with the envelope-first architecture, preserving tool execution and multimodal support. - Added comprehensive logging for envelope-based context construction to aid debugging. - Documented changes in the context engineering documentation to reflect the new integration and design points. * Support loading saved chat context to L2 context library * Fix ChatPersistenceManager tests for context handling	2025-10-27 23:28:36 -07:00
Logan Yang	92d6f98af2	Implement inline citation (#1821 ) * Implement inline citation for vault search for both vault QA and Plus chains refactor: enhance citation utilities and improve source handling in chain runners - Introduced new utility functions for citation management, including sanitization and formatting of source catalogs. - Updated CopilotPlusChainRunner and VaultQAChainRunner to utilize the new citation utilities for improved citation handling. - Enhanced ChatSingleMessage component to consolidate duplicate sources and ensure accurate citation numbering. - Added comprehensive tests for citation utilities to ensure functionality and edge case handling. refactor: dedupe chunks for unique titles in search tool sources refactor: enhance citation handling in CopilotPlus and VaultQA chain runners - Introduced a sanitization function to remove pre-existing citation markers from document content, preventing number leakage. - Updated the formatting of retrieved documents in VaultQA to utilize the new sanitization function for cleaner output. - Improved footnote renumbering logic in ChatSingleMessage to prioritize first-mention order, ensuring accurate citation mapping. refactor: streamline source citation handling in ChatSingleMessage component and enhance web search tool instructions - Simplified the processing of source links in ChatSingleMessage to use a more efficient mapping approach. - Updated the web search tool's instruction to clarify the use of footnote-style citations, ensuring compatibility with Markdown formatting. * feat: add inline citation setting toggle and refactor - Introduced a new setting to enable inline citations in AI-generated responses, allowing for footnote-style references within the text. - Updated citation utilities to manage fallback sources and improve citation instructions based on user settings. - Enhanced CopilotPlusChainRunner and VaultQAChainRunner to utilize the new inline citation feature, ensuring accurate source representation. - Modified ChatSingleMessage component to process inline citations based on the new setting, improving the display of sources in chat messages. - Added tests for new citation functionalities to ensure reliability and correctness. * feat: enhance citation utilities and add tests for normalization - Introduced the `normalizeCitations` function to improve citation formatting by removing periods after citations and handling footnote references. - Updated `processInlineCitations` to ensure correct mapping of non-sequential citations and consolidate duplicate sources. - Added comprehensive tests for the new citation functionalities, ensuring accurate citation processing and representation in AI-generated responses. - Modified QASettings description for clarity regarding the inline citations feature. * fix: address critical bug in citation processing for consecutive citations - Enhanced the `normalizeCitations` function to correctly handle consecutive citations without spaces, ensuring accurate mapping and representation in output. - Added a test case to replicate and validate the fix for the identified bug, confirming that both citations are processed and displayed correctly. - Updated existing citation handling logic to improve robustness and prevent unconverted citations in the output. * refactor: improve citation handling and enhance citation utilities - Refactored `CopilotPlusChainRunner` to streamline citation source management, replacing `lastCitationLines` with `lastCitationSources` for better clarity and structure. - Updated the `addFallbackSources` function to handle citation formatting more effectively, ensuring proper display of titles and paths. - Enhanced the `normalizeCitations` function to support multiple citations in brackets, ensuring correct sorting and mapping of citation numbers. - Added new test cases to validate the handling of alternate headings, HTML summaries, and multiple citations, improving overall robustness of citation processing. - Introduced `getWebSearchCitationInstructions` for web-specific citation guidance, enhancing user experience with clear instructions for inline citations. * fix: move documentations * refactor: normalize footnote rendering in ChatSingleMessage component - Introduced `normalizeFootnoteRendering` function to streamline footnote display by removing separators and backreferences, and fixing numbering artifacts. - Updated `ChatSingleMessage` to utilize the new normalization function, enhancing the clarity and presentation of footnotes in chat messages. - Added unit tests for `normalizeFootnoteRendering` to ensure proper functionality and handling of various footnote scenarios. * test: enhance ChatSingleMessage tests and add deduplication tests for search tools - Updated `ChatSingleMessage.test.tsx` to include `TooltipProvider` for improved rendering context. - Refactored mock implementation for `obsidian` to streamline testing setup. - Introduced new test file `SearchTools.dedupe.test.ts` to validate the deduplication logic in search tool sources, ensuring correct handling of duplicate entries while preserving the highest score. - Added tests to verify deduplication behavior for various scenarios, enhancing overall test coverage and reliability.	2025-09-21 20:03:09 -07:00

Author

SHA1

Message

Date

Logan Yang

05beb941eb

Context revamp (#1971 )

* Implement context engine for a layered context

* Wire in new context library to LLM chain

- Updated the context engine to maintain a cumulative library of all context items (L2) without deduplication, enhancing cache stability.
- Introduced smart referencing in the user message (L3) to reference existing items in L2 by ID and include full content for new items.
- Improved the LayerToMessagesConverter to format and handle layered context envelopes for better debugging and clarity.
- Fixed critical bugs related to context deduplication that previously broke cache stability.
- Enhanced tests to validate the new context handling and ensure proper functionality across various scenarios.

* Enhance VaultQAChainRunner with tag extraction and inline citation detection

- Integrated tag extraction from user queries to improve context-aware retrieval.
- Implemented `hasInlineCitations` function to detect inline citation markers in responses.
- Updated message construction logic to utilize envelope-based context, enhancing the quality of AI interactions.
- Added comprehensive tests for the new inline citation detection functionality, ensuring robustness and accuracy.
- Improved logging for better debugging and traceability of the message processing flow.

Refactor VaultQAChainRunner to enhance context handling and citation instructions

- Integrated LayerToMessagesConverter for improved message construction, ensuring smart referencing and context preservation.
- Updated logging to reflect the use of the new converter in envelope-based context construction.
- Revised comments for clarity on context preparation and citation instructions.
- Minor adjustment in citation utility to improve regex handling for citation updates.

* Complete migration to envelope-based context in CopilotPlus and LLM chain runners

- Removed all legacy fallback paths, ensuring that context envelopes are now mandatory for operations in CopilotPlusChainRunner and LLMChainRunner.
- Enhanced image extraction to only pull from the active note, preventing unintended leaks from attached context notes.
- Fixed critical issues with context-in-context (CiC) formatting and added early envelope guards in VaultQAChainRunner to prevent silent failures.
- Updated message construction logic to utilize LayerToMessagesConverter for improved smart referencing and context preservation.
- Comprehensive tests added to validate the new envelope-based context handling and ensure robustness across all chain runners.

* Integrate Plus chain with context envelope

* Enhance AutonomousAgentChainRunner with envelope-based context integration

- Introduced `LayerToMessagesConverter` for improved message construction, ensuring system and user messages are derived from the context envelope.
- Implemented envelope validation to ensure context is available before processing.
- Updated message preparation logic to maintain consistency with the envelope-first architecture, preserving tool execution and multimodal support.
- Added comprehensive logging for envelope-based context construction to aid debugging.
- Documented changes in the context engineering documentation to reflect the new integration and design points.

* Support loading saved chat context to L2 context library

* Fix ChatPersistenceManager tests for context handling

2025-10-27 23:28:36 -07:00

Logan Yang

92d6f98af2

Implement inline citation (#1821 )

* Implement inline citation for vault search for both vault QA and Plus chains

refactor: enhance citation utilities and improve source handling in chain runners

- Introduced new utility functions for citation management, including sanitization and formatting of source catalogs.
- Updated CopilotPlusChainRunner and VaultQAChainRunner to utilize the new citation utilities for improved citation handling.
- Enhanced ChatSingleMessage component to consolidate duplicate sources and ensure accurate citation numbering.
- Added comprehensive tests for citation utilities to ensure functionality and edge case handling.

refactor: dedupe chunks for unique titles in search tool sources

refactor: enhance citation handling in CopilotPlus and VaultQA chain runners

- Introduced a sanitization function to remove pre-existing citation markers from document content, preventing number leakage.
- Updated the formatting of retrieved documents in VaultQA to utilize the new sanitization function for cleaner output.
- Improved footnote renumbering logic in ChatSingleMessage to prioritize first-mention order, ensuring accurate citation mapping.

refactor: streamline source citation handling in ChatSingleMessage component and enhance web search tool instructions

- Simplified the processing of source links in ChatSingleMessage to use a more efficient mapping approach.
- Updated the web search tool's instruction to clarify the use of footnote-style citations, ensuring compatibility with Markdown formatting.

* feat: add inline citation setting toggle and refactor

- Introduced a new setting to enable inline citations in AI-generated responses, allowing for footnote-style references within the text.
- Updated citation utilities to manage fallback sources and improve citation instructions based on user settings.
- Enhanced CopilotPlusChainRunner and VaultQAChainRunner to utilize the new inline citation feature, ensuring accurate source representation.
- Modified ChatSingleMessage component to process inline citations based on the new setting, improving the display of sources in chat messages.
- Added tests for new citation functionalities to ensure reliability and correctness.

* feat: enhance citation utilities and add tests for normalization

- Introduced the `normalizeCitations` function to improve citation formatting by removing periods after citations and handling footnote references.
- Updated `processInlineCitations` to ensure correct mapping of non-sequential citations and consolidate duplicate sources.
- Added comprehensive tests for the new citation functionalities, ensuring accurate citation processing and representation in AI-generated responses.
- Modified QASettings description for clarity regarding the inline citations feature.

* fix: address critical bug in citation processing for consecutive citations

- Enhanced the `normalizeCitations` function to correctly handle consecutive citations without spaces, ensuring accurate mapping and representation in output.
- Added a test case to replicate and validate the fix for the identified bug, confirming that both citations are processed and displayed correctly.
- Updated existing citation handling logic to improve robustness and prevent unconverted citations in the output.

* refactor: improve citation handling and enhance citation utilities

- Refactored `CopilotPlusChainRunner` to streamline citation source management, replacing `lastCitationLines` with `lastCitationSources` for better clarity and structure.
- Updated the `addFallbackSources` function to handle citation formatting more effectively, ensuring proper display of titles and paths.
- Enhanced the `normalizeCitations` function to support multiple citations in brackets, ensuring correct sorting and mapping of citation numbers.
- Added new test cases to validate the handling of alternate headings, HTML summaries, and multiple citations, improving overall robustness of citation processing.
- Introduced `getWebSearchCitationInstructions` for web-specific citation guidance, enhancing user experience with clear instructions for inline citations.

* fix: move documentations

* refactor: normalize footnote rendering in ChatSingleMessage component

- Introduced `normalizeFootnoteRendering` function to streamline footnote display by removing separators and backreferences, and fixing numbering artifacts.
- Updated `ChatSingleMessage` to utilize the new normalization function, enhancing the clarity and presentation of footnotes in chat messages.
- Added unit tests for `normalizeFootnoteRendering` to ensure proper functionality and handling of various footnote scenarios.

* test: enhance ChatSingleMessage tests and add deduplication tests for search tools

- Updated `ChatSingleMessage.test.tsx` to include `TooltipProvider` for improved rendering context.
- Refactored mock implementation for `obsidian` to streamline testing setup.
- Introduced new test file `SearchTools.dedupe.test.ts` to validate the deduplication logic in search tool sources, ensuring correct handling of duplicate entries while preserving the highest score.
- Added tests to verify deduplication behavior for various scenarios, enhancing overall test coverage and reliability.

2025-09-21 20:03:09 -07:00

Renamed from MESSAGE_ARCHITECTURE.md (Browse further)

2 commits