Audit context envelope, tag alignment, artifact dedup, and logging (#2164)

* fix: align context block tags and make parseContextIntoSegments registry-driven

The context envelope L2 layer was silently dropping URL/media context across
turns due to three compounding issues:

1. Tag mismatch: Mention.ts created <youtube_transcript> blocks but the
   contextBlockRegistry expected <youtube_video_context>. Fixed the tag and
   inner element (<transcript> → <content>) to match the registry.

2. Missing registration: twitter_content blocks had no registry entry, so
   they were invisible to compaction and segment parsing. Added the entry.

3. Hardcoded parser: parseContextIntoSegments only matched <note_context>,
   <active_note>, and <prior_context> via hardcoded regexes, silently
   ignoring all URL/media block types. Rewrote it to dynamically build its
   regex from CONTEXT_BLOCK_TYPES, ensuring any future block type added to
   the registry is automatically handled.

Extracted parseContextIntoSegments into src/context/parseContextSegments.ts
as a standalone pure function for direct testability.

Added 19 new tests covering all registered block types, mixed blocks,
stable/unstable flags, and a registry completeness guard test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: context envelope improvements - regeneration, dedup, per-artifact IDs, XML escaping

Phase A: Fix regeneration from loaded chats by lazily reprocessing context
when contextEnvelope is missing (e.g., messages loaded from disk).

Phase B: Replace static segment IDs ("urls", "selected_text", "web_tabs")
with per-artifact IDs from parseContextIntoSegments, enabling accurate
smart referencing in LayerToMessagesConverter.

Phase C: Deduplicate L2 content by segment ID (last-write-wins) to prevent
linear growth when the same artifact appears across multiple turns.

Phase D: Escape note title/basename in XML context blocks for consistency
with URL/YouTube content. Keep path unescaped as it's used as identifier.

Additional fixes from Codex review:
- Unique IDs for blocks without source extractors (selected_text counter)
- Keep note.path raw to avoid dedup mismatch with TFile.path

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: prevent context artifact duplication in LLM payload and L4 memory

CopilotPlusChainRunner was re-injecting processedText (which includes
context XML artifacts) into the user message via ensureUserQueryLabel,
bypassing the envelope's clean L2/L3/L5 separation. This caused context
like YouTube transcripts to appear 3x in the LLM payload.

Fix: use L5_USER envelope text (expanded user query without context XML)
instead of processedText for cleanedUserMessage, trimmedQuestion fallback,
and messageForAnalysis. Also fix BaseChainRunner.handleResponse to save
L5 text to L4 memory instead of processedText.

Other changes:
- Remove dead updateMemoryWithLoadedMessages from chainManager
- Update CONTEXT_ENGINEERING.md with example walkthrough, L4 behavior
  docs, and Phase 6 integration test suite roadmap

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add chain runner envelope usage section and fix Step 0 log

Document per-runner envelope behavior, CopilotPlus/Agent tool flows,
and token efficiency audit in CONTEXT_ENGINEERING.md. Fix Step 0 log
to show L5 user query instead of processedText with context XML.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: trim trailing whitespace from chat input display text

Trailing newlines from Lexical editor were preserved in displayText
and rendered as empty lines in chat bubbles due to whitespace-pre-wrap.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

src/LLMProviders/chainManager.ts

@@ -333,7 +333,11 @@ export default class ChainManager {
   ) {
     const { ignoreSystemMessage = false } = options;
 
-    logInfo("Step 0: Initial user message:\n", userMessage.message);
+    const l5Text = userMessage.contextEnvelope?.layers.find((l) => l.id === "L5_USER")?.text;
+    logInfo(
+      "Step 0: Initial user message:\n",
+      l5Text || userMessage.originalMessage || userMessage.message
+    );
 
     this.validateChatModel();
     this.validateChainInitialization();
@@ -371,16 +375,4 @@ export default class ChainManager {
       options
     );
   }
-
-  async updateMemoryWithLoadedMessages(messages: ChatMessage[]) {
-    await this.memoryManager.clearChatMemory();
-    // Use memoryManager.saveContext to apply compaction for any old uncompacted messages
-    for (let i = 0; i < messages.length; i += 2) {
-      const userMsg = messages[i];
-      const aiMsg = messages[i + 1];
-      if (userMsg && aiMsg && userMsg.sender === USER_SENDER) {
-        await this.memoryManager.saveContext({ input: userMsg.message }, { output: aiMsg.message });
-      }
-    }
-  }
 }

src/LLMProviders/chainRunner/BaseChainRunner.ts

@@ -70,14 +70,18 @@ export abstract class BaseChainRunner implements ChainRunner {
       !(abortController.signal.aborted && abortController.signal.reason === ABORT_REASON.NEW_CHAT);
 
     if (shouldAddMessage) {
-      // Use MemoryManager.saveContext for atomic operation and proper memory management
-      // This applies chat history compaction to reduce memory bloat from tool results
-      // Note: LangChain's memory expects text content, not multimodal arrays
-      // For truncated empty responses, save a placeholder message
+      // Save the expanded user message (L5) to memory — NOT the full processedText.
+      // processedText includes context artifact XML (L3), which already lives in the
+      // envelope's L2/L3 layers. Baking it into L4 chat history would cause
+      // triple-inclusion and waste tokens.
+      // L5 preserves prompt-expanded content (e.g. {include_note_content} placeholders)
+      // while excluding context artifact blocks.
+      const l5Text = userMessage.contextEnvelope?.layers.find((l) => l.id === "L5_USER")?.text;
+      const inputForMemory = l5Text || userMessage.originalMessage || userMessage.message;
       const outputForMemory =
         llmFormattedOutput || fullAIResponse || "[Response truncated - no content generated]";
       await this.chainManager.memoryManager.saveContext(
-        { input: userMessage.message },
+        { input: inputForMemory },
         { output: outputForMemory }
       );
 

src/LLMProviders/chainRunner/CopilotPlusChainRunner.ts

@@ -606,10 +606,7 @@ Include your extracted terms as: [SALIENT_TERMS: term1, term2, term3]`;
         }
 
         const trimmedQuestion =
-          originalUserQuestion.trim() ||
-          userMessage.message?.trim() ||
-          userMessage.originalMessage?.trim() ||
-          "";
+          originalUserQuestion.trim() || userMessage.originalMessage?.trim() || "";
         if (trimmedQuestion.length > 0) {
           sections.push(`${userQueryLabel}\n${trimmedQuestion}`);
         } else {
@@ -740,7 +737,7 @@ Include your extracted terms as: [SALIENT_TERMS: term1, term2, term3]`;
         );
       }
       const l5User = envelope.layers.find((l) => l.id === "L5_USER");
-      const messageForAnalysis = l5User?.text || userMessage.originalMessage || userMessage.message;
+      const messageForAnalysis = l5User?.text || userMessage.originalMessage || "";
 
       try {
         // Use model-based planning instead of Broca
@@ -816,7 +813,13 @@ Include your extracted terms as: [SALIENT_TERMS: term1, term2, term3]`;
       }
 
       // Clean user message by removing @command tokens
-      const cleanedUserMessage = this.removeAtCommands(userMessage.message);
+      // Use L5 text (expanded user query without context XML) or displayText as fallback.
+      // userMessage.message is processedText which includes context artifact XML — using it
+      // here would re-inject L3 context into the user message, bypassing envelope separation.
+      const l5Text = userMessage.contextEnvelope?.layers.find((l) => l.id === "L5_USER")?.text;
+      const cleanedUserMessage = this.removeAtCommands(
+        l5Text || userMessage.originalMessage || userMessage.message
+      );
 
       const { toolOutputs, sources: toolSources } = await this.executeToolCalls(
         toolCalls,

src/components/Chat.tsx

@@ -259,7 +259,7 @@ const ChatInternal: React.FC<ChatProps & { chatInput: ReturnType<typeof useChatI
       );
 
       // Handle composer prompt
-      let displayText = inputMessage;
+      let displayText = inputMessage.trim();
 
       // Add tool calls if present
       if (toolCalls) {

src/context/contextBlockRegistry.test.ts

@@ -17,6 +17,7 @@ describe("contextBlockRegistry", () => {
       expect(tags).toContain("active_note");
       expect(tags).toContain("url_content");
       expect(tags).toContain("youtube_video_context");
+      expect(tags).toContain("twitter_content");
       expect(tags).toContain("selected_text");
       expect(tags).toContain("localSearch");
     });
@@ -45,6 +46,7 @@ describe("contextBlockRegistry", () => {
     it("should return correct source type for URL blocks", () => {
       expect(getSourceType("url_content")).toBe("url");
       expect(getSourceType("web_tab_context")).toBe("url");
+      expect(getSourceType("twitter_content")).toBe("url");
     });
 
     it("should return correct source type for YouTube blocks", () => {
@@ -61,6 +63,7 @@ describe("contextBlockRegistry", () => {
       expect(isRecoverable("note_context")).toBe(true);
       expect(isRecoverable("url_content")).toBe(true);
       expect(isRecoverable("youtube_video_context")).toBe(true);
+      expect(isRecoverable("twitter_content")).toBe(true);
     });
 
     it("should return false for non-recoverable types", () => {
@@ -108,6 +111,14 @@ describe("contextBlockRegistry", () => {
       expect(extractSourceFromBlock(xml, "embedded_pdf")).toBe("document.pdf");
     });
 
+    it("should extract URL from twitter_content blocks", () => {
+      const xml = `<twitter_content>
+<url>https://x.com/user/status/123</url>
+<content>Tweet</content>
+</twitter_content>`;
+      expect(extractSourceFromBlock(xml, "twitter_content")).toBe("https://x.com/user/status/123");
+    });
+
     it("should return empty string for blocks without source extractor", () => {
       const xml = `<selected_text><content>Just text</content></selected_text>`;
       expect(extractSourceFromBlock(xml, "selected_text")).toBe("");
@@ -140,4 +151,17 @@ describe("contextBlockRegistry", () => {
       expect(detectBlockTag("")).toBeNull();
     });
   });
+
+  describe("tag alignment", () => {
+    it("should have all URL-producing tags registered (prevents tag mismatch bugs)", () => {
+      // These are the tags that Mention.ts and other URL processors create.
+      // If a new tag is added to URL processing, it MUST be registered here.
+      const registeredTags = new Set(CONTEXT_BLOCK_TYPES.map((bt) => bt.tag));
+      const urlProducerTags = ["url_content", "youtube_video_context", "twitter_content"];
+
+      for (const tag of urlProducerTags) {
+        expect(registeredTags.has(tag)).toBe(true);
+      }
+    });
+  });
 });

src/context/contextBlockRegistry.ts

@@ -56,6 +56,9 @@ export const CONTEXT_BLOCK_TYPES: ContextBlockType[] = [
     sourceExtractor: "url",
   },
 
+  // Twitter/X (recoverable via URL processing)
+  { tag: "twitter_content", sourceType: "url", recoverable: true, sourceExtractor: "url" },
+
   // PDF (partially recoverable - needs the file in vault)
   { tag: "embedded_pdf", sourceType: "pdf", recoverable: true, sourceExtractor: "name" },
 

src/context/parseContextSegments.ts

@@ -0,0 +1,76 @@
+import { PromptLayerSegment } from "@/context/PromptContextTypes";
+import {
+  CONTEXT_BLOCK_TYPES,
+  extractSourceFromBlock,
+  getSourceType,
+} from "@/context/contextBlockRegistry";
+
+/**
+ * Parse context XML string into individual segments (one per context item).
+ * Uses the contextBlockRegistry to dynamically match ALL registered block types,
+ * plus <prior_context> blocks (compaction artifacts from L2).
+ */
+export function parseContextIntoSegments(
+  contextXml: string,
+  stable: boolean
+): PromptLayerSegment[] {
+  if (!contextXml.trim()) {
+    return [];
+  }
+
+  const segments: PromptLayerSegment[] = [];
+
+  // Build regex dynamically from all registered block types + prior_context
+  const registeredTags = CONTEXT_BLOCK_TYPES.map((bt) => bt.tag);
+  const allTags = [...registeredTags, "prior_context"];
+  const allBlocksRegex = new RegExp(`<(${allTags.join("|")})(\\s[^>]*)?>[\\s\\S]*?</\\1>`, "g");
+
+  // Track tag-based fallback IDs to ensure uniqueness for blocks without source extractors
+  // (e.g., multiple selected_text blocks should not share the same ID)
+  const tagIdCounts = new Map<string, number>();
+
+  let match: RegExpExecArray | null;
+  while ((match = allBlocksRegex.exec(contextXml)) !== null) {
+    const block = match[0];
+    const tag = match[1];
+
+    if (tag === "prior_context") {
+      // Compacted blocks have source in attribute: <prior_context source="path" type="note">
+      const sourceMatch = /source="([^"]+)"/.exec(block);
+      const source = sourceMatch?.[1] ?? "prior_context";
+      segments.push({
+        id: source,
+        content: block,
+        stable,
+        metadata: {
+          source: "previous_turns_compacted",
+          notePath: source,
+        },
+      });
+    } else {
+      // Use registry to extract the source identifier
+      const extractedId = extractSourceFromBlock(block, tag);
+      let sourceId: string;
+      if (extractedId) {
+        sourceId = extractedId;
+      } else {
+        // Fallback: use tag name with counter to ensure uniqueness
+        const count = (tagIdCounts.get(tag) || 0) + 1;
+        tagIdCounts.set(tag, count);
+        sourceId = count === 1 ? tag : `${tag}:${count}`;
+      }
+      const isNote = getSourceType(tag) === "note";
+      segments.push({
+        id: sourceId,
+        content: block,
+        stable,
+        metadata: {
+          source: stable ? "previous_turns" : "current_turn",
+          ...(isNote ? { notePath: sourceId } : {}),
+        },
+      });
+    }
+  }
+
+  return segments;
+}

src/contextProcessor.ts

@@ -573,10 +573,10 @@ export class ContextProcessor {
         const ctime = stats ? new Date(stats.ctime).toISOString() : "Unknown";
         const mtime = stats ? new Date(stats.mtime).toISOString() : "Unknown";
 
-        additionalContext += `\n\n<${prompt_tag}>\n<title>${note.basename}</title>\n<path>${note.path}</path>\n<ctime>${ctime}</ctime>\n<mtime>${mtime}</mtime>\n<content>\n${content}\n</content>\n</${prompt_tag}>`;
+        additionalContext += `\n\n<${prompt_tag}>\n<title>${escapeXml(note.basename)}</title>\n<path>${note.path}</path>\n<ctime>${ctime}</ctime>\n<mtime>${mtime}</mtime>\n<content>\n${content}\n</content>\n</${prompt_tag}>`;
       } catch (error) {
         logError(`Error processing file ${note.path}:`, error);
-        additionalContext += `\n\n<${prompt_tag}_error>\n<title>${note.basename}</title>\n<path>${note.path}</path>\n<error>[Error: Could not process file]</error>\n</${prompt_tag}_error>`;
+        additionalContext += `\n\n<${prompt_tag}_error>\n<title>${escapeXml(note.basename)}</title>\n<path>${note.path}</path>\n<error>[Error: Could not process file]</error>\n</${prompt_tag}_error>`;
       }
     };
 

src/core/ChatManager.test.ts

@@ -28,6 +28,7 @@ jest.mock("./ChatPersistenceManager", () => ({
 
 jest.mock("@/aiParams", () => ({
   getCurrentProject: jest.fn().mockReturnValue(null),
+  getChainType: jest.fn().mockReturnValue("copilot_plus_chain"),
 }));
 
 jest.mock("@/LLMProviders/projectManager", () => {
@@ -329,7 +330,10 @@ describe("ChatManager", () => {
     it("should regenerate AI message successfully", async () => {
       const mockAiMessage = createMockMessage("msg-2", "AI response", "AI");
       const mockUserMessage = createMockMessage("msg-1", "Hello", USER_SENDER);
-      const mockLLMMessage = createMockMessage("msg-1", "Hello with context", USER_SENDER);
+      const mockLLMMessage = {
+        ...createMockMessage("msg-1", "Hello with context", USER_SENDER),
+        contextEnvelope: { layers: [] } as any, // Has envelope, no lazy reprocessing needed
+      };
 
       mockMessageRepo.getMessage.mockReturnValue(mockAiMessage);
       mockMessageRepo.getDisplayMessages.mockReturnValue([mockUserMessage, mockAiMessage]);
@@ -357,6 +361,53 @@ describe("ChatManager", () => {
       );
     });
 
+    it("should lazily reprocess context when envelope is missing (loaded from disk)", async () => {
+      const mockAiMessage = createMockMessage("msg-2", "AI response", "AI");
+      const mockUserMessage = createMockMessage("msg-1", "Hello", USER_SENDER);
+      // First call: no envelope (loaded from disk)
+      const mockLLMMessageNoEnvelope = createMockMessage(
+        "msg-1",
+        "Hello with context",
+        USER_SENDER
+      );
+      // Second call: after reprocessing, has envelope
+      const mockLLMMessageWithEnvelope = {
+        ...createMockMessage("msg-1", "Hello with context", USER_SENDER),
+        contextEnvelope: { layers: [] } as any,
+      };
+
+      mockMessageRepo.getMessage.mockReturnValue(mockAiMessage);
+      mockMessageRepo.getDisplayMessages.mockReturnValue([mockUserMessage, mockAiMessage]);
+      mockMessageRepo.getLLMMessage
+        .mockReturnValueOnce(mockLLMMessageNoEnvelope)
+        .mockReturnValueOnce(mockLLMMessageWithEnvelope);
+      mockMessageRepo.truncateAfter.mockReturnValue(undefined);
+      mockChainManager.runChain.mockResolvedValue(undefined);
+      mockContextManager.reprocessMessageContext.mockResolvedValue(undefined);
+
+      const result = await chatManager.regenerateMessage("msg-2", jest.fn(), jest.fn());
+
+      expect(result).toBe(true);
+      expect(mockContextManager.reprocessMessageContext).toHaveBeenCalledWith(
+        "msg-1",
+        expect.anything(), // messageRepo
+        expect.anything(), // fileParserManager
+        undefined, // vault (undefined in mock)
+        "copilot_plus_chain", // chainType
+        false, // includeActiveNote
+        undefined, // activeNote
+        "Test system prompt", // systemPrompt
+        [] // systemPromptIncludedFiles
+      );
+      expect(mockChainManager.runChain).toHaveBeenCalledWith(
+        mockLLMMessageWithEnvelope,
+        expect.any(AbortController),
+        expect.any(Function),
+        expect.any(Function),
+        expect.any(Object)
+      );
+    });
+
     it("should return false when message not found", async () => {
       mockMessageRepo.getMessage.mockReturnValue(undefined);
 
@@ -621,7 +672,10 @@ describe("ChatManager", () => {
       it("should handle regeneration with proper message truncation", async () => {
         const mockAiMessage = createMockMessage("msg-2", "AI response", "AI");
         const mockUserMessage = createMockMessage("msg-1", "Hello", USER_SENDER);
-        const mockLLMMessage = createMockMessage("msg-1", "Hello with context", USER_SENDER);
+        const mockLLMMessage = {
+          ...createMockMessage("msg-1", "Hello with context", USER_SENDER),
+          contextEnvelope: { layers: [] } as any,
+        };
 
         mockMessageRepo.getMessage.mockReturnValue(mockAiMessage);
         mockMessageRepo.getDisplayMessages.mockReturnValue([mockUserMessage, mockAiMessage]);

src/core/ChatManager.ts

@@ -5,7 +5,7 @@ import {
   getSystemPromptWithMemory,
 } from "@/system-prompts/systemPromptBuilder";
 import { ChainType } from "@/chainFactory";
-import { getCurrentProject } from "@/aiParams";
+import { getChainType, getCurrentProject } from "@/aiParams";
 import { logInfo, logWarn } from "@/logger";
 import { ChatMessage, MessageContext, WebTabContext } from "@/types/message";
 import { processPrompt, type ProcessedPromptResult } from "@/commands/customCommandUtils";
@@ -589,12 +589,35 @@ export class ChatManager {
         return false;
       }
 
-      const llmMessage = currentRepo.getLLMMessage(userMessage.id);
+      let llmMessage = currentRepo.getLLMMessage(userMessage.id);
       if (!llmMessage) {
         logInfo(`[ChatManager] LLM message not found for regeneration`);
         return false;
       }
 
+      // Lazy reprocess: if contextEnvelope is missing (e.g., loaded from disk),
+      // reprocess context before running the chain
+      if (!llmMessage.contextEnvelope) {
+        logInfo(`[ChatManager] Context envelope missing, reprocessing context for regeneration`);
+        const chainType = getChainType();
+        const activeNote = this.plugin.app.workspace.getActiveFile();
+        const { processedPrompt: systemPrompt, includedFiles: systemPromptIncludedFiles } =
+          await this.getSystemPromptForMessage(chainType, this.plugin.app.vault, activeNote);
+        await this.contextManager.reprocessMessageContext(
+          userMessage.id,
+          currentRepo,
+          this.fileParserManager,
+          this.plugin.app.vault,
+          chainType,
+          false,
+          activeNote,
+          systemPrompt,
+          systemPromptIncludedFiles
+        );
+        // Re-fetch the LLM message with the newly created envelope
+        llmMessage = currentRepo.getLLMMessage(userMessage.id)!;
+      }
+
       // Run the chain to regenerate the response
       const abortController = new AbortController();
       await this.chainManager.runChain(

src/core/ContextManager.parseSegments.test.ts

@@ -0,0 +1,270 @@
+import { parseContextIntoSegments } from "@/context/parseContextSegments";
+import { CONTEXT_BLOCK_TYPES } from "@/context/contextBlockRegistry";
+
+describe("parseContextIntoSegments", () => {
+  describe("empty input", () => {
+    it("should return empty array for empty string", () => {
+      expect(parseContextIntoSegments("", false)).toEqual([]);
+    });
+
+    it("should return empty array for whitespace-only string", () => {
+      expect(parseContextIntoSegments("   \n  ", false)).toEqual([]);
+    });
+  });
+
+  describe("note blocks", () => {
+    it("should parse note_context blocks with path as ID", () => {
+      const xml = `<note_context>
+<title>My Note</title>
+<path>folder/my-note.md</path>
+<content>Some content</content>
+</note_context>`;
+      const segments = parseContextIntoSegments(xml, false);
+      expect(segments).toHaveLength(1);
+      expect(segments[0].id).toBe("folder/my-note.md");
+      expect(segments[0].content).toBe(xml);
+      expect(segments[0].stable).toBe(false);
+      expect(segments[0].metadata?.source).toBe("current_turn");
+      expect(segments[0].metadata?.notePath).toBe("folder/my-note.md");
+    });
+
+    it("should parse active_note blocks", () => {
+      const xml = `<active_note>
+<title>Active</title>
+<path>active.md</path>
+<content>Active content</content>
+</active_note>`;
+      const segments = parseContextIntoSegments(xml, true);
+      expect(segments).toHaveLength(1);
+      expect(segments[0].id).toBe("active.md");
+      expect(segments[0].metadata?.source).toBe("previous_turns");
+      expect(segments[0].metadata?.notePath).toBe("active.md");
+    });
+  });
+
+  describe("URL blocks", () => {
+    it("should parse url_content blocks with URL as ID", () => {
+      const xml = `<url_content>
+<url>https://example.com/page</url>
+<content>Page content here</content>
+</url_content>`;
+      const segments = parseContextIntoSegments(xml, false);
+      expect(segments).toHaveLength(1);
+      expect(segments[0].id).toBe("https://example.com/page");
+      expect(segments[0].content).toBe(xml);
+      expect(segments[0].metadata?.source).toBe("current_turn");
+      expect(segments[0].metadata?.notePath).toBeUndefined();
+    });
+
+    it("should parse web_tab_context blocks", () => {
+      const xml = `<web_tab_context>
+<url>https://docs.example.com</url>
+<content>Documentation</content>
+</web_tab_context>`;
+      const segments = parseContextIntoSegments(xml, false);
+      expect(segments).toHaveLength(1);
+      expect(segments[0].id).toBe("https://docs.example.com");
+    });
+  });
+
+  describe("YouTube blocks", () => {
+    it("should parse youtube_video_context blocks with URL as ID", () => {
+      const xml = `<youtube_video_context>
+<url>https://www.youtube.com/watch?v=abc123</url>
+<content>Transcript of the video</content>
+</youtube_video_context>`;
+      const segments = parseContextIntoSegments(xml, false);
+      expect(segments).toHaveLength(1);
+      expect(segments[0].id).toBe("https://www.youtube.com/watch?v=abc123");
+      expect(segments[0].content).toBe(xml);
+      expect(segments[0].metadata?.source).toBe("current_turn");
+      expect(segments[0].metadata?.notePath).toBeUndefined();
+    });
+  });
+
+  describe("Twitter blocks", () => {
+    it("should parse twitter_content blocks with URL as ID", () => {
+      const xml = `<twitter_content>
+<url>https://x.com/user/status/123</url>
+<content>Tweet content</content>
+</twitter_content>`;
+      const segments = parseContextIntoSegments(xml, false);
+      expect(segments).toHaveLength(1);
+      expect(segments[0].id).toBe("https://x.com/user/status/123");
+      expect(segments[0].metadata?.source).toBe("current_turn");
+    });
+  });
+
+  describe("PDF blocks", () => {
+    it("should parse embedded_pdf blocks with name as ID", () => {
+      const xml = `<embedded_pdf>
+<name>document.pdf</name>
+<content>PDF text content</content>
+</embedded_pdf>`;
+      const segments = parseContextIntoSegments(xml, false);
+      expect(segments).toHaveLength(1);
+      expect(segments[0].id).toBe("document.pdf");
+    });
+  });
+
+  describe("selected_text blocks", () => {
+    it("should parse selected_text blocks with tag as ID (no source extractor)", () => {
+      const xml = `<selected_text>
+<content>User selected this text</content>
+</selected_text>`;
+      const segments = parseContextIntoSegments(xml, false);
+      expect(segments).toHaveLength(1);
+      expect(segments[0].id).toBe("selected_text");
+    });
+
+    it("should assign unique IDs to multiple selected_text blocks", () => {
+      const xml = `<selected_text>
+<content>First selection</content>
+</selected_text>
+
+<selected_text>
+<content>Second selection</content>
+</selected_text>`;
+      const segments = parseContextIntoSegments(xml, false);
+      expect(segments).toHaveLength(2);
+      expect(segments[0].id).toBe("selected_text");
+      expect(segments[1].id).toBe("selected_text:2");
+    });
+  });
+
+  describe("prior_context blocks (compaction artifacts)", () => {
+    it("should parse prior_context blocks with source attribute as ID", () => {
+      const xml = `<prior_context source="folder/old-note.md" type="note">
+Compacted summary of old note
+</prior_context>`;
+      const segments = parseContextIntoSegments(xml, true);
+      expect(segments).toHaveLength(1);
+      expect(segments[0].id).toBe("folder/old-note.md");
+      expect(segments[0].metadata?.source).toBe("previous_turns_compacted");
+      expect(segments[0].metadata?.notePath).toBe("folder/old-note.md");
+    });
+
+    it("should fall back to 'prior_context' as ID when no source attribute", () => {
+      const xml = `<prior_context>
+Compacted content
+</prior_context>`;
+      const segments = parseContextIntoSegments(xml, true);
+      expect(segments).toHaveLength(1);
+      expect(segments[0].id).toBe("prior_context");
+    });
+  });
+
+  describe("stable flag", () => {
+    it("should set metadata.source to 'previous_turns' when stable=true", () => {
+      const xml = `<url_content>
+<url>https://example.com</url>
+<content>Content</content>
+</url_content>`;
+      const segments = parseContextIntoSegments(xml, true);
+      expect(segments[0].stable).toBe(true);
+      expect(segments[0].metadata?.source).toBe("previous_turns");
+    });
+
+    it("should set metadata.source to 'current_turn' when stable=false", () => {
+      const xml = `<url_content>
+<url>https://example.com</url>
+<content>Content</content>
+</url_content>`;
+      const segments = parseContextIntoSegments(xml, false);
+      expect(segments[0].stable).toBe(false);
+      expect(segments[0].metadata?.source).toBe("current_turn");
+    });
+  });
+
+  describe("mixed block types", () => {
+    it("should parse multiple different block types from a single string", () => {
+      const xml = `<note_context>
+<title>Note</title>
+<path>notes/test.md</path>
+<content>Note content</content>
+</note_context>
+
+<url_content>
+<url>https://example.com</url>
+<content>URL content</content>
+</url_content>
+
+<youtube_video_context>
+<url>https://youtube.com/watch?v=xyz</url>
+<content>Transcript</content>
+</youtube_video_context>
+
+<prior_context source="old/note.md" type="note">
+Compacted note
+</prior_context>`;
+
+      const segments = parseContextIntoSegments(xml, false);
+      expect(segments).toHaveLength(4);
+      expect(segments[0].id).toBe("notes/test.md");
+      expect(segments[1].id).toBe("https://example.com");
+      expect(segments[2].id).toBe("https://youtube.com/watch?v=xyz");
+      expect(segments[3].id).toBe("old/note.md");
+    });
+
+    it("should parse multiple blocks of the same type", () => {
+      const xml = `<note_context>
+<title>Note 1</title>
+<path>note1.md</path>
+<content>Content 1</content>
+</note_context>
+
+<note_context>
+<title>Note 2</title>
+<path>note2.md</path>
+<content>Content 2</content>
+</note_context>`;
+
+      const segments = parseContextIntoSegments(xml, false);
+      expect(segments).toHaveLength(2);
+      expect(segments[0].id).toBe("note1.md");
+      expect(segments[1].id).toBe("note2.md");
+    });
+  });
+
+  describe("registry completeness", () => {
+    it("should parse every block type registered in CONTEXT_BLOCK_TYPES", () => {
+      for (const blockType of CONTEXT_BLOCK_TYPES) {
+        const tag = blockType.tag;
+        let xml: string;
+
+        // Build a valid XML block based on the source extractor type
+        switch (blockType.sourceExtractor) {
+          case "path":
+            xml = `<${tag}><path>test/file.md</path><content>test</content></${tag}>`;
+            break;
+          case "url":
+            xml = `<${tag}><url>https://example.com</url><content>test</content></${tag}>`;
+            break;
+          case "name":
+            xml = `<${tag}><name>file.pdf</name><content>test</content></${tag}>`;
+            break;
+          default:
+            xml = `<${tag}><content>test</content></${tag}>`;
+            break;
+        }
+
+        const segments = parseContextIntoSegments(xml, false);
+        expect(segments).toHaveLength(1);
+        expect(segments[0].content).toBe(xml);
+      }
+    });
+  });
+
+  describe("non-matching content", () => {
+    it("should return empty array for plain text without XML blocks", () => {
+      const segments = parseContextIntoSegments("Just some plain text", false);
+      expect(segments).toEqual([]);
+    });
+
+    it("should return empty array for unregistered XML tags", () => {
+      const xml = `<unknown_tag><content>Something</content></unknown_tag>`;
+      const segments = parseContextIntoSegments(xml, false);
+      expect(segments).toEqual([]);
+    });
+  });
+});

src/core/ContextManager.ts

@@ -5,6 +5,7 @@ import { LOADING_MESSAGES } from "@/constants";
 import { PromptContextEngine } from "@/context/PromptContextEngine";
 import { compactXmlBlock, getL2RefetchInstruction } from "@/context/L2ContextCompactor";
 import { CONTEXT_BLOCK_TYPES, detectBlockTag } from "@/context/contextBlockRegistry";
+import { parseContextIntoSegments } from "@/context/parseContextSegments";
 import {
   PromptContextEnvelope,
   PromptLayerId,
@@ -359,10 +360,9 @@ export class ContextManager {
    * - Comparing current file mtime with stored mtime when building L2
    * - Skipping path deduplication for stale compacted segments
    *
-   * TODO: Deduplicate L2 content by notePath when building l2Context.
-   * Currently, if the same note is included across multiple turns (e.g., auto-added active note),
-   * its content is appended to L2 once per turn, causing linear growth. Consider deduplicating
-   * by notePath when concatenating L3 segments, not just collecting paths for exclusions.
+   * L2 deduplication: Segments are deduplicated by ID (last-write-wins). When the same
+   * artifact (note, URL, etc.) appears in multiple turns, only the most recent version
+   * is kept in L2, preventing linear growth.
    */
   private buildL2ContextFromPreviousTurns(
     currentMessageId: string,
@@ -379,7 +379,10 @@ export class ContextManager {
       .slice(0, currentIndex)
       .filter((msg) => msg.sender === "user");
 
-    const l2Parts: string[] = [];
+    // Deduplicated map: segment ID → compacted content (last-write-wins)
+    const l2SegmentMap = new Map<string, string>();
+    // Insertion order tracking for stable output
+    const l2SegmentOrder: string[] = [];
     const l2Paths = new Set<string>();
 
     // Find the most recent compacted message index.
@@ -404,18 +407,16 @@ export class ContextManager {
         // Only include L3 content from the most recent compacted message onwards.
         // Earlier messages' content is already included in the compacted L3.
         if (i >= mostRecentCompactedIndex) {
-          const segmentContent: string[] = [];
           for (const segment of l3Layer.segments || []) {
             if (segment.content) {
-              // Compact large L3 segments for L2 to reduce context size
-              // This extracts structure + preview with tool hints for re-fetching
               const compacted = this.compactSegmentForL2(segment.content);
-              segmentContent.push(compacted);
+              // Deduplicate by segment ID: later turns overwrite earlier ones
+              if (!l2SegmentMap.has(segment.id)) {
+                l2SegmentOrder.push(segment.id);
+              }
+              l2SegmentMap.set(segment.id, compacted);
             }
           }
-          if (segmentContent.length > 0) {
-            l2Parts.push(segmentContent.join("\n"));
-          }
         }
 
         // Track paths from ALL messages for deduplication
@@ -423,13 +424,11 @@ export class ContextManager {
           if (segment.metadata?.notePath) {
             l2Paths.add(segment.metadata.notePath as string);
           }
-          // Handle compacted segments
           if (segment.metadata?.compactedPaths) {
             for (const path of segment.metadata.compactedPaths as string[]) {
               l2Paths.add(path);
             }
           }
-          // Handle tag/folder segments that store paths in notePaths
           if (segment.metadata?.notePaths) {
             for (const path of segment.metadata.notePaths as string[]) {
               l2Paths.add(path);
@@ -437,24 +436,10 @@ export class ContextManager {
           }
         }
       }
-      // Note: Messages without envelopes (pre-envelope chat history or loaded from disk)
-      // are intentionally NOT tracked for L2 deduplication to avoid filtering notes
-      // without their content appearing in L2.
-      //
-      // TODO: Rebuild L2 context for loaded chats.
-      // ChatPersistenceManager saves context metadata (note paths, tags, folders) but not
-      // the full contextEnvelope. When a chat is loaded, messages have context but no envelope,
-      // so L2 context is lost. This means:
-      // 1. AI loses the "context library" from turns before the save
-      // 2. Same notes can be duplicated if re-attached after loading
-      // Fix options:
-      // - Persist envelopes to disk (increases storage, requires migration)
-      // - Rebuild L2 content by re-reading files from message.context on load (expensive)
-      // - Lazy rebuild: process context.notes when first accessed after load
     }
 
-    // Build the L2 context string
-    const l2Content = l2Parts.join("\n");
+    // Build the L2 context string in insertion order (deduplicated)
+    const l2Content = l2SegmentOrder.map((id) => l2SegmentMap.get(id)!).join("\n");
 
     // Append re-fetch instruction if there's any L2 content with prior_context blocks
     const hasCompactedContent = l2Content.includes("<prior_context ");
@@ -504,25 +489,20 @@ export class ContextManager {
       turnSegments.push(...noteSegments);
     }
 
-    // Parse other context types (tags, folders, URLs, selected text)
-    // Store note paths in metadata for L2 deduplication
-    this.appendTurnContextSegment(turnSegments, "tags", params.tagContextAddition, {
+    // Parse other context types into per-artifact segments for accurate smart referencing.
+    // Each XML block gets its own segment with a content-derived ID (URL, path, etc.)
+    // so L2/L3 deduplication works at the individual artifact level.
+    this.appendParsedSegments(turnSegments, params.tagContextAddition, {
       source: "tags",
       notePaths: params.tagNotePaths,
     });
-    this.appendTurnContextSegment(turnSegments, "folders", params.folderContextAddition, {
+    this.appendParsedSegments(turnSegments, params.folderContextAddition, {
       source: "folders",
       notePaths: params.folderNotePaths,
     });
-    this.appendTurnContextSegment(turnSegments, "urls", params.urlContext, {
-      source: "urls",
-    });
-    this.appendTurnContextSegment(turnSegments, "selected_text", params.selectedText, {
-      source: "selected_text",
-    });
-    this.appendTurnContextSegment(turnSegments, "web_tabs", params.webTabContext, {
-      source: "web_tabs",
-    });
+    this.appendParsedSegments(turnSegments, params.urlContext);
+    this.appendParsedSegments(turnSegments, params.selectedText);
+    this.appendParsedSegments(turnSegments, params.webTabContext);
 
     if (turnSegments.length > 0) {
       layerSegments.L3_TURN = turnSegments;
@@ -620,78 +600,46 @@ export class ContextManager {
   }
 
   /**
-   * Parse context XML string into individual segments (one per note/context item)
-   * Extracts <note_context>, <active_note>, and <prior_context> blocks and creates segments with path as ID
+   * Parse context XML string into individual segments (one per context item).
+   * Delegates to the standalone parseContextIntoSegments function.
    */
   private parseContextIntoSegments(contextXml: string, stable: boolean): PromptLayerSegment[] {
-    if (!contextXml.trim()) {
-      return [];
-    }
-
-    const segments: PromptLayerSegment[] = [];
-
-    // Match <note_context>, <active_note>, and <prior_context> blocks
-    // prior_context has attributes: <prior_context source="path" type="note">
-    const noteContextRegex =
-      /<(?:note_context|active_note)>[\s\S]*?<\/(?:note_context|active_note)>/g;
-    const priorContextRegex = /<prior_context\s+source="([^"]+)"[^>]*>[\s\S]*?<\/prior_context>/g;
-
-    let match: RegExpExecArray | null;
-
-    // Parse original note blocks
-    while ((match = noteContextRegex.exec(contextXml)) !== null) {
-      const block = match[0];
-      const pathMatch = /<path>([^<]+)<\/path>/.exec(block);
-      if (!pathMatch) continue;
-
-      const notePath = pathMatch[1];
-      segments.push({
-        id: notePath,
-        content: block,
-        stable,
-        metadata: {
-          source: stable ? "previous_turns" : "current_turn",
-          notePath,
-        },
-      });
-    }
-
-    // Parse compacted prior_context blocks (L2 from previous turns)
-    while ((match = priorContextRegex.exec(contextXml)) !== null) {
-      const block = match[0];
-      const source = match[1]; // Extracted from source="..." attribute
-
-      segments.push({
-        id: source,
-        content: block,
-        stable,
-        metadata: {
-          source: "previous_turns_compacted",
-          notePath: source,
-        },
-      });
-    }
-
-    return segments;
+    return parseContextIntoSegments(contextXml, stable);
   }
 
-  private appendTurnContextSegment(
+  /**
+   * Parse context XML into individual per-artifact segments and append to target.
+   * Each XML block gets its own segment with a content-derived ID (URL, path, etc.)
+   * from the registry. Extra metadata (e.g., notePaths for tags/folders) is merged
+   * into each resulting segment.
+   */
+  private appendParsedSegments(
     target: PromptLayerSegment[],
-    segmentId: string,
     content: string,
-    metadata: Record<string, unknown>
+    extraMetadata?: Record<string, unknown>
   ) {
     const normalized = (content || "").trim();
     if (!normalized) {
       return;
     }
 
-    target.push({
-      id: `${segmentId}`,
-      content: normalized,
-      stable: false,
-      metadata,
-    });
+    const segments = this.parseContextIntoSegments(normalized, false);
+    if (segments.length > 0) {
+      for (const seg of segments) {
+        if (extraMetadata) {
+          seg.metadata = { ...seg.metadata, ...extraMetadata };
+        }
+        target.push(seg);
+      }
+    } else {
+      // Fallback: content didn't parse into known XML blocks, keep as single segment
+      target.push({
+        id: `unparsed-${Date.now()}`,
+        content: normalized,
+        stable: false,
+        metadata: { source: "unparsed", ...extraMetadata },
+      });
+    }
   }
 
   /**

src/mentions/Mention.ts

@@ -160,7 +160,7 @@ export class Mention {
 
       if (urlData.processed) {
         if (result.type === "youtube") {
-          urlContext += `\n\n<youtube_transcript>\n<url>${urlData.original}</url>\n<transcript>\n${urlData.processed}\n</transcript>\n</youtube_transcript>`;
+          urlContext += `\n\n<youtube_video_context>\n<url>${urlData.original}</url>\n<content>\n${urlData.processed}\n</content>\n</youtube_video_context>`;
         } else if (result.type === "twitter") {
           urlContext += `\n\n<twitter_content>\n<url>${urlData.original}</url>\n<content>\n${urlData.processed}\n</content>\n</twitter_content>`;
         } else {