logancyang_obsidian-copilot/designdocs/CONTEXT_ENGINEERING.md

Context Engineering - Layered Prefix System

Table of Contents

1. Purpose
2. First-Principles Goals
3. Current Architecture (Verified)
4. Example Chat Walkthrough
5. Chain Runner Envelope Usage
6. Strengths
7. Known Gaps
8. Improvement Roadmap
9. Testing and Observability
10. References

---

Purpose

The context envelope system is the canonical prompt-construction pipeline for chat turns.

It exists to guarantee:

• reproducible prompt assembly,
• minimal duplication across L2/L3/L4,
• safe compaction behavior on large context,
• and cache-friendly request prefixes for major providers.

This document is an implementation audit and roadmap based on current production code.

---

First-Principles Goals

1. Reproducibility

For the same turn inputs, envelope construction should be deterministic and byte-stable.

2. Token Efficiency

Context artifacts should appear once in canonical form (or as references), not duplicated across layers.

3. Prefix Cache Stability

Stable content should stay in early request tokens (L1/L2) so implicit provider caching has maximal hit rate.

4. Compaction Safety

Compaction must preserve answerability and recovery affordances, especially for non-recoverable context.

5. Persistence Parity

Loading chat history should preserve envelope quality (or deterministically reconstruct it), so behavior after load matches in-session behavior.

---

Current Architecture (Verified)

Layer Definitions

Layer	Current Source	Update Trigger	Stability
L1_SYSTEM	ChatManager.getSystemPromptForMessage()	settings/memory/project changes	High
L2_PREVIOUS	Auto-promoted previous user-turn L3 segments	per user turn	Medium
L3_TURN	Current-turn context artifacts (notes/URLs/tags/folders/etc)	every user turn	Low
L4_STRIP	Deferred in envelope, injected from LangChain memory	every turn	Low
L5_USER	processed user query (templated user text)	every user turn	Lowest

End-to-End Flow

1. ChatManager.sendMessage() creates a user message and resolves L1 system prompt.
2. ContextManager.processMessageContext():
   • builds L2 from previous user messages' stored envelopes,
   • processes current-turn context artifacts,
   • optionally compacts large context,
   • builds PromptContextEnvelope via PromptContextEngine.
3. MessageRepository.updateProcessedText() stores both legacy processedText and contextEnvelope.
4. Chain runners require contextEnvelope, convert with LayerToMessagesConverter, inject L4 from memory, then append tool context into user-side payload.

L2/L3 Smart Referencing

• L2 is now deduplicated by segment ID with last-write-wins content updates and stable first-seen ordering.
• L3 segments whose IDs already exist in L2 are rendered as references; new IDs include full content.
• Segment parsing is centralized in parseContextIntoSegments() using contextBlockRegistry tags.

Tool Placement Model

• System message contains only L1 + L2.
• Tool outputs remain turn-scoped and are prepended to user-side content (CiC ordering).
• This keeps cacheable prefix isolated from tool variability.

Persistence Behavior

• Chat markdown persists message text plus context references ([Context: ...]), not full envelopes.
• On load, messages are restored without contextEnvelope.
• Regeneration now has lazy reprocessing: if envelope is missing, ChatManager.regenerateMessage() reprocesses the target user message before running the chain.
• Continuing chat after load still does not automatically reconstruct historical envelopes for prior turns.

Compaction Stack

• Turn-time compaction (ContextCompactor): map-reduce summarization when total context exceeds threshold.
• L2 carry-forward compaction (compactSegmentForL2 + L2ContextCompactor): deterministic structure+preview compression for promoted previous context.

L4 Memory Behavior

• L4 (chat history) is injected by chain runners from LangChain BufferWindowMemory.
• Only L5 text (bare user message) is saved to memory — context artifacts are NOT included. BaseChainRunner.handleResponse() extracts l5Text from the envelope, falling back to originalMessage or message.
• This prevents duplication: context artifacts already live in L2/L3 via the envelope; baking them into L4 would cause triple-inclusion and waste tokens.
• Assistant responses are compacted at save time by ChatHistoryCompactor (strips tool result XML) before storage. Agent-mode responses save only the final answer, not the full reasoning/tool-call chain.

---

Example Chat Walkthrough

This shows the concrete layer contents across a 3-turn conversation. The user attaches project-spec.md in Turn 1, adds api-docs.md in Turn 2, then drops api-docs.md in Turn 3.

Turn 1: User adds project-spec.md

L1 (System):
  [system prompt + user memory + project instructions]

L2 (Previous Context Library):
  (empty — first turn, no prior context)

L3 (Current Turn Context):
  <note_context>
  <title>project-spec</title>
  <path>project-spec.md</path>
  <content>... full note content ...</content>
  </note_context>
  → Segment ID: "project-spec.md" (NEW — full content included)

L4 (Chat History):
  (empty — first turn)

L5 (User Message):
  "Summarize this"

After Turn 1, BaseChainRunner.handleResponse() saves to memory:

• Input: "Summarize this" (displayText only — no context XML)
• Output: "Here is a summary of the project spec..."

Turn 2: User keeps project-spec.md, adds api-docs.md

L1 (System):
  [system prompt — stable ✅, cache-friendly]

L2 (Previous Context Library):
  <prior_context source="project-spec.md" type="note">
  Structure: project-spec (project-spec.md) | Preview: ...first 200 chars...
  </prior_context>
  → Segment ID: "project-spec.md" (promoted from Turn 1 L3, compacted for L2)

L3 (Current Turn Context):
  Context attached to this message:
  - project-spec.md

  Find them in the Context Library in the system prompt above.

  <note_context>
  <title>api-docs</title>
  <path>docs/api-docs.md</path>
  <content>... full note content ...</content>
  </note_context>
  → "project-spec.md" rendered as REFERENCE (already in L2)
  → "docs/api-docs.md" is NEW — full content included

L4 (Chat History):
  Human: "Summarize this"
  AI: "Here is a summary of the project spec..."
  → Only displayText — no context XML in L4

L5 (User Message):
  "What endpoints does the API support?"

Turn 3: User keeps project-spec.md only (drops api-docs.md)

L1 (System):
  [system prompt — stable ✅]

L2 (Previous Context Library):
  <prior_context source="project-spec.md" type="note">
  Structure: project-spec (project-spec.md) | Preview: ...first 200 chars...
  </prior_context>
  <prior_context source="docs/api-docs.md" type="note">
  Structure: api-docs (docs/api-docs.md) | Preview: ...first 200 chars...
  </prior_context>
  → Both deduplicated by segment ID. "project-spec.md" retains its
    first-seen position; "docs/api-docs.md" added after.
  → L2 is CUMULATIVE and STABLE — cache hit for the prefix ✅

L3 (Current Turn Context):
  Context attached to this message:
  - project-spec.md

  Find them in the Context Library in the system prompt above.
  → "project-spec.md" is a REFERENCE (in L2)
  → "docs/api-docs.md" is NOT referenced (user didn't attach it this turn)
    but it remains in L2 for cache stability and potential follow-up use

L4 (Chat History):
  Human: "Summarize this"
  AI: "Here is a summary of the project spec..."
  Human: "What endpoints does the API support?"
  AI: "The API supports the following endpoints..."
  → Clean displayText only — no bloat

L5 (User Message):
  "Explain the auth flow from the spec"

Key Behaviors Demonstrated

Behavior	Where	Example
Per-artifact segment IDs	L3 parsing	"project-spec.md", "docs/api-docs.md" — not generic "notes"
L2 dedup (last-write-wins)	L2 build	Same ID across turns → content updated, position preserved
Smart referencing	L3 render	Items in L2 become - project-spec.md references
L2 cumulative growth	L2 library	api-docs.md stays in L2 even when dropped from L3
L2 carry-forward compaction	L2 content	Full <note_context> → <prior_context> with structure+preview
L4 displayText only	Memory save	"Summarize this" — no <note_context> XML
Prefix cache stability	L1+L2	L1 stable across turns; L2 grows monotonically, doesn't shrink

---

Chain Runner Envelope Usage

All four chain runners use the context envelope for LLM message construction. Each delegates final response handling to BaseChainRunner.handleResponse(), which saves only L5 text (expanded user query, no context XML) to L4 memory.

Per-Runner Behavior

Runner	Envelope Construction	Tool Results	User Message Source
LLMChainRunner	LayerToMessagesConverter.convert() → system (L1+L2), user (L3 refs + L5)	None	Envelope only
CopilotPlusChainRunner	Same converter, then ensureUserQueryLabel adds [User query]: separator	Prepended to user message in CiC order	L5 text via envelope
AutonomousAgentChainRunner	Same converter for initial messages; ReAct loop appends AI + ToolMessages iteratively	Native tool calling — each result is a separate ToolMessage	L5 text via envelope
VaultQAChainRunner	Same converter	Retrieval results via hybrid/lexical retriever	Envelope only

CopilotPlus: Single-Shot Tool Flow

1. Planning phase analyzes L5 text to determine which @commands to execute.
2. Tool results (localSearch, web fetch, etc.) are formatted and prepended to the user message using CiC ordering: [tool results] → [L3 references + L5 with User query label].
3. Single LLM call with the complete message array: [system (L1+L2)] → [L4 history] → [user (tools + L3 + L5)].

Autonomous Agent: ReAct Loop Flow

1. Initial message array built identically to CopilotPlus: [system (L1+L2+tool guidelines)] → [L4 history] → [user (L3 refs + L5)].
2. Model responds with native tool calls (e.g., localSearch, readFile).
3. Each tool result becomes a ToolMessage appended to the growing messages array.
4. localSearch results get CiC ordering: the user's question (from L5 originalUserPrompt) is appended after the search payload via ensureCiCOrderingWithQuestion.
5. Loop repeats until model responds without tool calls (final answer).

Token Efficiency Audit

Verified efficient (no action needed):

• L1+L2 prefix is stable and cacheable across turns — tool results never enter the system message.
• L3 uses smart references for artifacts already in L2 — no content duplication in the user message.
• L4 contains only displayText (via L5 extraction in handleResponse) — no context XML leakage.
• Chain runners extract L5 text from the envelope for cleanedUserMessage and originalUserPrompt, never using processedText (which contains L2+L3+L5 concatenated).

Known inefficiencies (accepted tradeoffs):

Issue	Severity	Tokens Wasted	Rationale
CiC user-question repeated per localSearch tool call in agent loop	LOW	~50 tokens x N searches	Intentional — each ToolMessage is independent; the model needs the question for grounding across ReAct iterations
L2 content may overlap with localSearch results	MEDIUM	Variable (L2 has compacted preview, search has relevant chunks)	Structural limitation — search doesn't know about L2. Overlap is partial since L2 is compacted to structure+preview while search returns precision chunks
Legacy processedText stores L2+L3+L5 concatenation in MessageRepository	LOW	0 (not sent to LLM)	Storage-only waste. Field is unused by envelope-based chain runners but persisted for backward compatibility

---

Strengths

1. Envelope-first prompt construction is now consistent across all active chain runners.
2. L2 deduplication by artifact ID prevents linear repeated growth for repeated attachments.
3. Segment parsing is centralized and registry-driven, avoiding ad-hoc per-chain parsing logic.
4. Uniform tool placement removes previous cache-boundary ambiguity.
5. Memory-side assistant compaction reduces L4 bloat from tool payloads.
6. Regeneration path is now resilient to missing envelopes on loaded history.

---

Known Gaps

P0: No Token Budget Enforcement on Full Payload

• No compaction mechanism checks the total assembled payload (L1+L2+L3+L4+L5) against any budget. Each compactor guards only its own subset.
• L1 (project context) is never budgeted or compacted: In Projects mode, all project files are concatenated verbatim into L1 with no size limit. This is often the largest single layer.
• Compaction threshold is blind to L1: ContextManager checks L2+L3 against autoCompactThreshold (or hardcoded PROJECT_COMPACT_THRESHOLD = 1M), but L1 size is never subtracted. The threshold acts as if L2+L3 is the entire payload.
• L4 (chat history) has no remaining-budget awareness: loadAndAddChatHistory() loads all history messages regardless of how much budget L1+L2+L3+L5 have already consumed.
• contextTurns is a crude count-based proxy: BufferWindowMemory.k = contextTurns * 2 limits by message count, not token size, providing no actual overflow protection.
• See TOKEN_BUDGET_ENFORCEMENT.md for detailed analysis and fix plan.

P0: Persistence Parity Is Still Incomplete

• Loaded chats do not rehydrate historical envelopes.
• Result: follow-up turns after load do not benefit from historical L2 library unless messages are individually reprocessed.

P0: Non-Deterministic Fallback Segment IDs

• appendParsedSegments() uses unparsed-${Date.now()} when parsing fails.
• This breaks deterministic envelope identity and degrades cache behavior in edge cases.

P1: Parser Still Depends on Regex Over Rendered XML

• parseContextIntoSegments() is significantly better than earlier local regex logic, but it still parses serialized XML strings rather than typed artifacts.
• Malformed/nested edge cases can still create parse misses and fallback behavior.

P1: Compaction Semantics Need Stronger Invariants

• Two compaction modes exist (LLM summarization vs deterministic L2 preview compaction), but explicit invariants on what must remain verbatim are not enforced centrally.
• Non-recoverable context (e.g., selected text) needs stricter protection policies under heavy compaction.

P1: L2 Mutation Tradeoff Not Formalized

• Current policy is ID-dedup + content overwrite.
• This is token-efficient, but mutable artifacts can invalidate long cached prefixes when content changes.
• The system needs an explicit "freshness vs cache stability" policy.

P2: Envelope Metadata Is Underused

• conversationId is currently null.
• Missing stable conversation-level identity weakens observability and future caching strategies.

P2: Documentation Drift Risk

• Prior docs contained stale statements (e.g., fallback-to-processedText behavior, on-load envelope reconstruction).
• This doc now reflects current code; future changes should keep this aligned.

---

Improvement Roadmap

Phase 1: Correctness and Determinism

1. Replace timestamp fallback IDs with deterministic IDs:
   • unparsed:${sha256(content)} (plus optional short prefix by source type).
2. Add envelope invariants checker (debug + tests):
   • no duplicate segment IDs inside a layer,
   • no L3 full-content block when ID exists in L2 unless explicitly marked override,
   • stable layer ordering and hash consistency.
3. Add robust parse-failure telemetry:
   • count parse misses,
   • log failing tag/source metadata,
   • capture hash-only samples in debug mode.

Phase 2: Persistence Parity

1. Add lazy historical envelope reconstruction on first post-load send:
   • reprocess only prior user messages that have context references and missing envelopes,
   • skip URL/web-tab refetch by policy where needed.
2. Optional long-term path:
   • persist compact envelope metadata (or typed artifact snapshots) alongside markdown history for deterministic restoration.

Phase 3: Compaction Safety and Policy

1. Define explicit compaction classes:
   • recoverable artifacts: can be summarized with re-fetch instructions,
   • non-recoverable artifacts: preserve verbatim or bounded extractive compaction only.
2. Add post-compaction validation:
   • each compacted artifact must retain deterministic source identity and recoverability hints.
3. Make compaction strategy configurable by chain type and context source type.

Phase 4: Cache Optimization

1. Split L1 into stable and mutable subsections (for example:
   • static system contract,
   • user memory and project overlays) to reduce unnecessary prefix invalidation.
2. Introduce provider-aware cache hooks (opt-in):
   • Anthropic cache_control,
   • Gemini explicit cache primitives,
   • keep model-agnostic baseline unchanged.
3. Add per-turn prefix hash diff reporting:
   • L1 hash, L2 hash, combined prefix hash,
   • classify why prefix changed (settings, context attach, file change, memory update).

Phase 5: Typed Artifact Pipeline (Strategic)

Move from "render XML then parse XML" to a typed artifact graph:

• ContextProcessor emits typed artifacts directly (artifactKey, sourceType, recoverable, payload, contentHash).
• Envelope stores typed segments as canonical source-of-truth.
• XML remains a rendering format, not parsing substrate.

This is the highest-leverage change for long-term reproducibility and parser robustness.

Phase 6: Context Envelope Integration Test Suite

Build a comprehensive test suite that validates multi-turn envelope behavior without requiring manual UI testing:

1. Multi-turn envelope simulation tests:

   • Simulate 3+ turn conversations with various artifact combinations (notes, URLs, YouTube, PDFs, selected text).
   • Assert correct L2 promotion, dedup, smart referencing, and compaction at each turn.
   • Validate that L4 memory contains only displayText (no context XML leakage).

2. Layer composition snapshot tests:

   • For canonical conversation trajectories, snapshot the full [L1, L2, L3, L4, L5] payload sent to the LLM.
   • Detect unintended regressions in layer ordering, dedup behavior, or content placement.

3. Round-trip persistence tests:

   • Save a conversation to markdown, reload it, send a follow-up turn.
   • Assert that lazy reprocessing reconstructs envelopes and L2 correctly.

4. Edge-case regression tests:

   • Same artifact attached across 5+ turns (dedup stability).
   • Artifact added, removed, re-added (L2 cumulative behavior).
   • Multiple selected_text blocks in same turn (unique ID generation).
   • Malformed XML blocks (graceful fallback, no silent data loss).
   • Very large context triggering compaction (invariants preserved).

5. Property-based tests (optional, aspirational):

   • Generate random artifact sequences and assert envelope invariants hold: no duplicate segment IDs within a layer, L3 references only exist if ID is in L2, L4 never contains XML block tags.

This suite replaces the need for manual multi-turn chat testing in the UI and provides a safety net for all future envelope changes.

---

Testing and Observability

Core Tests to Add/Strengthen

1. Post-load follow-up turn should rebuild/rehydrate envelope behavior deterministically.
2. Deterministic fallback ID behavior (no wall-clock dependence).
3. Property tests for parser with malformed/nested blocks.
4. Compaction invariants:
   • non-recoverable blocks never become unrecoverable summaries without explicit guardrails.
5. Prefix-hash stability tests across common conversation trajectories.

Runtime Metrics (Debug Mode)

• Envelope build time by phase (L2 build, context processing, compaction, render).
• Segment counts per layer and dedup ratio.
• Prefix hash change reason classification.
• Parse-failure count and compacted-context proportion.

---

References

Primary Implementation Files

• src/core/ChatManager.ts
• src/core/ContextManager.ts
• src/context/PromptContextTypes.ts
• src/context/PromptContextEngine.ts
• src/context/parseContextSegments.ts
• src/context/LayerToMessagesConverter.ts
• src/core/MessageRepository.ts
• src/core/ChatPersistenceManager.ts
• src/LLMProviders/chainRunner/LLMChainRunner.ts
• src/LLMProviders/chainRunner/VaultQAChainRunner.ts
• src/LLMProviders/chainRunner/CopilotPlusChainRunner.ts
• src/LLMProviders/chainRunner/AutonomousAgentChainRunner.ts

Related Docs

• designdocs/MESSAGE_ARCHITECTURE.md
• designdocs/TOOLS.md
• designdocs/NATIVE_TOOL_CALLING_MIGRATION.md
• designdocs/todo/TECHDEBT.md
• TODO.md