logancyang_obsidian-copilot/docs/TOKEN_BUDGET_ENFORCEMENT.md

Token Budget Enforcement

Table of Contents

1. Problem Statement
2. Current Compaction Architecture
3. Root Cause Analysis
4. Fix Plan
5. References

---

Problem Statement

The model proxy receives requests with token counts far exceeding the model's context window (e.g., 2.7M tokens sent to a 1M-token Vertex AI model). The plugin's auto-compaction system was expected to prevent this but fails because no compaction mechanism checks the total assembled payload — each compactor guards only its own subset.

ContextWindowExceededError: The input token count (2769478)
exceeds the maximum number of tokens allowed (1048575).

---

Current Compaction Architecture

There are three separate compaction mechanisms in the plugin. None of them enforce a total token budget against the autoCompactThreshold setting.

1. Turn-Time Context Compaction (ContextCompactor)

Where: ContextManager.processMessageContext() (src/core/ContextManager.ts:231-258) When: Every time a user message is processed, before the envelope is built. What it covers: L2 (previous turn context) + L3 (current turn context) combined. What it does NOT cover: L1 (system prompt), L4 (chat history), L5 (user message).

Trigger condition:
  (processedUserMessage + contextPortion).length > autoCompactThreshold * 4

Where:
  autoCompactThreshold = settings.autoCompactThreshold (default: 128,000 tokens)
  charThreshold = 128,000 * 4 = 512,000 chars

When triggered, ContextCompactor.compact() performs map-reduce LLM summarization on individual XML blocks larger than 50k chars. The user message itself is never compacted.

Key limitation: This threshold check measures processedUserMessage + contextPortion (which is L5 + L2 + L3). It does NOT include:

• L1 (system prompt) — typically 2-10k tokens
• L4 (chat history) — potentially hundreds of thousands of tokens

2. L2 Carry-Forward Compaction (L2ContextCompactor)

Where: ContextManager.compactSegmentForL2() (src/core/ContextManager.ts:706-733) When: When previous turn L3 segments are promoted into L2 for the next turn. What it does: Deterministic structure+preview compression (headings + truncated sections). No LLM calls.

This is a per-segment operation that reduces each context artifact to a <prior_context> block with ~500 chars per section. This prevents L2 from growing unbounded as turns accumulate.

3. Chat History Compaction (ChatHistoryCompactor)

Where: MemoryManager.saveContext() (src/LLMProviders/memoryManager.ts:61-72) When: After each assistant response, at memory save time. What it does: Compacts tool results (localSearch, readNote, etc.) in assistant responses before saving to BufferWindowMemory.

This compacts only the tool-result portions of assistant messages. The rest of the assistant text and all user messages are stored verbatim.

Summary: What Each System Protects

Compaction System	Scope	Token-Aware?	Covers Full Payload?
ContextCompactor (turn-time)	L2 + L3 context XML blocks	Threshold-based (char estimate)	No — misses L1, L4, L5
L2ContextCompactor (carry-forward)	Individual L2 segments	No — fixed per-segment	No — per-segment only
ChatHistoryCompactor (save-time)	Tool results in assistant messages	No — fixed size	No — only tool results

---

Root Cause Analysis

The Core Problem: No Total Payload Budget

The critical gap is systemic: no compaction mechanism checks the total assembled payload (L1+L2+L3+L4+L5) against any budget. Each compactor guards only its own subset, and no final safety net exists.

What L4 Actually Contains

L4 (chat history) is often assumed to be the main token consumer, but investigation shows it is relatively well-controlled:

• User messages in L4 = bare L5 text only (no context XML). BaseChainRunner.handleResponse() extracts l5Text from the envelope and saves only that to memory.
• Assistant responses in L4 = compacted at save time by ChatHistoryCompactor, which strips tool result XML (localSearch, readNote, note_context, etc.).
• Agent-mode responses: AutonomousAgentChainRunner saves only loopResult.finalResponse (the final answer), NOT the full reasoning/tool-call chain.

L4 does grow with conversation length, but it is not unbounded — BufferWindowMemory limits it to k = contextTurns * 2 messages (default: 30), and both user and assistant sides are relatively compact.

The Real Culprits: L1 and Unchecked Layer Accumulation

The overflow happens because multiple layers accumulate without any shared budget:

L1: Project Context Is Never Budgeted

In Projects mode, ChatManager.getSystemPromptForMessage() concatenates all project files, web content, and YouTube transcripts into a <project_context> block inside L1. This can easily reach hundreds of thousands of tokens for large projects.

L1 is never compacted by any system — no compactor even sees it.

Compaction Threshold Is Blind to L1

ContextManager.processMessageContext() uses a hardcoded PROJECT_COMPACT_THRESHOLD = 1,000,000 tokens for Projects mode compaction. This threshold checks only L2+L3 size — it is completely blind to L1 (project context) size. It is set as if L2+L3 is the entire budget, when in reality L1 may have already consumed most of the available context window.

For non-project chains, autoCompactThreshold (default 128k) is used, but it also only checks L2+L3.

L4: No Budget Awareness

loadAndAddChatHistory() loads all history messages without checking how much token budget remains after L1+L2+L3+L5 are assembled:

export async function loadAndAddChatHistory(
  memory: any,
  messages: Array<{ role: string; content: any }>
): Promise<ProcessedMessage[]> {
  const memoryVariables = await memory.loadMemoryVariables({});
  const rawHistory = memoryVariables.history || [];
  // ... processes and adds ALL history messages with NO size check
}

How 2.7M Tokens Happen

In a Projects-mode conversation:

L1 (system + project_context):   ~500k tokens  ← UNBUDGETED, never compacted
L2 (previous context, compacted):  ~20k tokens
L3 (current turn context):         ~50k tokens
                                   ─────────
  ContextCompactor checks L2+L3:    70k < 1,000k threshold → NO compaction triggered
                                   (threshold is blind to 500k in L1)

L4 (15 turns of chat history):    ~200k tokens  ← loaded with no remaining budget check
L5 (user message):                   ~2k tokens
─────────────────────────────────────────────────
TOTAL:                             ~772k tokens  → may exceed model's context window

In extreme cases (large projects + long conversations + heavy context attachments), totals can reach 2M+ tokens.

All Chain Runners Are Affected

All chain runners call loadAndAddChatHistory() without any token budget:

Runner	File	Line
LLMChainRunner	src/LLMProviders/chainRunner/LLMChainRunner.ts	45
CopilotPlusChainRunner	src/LLMProviders/chainRunner/CopilotPlusChainRunner.ts	606
AutonomousAgentChainRunner	src/LLMProviders/chainRunner/AutonomousAgentChainRunner.ts	597
VaultQAChainRunner	src/LLMProviders/chainRunner/VaultQAChainRunner.ts	191

The contextTurns Setting Is a Poor Proxy

BufferWindowMemory is configured with k = contextTurns * 2 (default: 30 messages). This is a crude count-based limit that:

• Has no relation to actual token consumption
• Cannot adapt to varying message sizes
• Provides no guarantees about total payload size

A token-based budget for L4 makes contextTurns redundant.

---

Fix Plan

Guiding Principles

1. Model-agnostic: The plugin supports many LLM providers. No model-specific context window logic. Use autoCompactThreshold (user-configurable) as the single total budget.
2. Single enforcement point: Token budget must be checked where all layers are assembled, not scattered across individual compactors.
3. History guarantee: The LLM must always see at least some recent chat history to resume conversation context, even when L1+L2+L3 consume most of the budget.
4. Graceful degradation: When over budget, drop the least-valuable content first (oldest history turns), then compact further if needed.
5. Backwards compatible: Existing compaction systems remain; this adds a final safety net.
6. No LLM calls in the hot path: Budget enforcement should use fast char-based estimation (chars / 4), not LLM summarization.

Phase 1: Token Budget Guard (Critical Fix)

Goal: Prevent over-budget payloads from ever reaching the LLM.

1.1 Make ContextManager L1-Aware

Currently ContextManager.processMessageContext() checks (L2+L3).length > threshold * 4 where threshold is either autoCompactThreshold or PROJECT_COMPACT_THRESHOLD. Both are blind to L1 size.

Fix: The compaction threshold for L2+L3 must account for L1:

effectiveThreshold = autoCompactThreshold - estimateTokens(L1)

This ensures that when L1 is large (e.g., Projects mode with many files), L2+L3 compaction triggers earlier, leaving room for L4 and L5.

Kill PROJECT_COMPACT_THRESHOLD — it is a hardcoded 1M value that pretends L1 doesn't exist. Replace with the same autoCompactThreshold - L1 formula for all chain types.

File: src/core/ContextManager.ts

1.2 Add Token Budget to loadAndAddChatHistory()

Add an optional tokenBudget parameter to loadAndAddChatHistory(). When provided:

1. Load all history messages from BufferWindowMemory
2. Estimate token count of each message (chars / 4)
3. Drop oldest complete turns (user+assistant pairs) until cumulative total fits within budget
4. Always keep at least the most recent turn (history guarantee)
5. Log a warning when turns are dropped

Token Budget Allocation:
  autoCompactThreshold (e.g., 128,000 tokens)
  - estimateTokens(L1)   system prompt + project context
  - estimateTokens(L2)   previous context library
  - estimateTokens(L3)   current turn context
  - estimateTokens(L5)   user message
  - reservedForOutput    (~4,096 for response generation)
  = remaining budget for L4 chat history

File: src/LLMProviders/chainRunner/utils/chatHistoryUtils.ts

1.3 Update All Chain Runners

Each chain runner calls loadAndAddChatHistory(). Update call sites to:

1. Calculate the token size of already-assembled non-L4 messages (L1+L2+L3+L5)
2. Compute historyBudget = autoCompactThreshold - nonL4Tokens - outputReserve
3. Pass historyBudget to loadAndAddChatHistory()

Files:

• src/LLMProviders/chainRunner/LLMChainRunner.ts
• src/LLMProviders/chainRunner/CopilotPlusChainRunner.ts
• src/LLMProviders/chainRunner/AutonomousAgentChainRunner.ts
• src/LLMProviders/chainRunner/VaultQAChainRunner.ts

1.4 Deprecate contextTurns Setting

Token-based budget trimming makes the count-based contextTurns setting redundant.

• Replace BufferWindowMemory.k = contextTurns * 2 with a generous internal constant (e.g., k = 100)
• The token budget in step 1.2 handles the actual trimming
• Remove the "Conversation turns in context" slider from ModelSettings.tsx

Files:

• src/LLMProviders/memoryManager.ts
• src/settings/v2/components/ModelSettings.tsx

Phase 2: Smarter History Trimming (Enhancement)

Goal: When budget is tight, trim intelligently rather than just dropping oldest turns.

2.1 Prioritized trimming strategy

When over budget, apply in order:

1. Drop oldest complete turns (user+assistant pairs) from L4
2. Truncate remaining long assistant responses in L4 (keep first N chars)
3. If still over budget after L4 is minimized, warn user and proceed — the LLM will still see the most recent turn

Phase 3: Observability (Enhancement)

3.1 Surface token usage to UI

Add a debug/info display showing:

• Estimated tokens per layer (L1, L2, L3, L4, L5)
• Total vs. autoCompactThreshold
• Whether any history turns were dropped

This helps users understand why responses might miss context from earlier turns.

Implementation Order

Step	Description	Files Changed	Risk
1.1	L1-aware compaction threshold	ContextManager.ts	Medium
1.2	Token budget in loadAndAddChatHistory()	chatHistoryUtils.ts	Medium
1.3	Update chain runner call sites	4 chain runner files	Medium
1.4	Deprecate contextTurns	memoryManager.ts, ModelSettings	Low
2.1	Prioritized trimming	chatHistoryUtils.ts	Low
3.1	Token usage debug display	UI components	Low

Phase 1 (steps 1.1-1.4) is the critical fix that prevents the overflow. Phases 2-3 are improvements.

---

References

Source Files

File	Role
src/core/ContextManager.ts	Turn-time compaction trigger (L2+L3)
src/core/ContextCompactor.ts	Map-reduce LLM summarization
src/context/L2ContextCompactor.ts	Deterministic L2 segment compaction
src/context/ChatHistoryCompactor.ts	Tool result compaction at save time
src/LLMProviders/memoryManager.ts	Memory save with compaction
src/LLMProviders/chainRunner/utils/chatHistoryUtils.ts	Chat history loading (no budget)
src/LLMProviders/chainRunner/AutonomousAgentChainRunner.ts	Agent message assembly
src/LLMProviders/chainRunner/CopilotPlusChainRunner.ts	CopilotPlus message assembly
src/LLMProviders/chainRunner/LLMChainRunner.ts	Basic LLM message assembly
src/LLMProviders/chainRunner/VaultQAChainRunner.ts	VaultQA message assembly
src/LLMProviders/chatModelManager.ts	Chat model management
src/constants.ts	Default settings (autoCompactThreshold: 128000)

Related Docs

• CONTEXT_ENGINEERING.md — L1-L5 layer architecture
• MESSAGE_ARCHITECTURE.md — Message flow and storage
• TECHDEBT.md — Known technical debt