Support dataview result in active note (#1918)

This commit is contained in:
Logan Yang 2025-10-12 19:05:55 -07:00 committed by GitHub
parent 231d020fea
commit cea72d61b1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 834 additions and 4 deletions

View file

@ -0,0 +1,214 @@
# Dataview Integration Plan
## Goal
Enable Copilot to see the rendered results of Dataview blocks when notes containing them are included as context (active note or context notes).
## Complexity Assessment
**Medium Complexity** - This is achievable but requires careful integration with the Dataview plugin API.
### Key Requirements
1. **Dataview Plugin Integration**: Access Dataview's public API to execute queries
2. **Block Detection & Parsing**: Identify Dataview blocks in markdown (`dataview, `dataviewjs, inline queries)
3. **Result Rendering**: Convert Dataview's output (tables, lists, task lists) into LLM-readable text
4. **Async Handling**: Dataview queries can be async and may take time to execute
## Implementation Plan
### Phase 1: Dataview Detection & API Access
**Location**: New utility file or within `contextProcessor.ts`
**Tasks**:
- Check if Dataview plugin is installed and enabled via `app.plugins.plugins['dataview']`
- Access Dataview's public API
- Create graceful fallback if Dataview is not available
- Document Dataview API methods we'll use
**Code Pattern**:
```typescript
const dataviewPlugin = app.plugins.plugins["dataview"];
if (!dataviewPlugin) {
// Fallback: return content as-is
return content;
}
const dataviewApi = dataviewPlugin.api;
```
### Phase 2: Block Detection
**Location**: `src/contextProcessor.ts` - new method
**Method Signature**:
```typescript
async processDataviewBlocks(
content: string,
sourcePath: string,
vault: Vault
): Promise<string>
```
**Detect**:
1. **Codeblock queries**:
- ` ```dataview` ... ` ``` `
- ` ```dataviewjs` ... ` ``` `
2. **Inline queries**:
- `` `= dv.pages()...` ``
- `` `$= ...` ``
**Detection Strategy**:
- Use regex to find all Dataview blocks
- Extract query content and type
- Preserve surrounding content structure
### Phase 3: Query Execution & Result Formatting
**For each detected Dataview block**:
1. **Execute the query** using Dataview API
2. **Format results** based on output type:
- **Tables** → Markdown tables or structured text
- **Lists** → Bulleted/numbered lists
- **Tasks** → Task lists with [ ] and [x] status
- **Single values** → Plain text
- **Calendar/Timeline** → Descriptive text representation
**Output Format**:
```xml
<dataview_block>
<query_type>dataview|dataviewjs</query_type>
<original_query>
[original query code]
</original_query>
<executed_result>
[formatted results]
</executed_result>
</dataview_block>
```
### Phase 4: Integration into Context Processing
**Location**: `src/contextProcessor.ts:109` - within `processNote` function
**Integration Point**:
```typescript
// Current code
let content = await fileParserManager.parseFile(note, vault);
// NEW: Process Dataview blocks if present
if (note.extension === "md") {
content = await this.processDataviewBlocks(content, note.path, vault);
}
// Special handling for embedded PDFs within markdown (only in Plus mode)
if (note.extension === "md" && isPlusChain(currentChain)) {
content = await this.processEmbeddedPDFs(content, vault, fileParserManager);
}
```
### Phase 5: Error Handling
**Handle gracefully**:
- Dataview plugin not installed → Skip processing
- Malformed queries → Show error message + original block
- Slow queries → Implement timeout (e.g., 5 seconds per query)
- Query execution errors → Show error + original block
- Empty results → Indicate "No results found"
**Error Format**:
```xml
<dataview_block>
<query_type>dataview</query_type>
<original_query>
[original query code]
</original_query>
<error>Query execution failed: [error message]</error>
</dataview_block>
```
## Estimated Effort
- **Core implementation**: 100-150 lines of code
- **Testing**: 2-3 hours across different query types
- **Documentation**: Update CLAUDE.md with Dataview support notes
- **Total time**: 4-6 hours
## Key Challenges
### 1. Dataview API Structure
- Need to understand Dataview's public API methods
- Different methods for different query types (DQL vs DataviewJS)
- API may differ across Dataview versions
### 2. Result Formatting
- Converting complex nested data structures into readable text
- DataviewJS can return arbitrary JavaScript objects
- Maintaining readability for LLMs
### 3. Performance
- Dataview queries can be slow on large vaults
- Multiple queries in one note could cause delays
- Need timeout mechanisms to prevent blocking
### 4. Query Context
- Dataview queries execute in the context of the source note
- Need to pass correct file path for `this.file` references
- Date-based queries need current date context
## Testing Checklist
- [ ] Basic LIST query
- [ ] Basic TABLE query
- [ ] TASK query with completion status
- [ ] DataviewJS query
- [ ] Inline queries (`` `= ...` ``)
- [ ] Multiple Dataview blocks in one note
- [ ] Queries with filters and sorting
- [ ] Queries referencing `this.file`
- [ ] Empty result sets
- [ ] Malformed queries (error handling)
- [ ] Performance with slow queries
- [ ] Behavior when Dataview plugin is disabled
## Benefits
1. **Enhanced Context Awareness**: Users can ask questions about dynamically generated content
2. **No UI Changes Required**: Works seamlessly with existing Copilot features
3. **Transparent Integration**: Users don't need to change their workflow
4. **Better Insights**: LLM can understand aggregated data, not just raw notes
## Future Enhancements
1. **Query Result Caching**: Cache results for repeated queries in same session
2. **Selective Processing**: Allow users to enable/disable Dataview processing via settings
3. **Result Summarization**: For very large result sets, provide summaries instead of full output
4. **Real-time Updates**: Reflect Dataview query updates when vault changes during chat session
## Related Files
- `src/contextProcessor.ts` - Main integration point
- `src/core/ContextManager.ts` - Orchestrates context processing
- `src/tools/FileParserManager.ts` - File parsing utilities
- `docs/CLAUDE.md` - Documentation updates needed
## References
- [Dataview Plugin API Documentation](https://blacksmithgu.github.io/obsidian-dataview/)
- [Obsidian Plugin API](https://github.com/obsidianmd/obsidian-api)

View file

@ -96,6 +96,7 @@ export const SELECTED_TEXT_TAG = "selected_text";
export const VARIABLE_TAG = "variable";
export const VARIABLE_NOTE_TAG = "variable_note";
export const EMBEDDED_PDF_TAG = "embedded_pdf";
export const DATAVIEW_BLOCK_TAG = "dataview_block";
export const VAULT_NOTE_TAG = "vault_note";
export const RETRIEVED_DOCUMENT_TAG = "retrieved_document";
export const EMPTY_INDEX_ERROR_MESSAGE =

View file

@ -0,0 +1,442 @@
// Mock chainFactory before importing anything else to avoid import errors
jest.mock("@/chainFactory", () => ({
ChainType: {
LLM_CHAIN: "llm_chain",
VAULT_QA_CHAIN: "vault_qa",
COPILOT_PLUS_CHAIN: "copilot_plus",
PROJECT_CHAIN: "project",
},
}));
import { ContextProcessor } from "@/contextProcessor";
import { DATAVIEW_BLOCK_TAG } from "@/constants";
// Mock the global app object for Dataview plugin access
global.app = {
plugins: {
plugins: {},
},
} as any;
describe("ContextProcessor - Dataview Integration", () => {
let contextProcessor: ContextProcessor;
let originalConsoleError: typeof console.error;
beforeEach(() => {
contextProcessor = ContextProcessor.getInstance();
// Reset plugins for each test
(global.app as any).plugins.plugins = {};
// Save and mock console.error to suppress expected error messages
originalConsoleError = console.error;
console.error = jest.fn();
});
afterEach(() => {
// Restore original console.error
console.error = originalConsoleError;
});
describe("processDataviewBlocks - Plugin Availability", () => {
it("should return content unchanged when Dataview plugin is not installed", async () => {
const content = "```dataview\nLIST\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toBe(content);
});
it("should return content unchanged when Dataview API is not available", async () => {
(global.app as any).plugins.plugins.dataview = {};
const content = "```dataview\nLIST\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toBe(content);
});
});
describe("processDataviewBlocks - Regex Pattern Matching", () => {
beforeEach(() => {
// Mock successful Dataview plugin with API
(global.app as any).plugins.plugins.dataview = {
api: {
query: jest.fn().mockResolvedValue({
successful: true,
value: {
type: "list",
values: [{ path: "note1.md" }, { path: "note2.md" }],
},
}),
},
};
});
it("should match dataview blocks with exact newline", async () => {
const content = "```dataview\nLIST\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain(`<${DATAVIEW_BLOCK_TAG}>`);
expect(result).toContain("<query_type>dataview</query_type>");
});
it("should match dataview blocks with trailing spaces", async () => {
const content = "```dataview \nLIST\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain(`<${DATAVIEW_BLOCK_TAG}>`);
expect(result).toContain("<query_type>dataview</query_type>");
});
it("should match dataview blocks with tabs", async () => {
const content = "```dataview\t\nLIST\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain(`<${DATAVIEW_BLOCK_TAG}>`);
expect(result).toContain("<query_type>dataview</query_type>");
});
it("should match dataview blocks with Windows CRLF line endings", async () => {
const content = "```dataview\r\nLIST\r\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain(`<${DATAVIEW_BLOCK_TAG}>`);
expect(result).toContain("<query_type>dataview</query_type>");
});
it("should match dataviewjs blocks", async () => {
const content = "```dataviewjs\ndv.list()\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain(`<${DATAVIEW_BLOCK_TAG}>`);
expect(result).toContain("<query_type>dataviewjs</query_type>");
});
});
describe("processDataviewBlocks - Multiple Blocks", () => {
beforeEach(() => {
(global.app as any).plugins.plugins.dataview = {
api: {
query: jest
.fn()
.mockResolvedValueOnce({
successful: true,
value: {
type: "list",
values: [{ path: "note1.md" }],
},
})
.mockResolvedValueOnce({
successful: true,
value: {
type: "list",
values: [{ path: "note2.md" }],
},
}),
},
};
});
it("should process multiple different dataview blocks", async () => {
const content = `
# First Query
\`\`\`dataview
LIST WHERE tag = "#project"
\`\`\`
# Second Query
\`\`\`dataview
TABLE file.name
\`\`\`
`;
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
// Should contain two dataview block tags
const blockMatches = result.match(new RegExp(`<${DATAVIEW_BLOCK_TAG}>`, "g"));
expect(blockMatches).toHaveLength(2);
});
it("should process multiple identical dataview blocks correctly", async () => {
const content = `
# First Instance
\`\`\`dataview
LIST
\`\`\`
# Second Instance
\`\`\`dataview
LIST
\`\`\`
`;
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
// Should contain two dataview block tags
const blockMatches = result.match(new RegExp(`<${DATAVIEW_BLOCK_TAG}>`, "g"));
expect(blockMatches).toHaveLength(2);
// Both should have different results (since mock returns different values)
expect(result).toContain("[[note1.md]]");
expect(result).toContain("[[note2.md]]");
});
});
describe("processDataviewBlocks - Query Execution", () => {
it("should include both original query and executed results", async () => {
(global.app as any).plugins.plugins.dataview = {
api: {
query: jest.fn().mockResolvedValue({
successful: true,
value: {
type: "list",
values: [{ path: "note1.md" }],
},
}),
},
};
const content = '```dataview\nLIST WHERE contains(tags, "#project")\n```';
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain("<original_query>");
expect(result).toContain('LIST WHERE contains(tags, "#project")');
expect(result).toContain("<executed_result>");
expect(result).toContain("[[note1.md]]");
});
it("should handle query timeout gracefully", async () => {
(global.app as any).plugins.plugins.dataview = {
api: {
query: jest.fn().mockImplementation(
() =>
new Promise((resolve) => {
// Never resolve to simulate timeout
setTimeout(resolve, 10000);
})
),
},
};
const content = "```dataview\nLIST\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain("<error>Query timeout</error>");
}, 10000); // Increase test timeout to 10s
it("should handle query execution errors", async () => {
(global.app as any).plugins.plugins.dataview = {
api: {
query: jest.fn().mockResolvedValue({
successful: false,
error: "Invalid syntax",
}),
},
};
const content = "```dataview\nINVALID QUERY\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain("<error>Invalid syntax</error>");
});
it("should handle dataviewjs with unsupported message", async () => {
(global.app as any).plugins.plugins.dataview = {
api: {
query: jest.fn(),
},
};
const content = "```dataviewjs\ndv.pages()\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain("DataviewJS execution not yet supported");
});
});
describe("processDataviewBlocks - Result Formatting", () => {
it("should format LIST results correctly", async () => {
(global.app as any).plugins.plugins.dataview = {
api: {
query: jest.fn().mockResolvedValue({
successful: true,
value: {
type: "list",
values: [{ path: "note1.md" }, { path: "note2.md" }, { path: "note3.md" }],
},
}),
},
};
const content = "```dataview\nLIST\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain("- [[note1.md]]");
expect(result).toContain("- [[note2.md]]");
expect(result).toContain("- [[note3.md]]");
});
it("should format TABLE results correctly", async () => {
(global.app as any).plugins.plugins.dataview = {
api: {
query: jest.fn().mockResolvedValue({
successful: true,
value: {
type: "table",
headers: ["File", "Size"],
values: [
[{ path: "note1.md" }, 100],
[{ path: "note2.md" }, 200],
],
},
}),
},
};
const content = "```dataview\nTABLE file.size\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain("| File | Size |");
expect(result).toContain("| --- | --- |");
expect(result).toContain("| [[note1.md]] | 100 |");
expect(result).toContain("| [[note2.md]] | 200 |");
});
it("should format TASK results correctly", async () => {
(global.app as any).plugins.plugins.dataview = {
api: {
query: jest.fn().mockResolvedValue({
successful: true,
value: {
type: "task",
values: [
{ text: "Task 1", completed: false },
{ text: "Task 2", completed: true },
],
},
}),
},
};
const content = "```dataview\nTASK\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain("- [ ] Task 1");
expect(result).toContain("- [x] Task 2");
});
it("should handle empty results", async () => {
(global.app as any).plugins.plugins.dataview = {
api: {
query: jest.fn().mockResolvedValue({
successful: true,
value: {
type: "list",
values: [],
},
}),
},
};
const content = "```dataview\nLIST WHERE false\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain("<executed_result>\nNo results\n</executed_result>");
});
it("should handle null values gracefully", async () => {
(global.app as any).plugins.plugins.dataview = {
api: {
query: jest.fn().mockResolvedValue({
successful: true,
value: {
type: "list",
values: [null, { path: "note1.md" }, undefined],
},
}),
},
};
const content = "```dataview\nLIST\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
// Should skip null/undefined values
expect(result).toContain("- ");
expect(result).toContain("- [[note1.md]]");
});
it("should handle arrays in values", async () => {
(global.app as any).plugins.plugins.dataview = {
api: {
query: jest.fn().mockResolvedValue({
successful: true,
value: {
type: "list",
values: [[{ path: "note1.md" }, { path: "note2.md" }]],
},
}),
},
};
const content = "```dataview\nLIST\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain("[[note1.md]], [[note2.md]]");
});
});
describe("processDataviewBlocks - Edge Cases", () => {
beforeEach(() => {
(global.app as any).plugins.plugins.dataview = {
api: {
query: jest.fn().mockResolvedValue({
successful: true,
value: {
type: "list",
values: [],
},
}),
},
};
});
it("should not process non-dataview code blocks", async () => {
const content = "```javascript\nconst x = 1;\n```";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toBe(content);
expect(result).not.toContain(`<${DATAVIEW_BLOCK_TAG}>`);
});
it("should handle content with no dataview blocks", async () => {
const content = "Just some regular markdown content.";
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toBe(content);
});
it("should preserve surrounding content", async () => {
const content = `
# Before
Some text before
\`\`\`dataview
LIST
\`\`\`
Some text after
# After
`;
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain("# Before");
expect(result).toContain("Some text before");
expect(result).toContain("Some text after");
expect(result).toContain("# After");
expect(result).toContain(`<${DATAVIEW_BLOCK_TAG}>`);
});
it("should handle multiline queries", async () => {
const content = `\`\`\`dataview
LIST
WHERE contains(tags, "#project")
AND file.mtime > date(today) - dur(7 days)
SORT file.mtime DESC
\`\`\``;
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
expect(result).toContain("<original_query>");
expect(result).toContain('WHERE contains(tags, "#project")');
expect(result).toContain("AND file.mtime > date(today) - dur(7 days)");
expect(result).toContain("SORT file.mtime DESC");
});
});
});

View file

@ -4,7 +4,12 @@ import { RESTRICTION_MESSAGES } from "@/constants";
import { FileParserManager } from "@/tools/FileParserManager";
import { isPlusChain } from "@/utils";
import { TFile, Vault, Notice } from "obsidian";
import { NOTE_CONTEXT_PROMPT_TAG, EMBEDDED_PDF_TAG, SELECTED_TEXT_TAG } from "./constants";
import {
NOTE_CONTEXT_PROMPT_TAG,
EMBEDDED_PDF_TAG,
SELECTED_TEXT_TAG,
DATAVIEW_BLOCK_TAG,
} from "./constants";
export class ContextProcessor {
private static instance: ContextProcessor;
@ -49,6 +54,168 @@ export class ContextProcessor {
return content;
}
/**
* Process Dataview blocks in content, executing queries and replacing them with structured results
*/
async processDataviewBlocks(content: string, sourcePath: string): Promise<string> {
// Check if Dataview plugin is available
const dataviewPlugin = (app as any).plugins?.plugins?.dataview;
if (!dataviewPlugin) {
return content; // Dataview not installed, return content as-is
}
const dataviewApi = dataviewPlugin.api;
if (!dataviewApi) {
return content; // API not available
}
// Match dataview and dataviewjs code blocks
// Fixed regex: \s* handles trailing spaces and different line endings
const blockRegex = /```(dataview|dataviewjs)\s*\n([\s\S]*?)```/g;
const matches = [...content.matchAll(blockRegex)];
// Process matches in reverse order to avoid position shifts when replacing
// This also handles multiple identical blocks correctly
for (let i = matches.length - 1; i >= 0; i--) {
const match = matches[i];
const queryType = match[1]; // 'dataview' or 'dataviewjs'
const query = match[2].trim();
const matchStart = match.index!;
const matchEnd = matchStart + match[0].length;
try {
// Execute query with timeout
const result = await Promise.race([
this.executeDataviewQuery(dataviewApi, query, queryType, sourcePath),
new Promise((_, reject) => setTimeout(() => reject(new Error("Query timeout")), 5000)),
]);
// Replace block with structured output using slice (position-based, handles duplicates)
const replacement = `\n\n<${DATAVIEW_BLOCK_TAG}>\n<query_type>${queryType}</query_type>\n<original_query>\n${query}\n</original_query>\n<executed_result>\n${result}\n</executed_result>\n</${DATAVIEW_BLOCK_TAG}>\n\n`;
content = content.slice(0, matchStart) + replacement + content.slice(matchEnd);
} catch (error) {
console.error(`Error executing Dataview query:`, error);
// On error, include query with error message
const replacement = `\n\n<${DATAVIEW_BLOCK_TAG}>\n<query_type>${queryType}</query_type>\n<original_query>\n${query}\n</original_query>\n<error>${error instanceof Error ? error.message : "Query execution failed"}</error>\n</${DATAVIEW_BLOCK_TAG}>\n\n`;
content = content.slice(0, matchStart) + replacement + content.slice(matchEnd);
}
}
return content;
}
/**
* Execute a Dataview query and format the results
*/
private async executeDataviewQuery(
dataviewApi: any,
query: string,
queryType: string,
sourcePath: string
): Promise<string> {
if (queryType === "dataviewjs") {
// DataviewJS requires more complex handling - for now, return a message
return "[DataviewJS execution not yet supported - showing original query]";
}
// Parse and execute DQL query
const result = await dataviewApi.query(query, sourcePath);
if (!result.successful) {
throw new Error(result.error || "Query failed");
}
// Format results based on type
return this.formatDataviewResult(result.value);
}
/**
* Format Dataview query results into readable text
*/
private formatDataviewResult(result: any): string {
if (!result) {
return "No results";
}
// Handle different result types
if (result.type === "list") {
return this.formatDataviewList(result.values);
} else if (result.type === "table") {
return this.formatDataviewTable(result.headers, result.values);
} else if (result.type === "task") {
return this.formatDataviewTasks(result.values);
} else if (Array.isArray(result)) {
return result.map((item) => this.formatDataviewValue(item)).join("\n");
}
return String(result);
}
/**
* Format Dataview list results
*/
private formatDataviewList(values: any[]): string {
if (!values || values.length === 0) {
return "No results";
}
return values.map((item) => `- ${this.formatDataviewValue(item)}`).join("\n");
}
/**
* Format Dataview table results
*/
private formatDataviewTable(headers: string[], rows: any[][]): string {
if (!rows || rows.length === 0) {
return "No results";
}
// Create markdown table
let table = `| ${headers.join(" | ")} |\n`;
table += `| ${headers.map(() => "---").join(" | ")} |\n`;
for (const row of rows) {
table += `| ${row.map((cell) => this.formatDataviewValue(cell)).join(" | ")} |\n`;
}
return table;
}
/**
* Format Dataview task results
*/
private formatDataviewTasks(tasks: any[]): string {
if (!tasks || tasks.length === 0) {
return "No results";
}
return tasks
.map((task) => {
const checkbox = task.completed ? "[x]" : "[ ]";
return `- ${checkbox} ${this.formatDataviewValue(task.text || task)}`;
})
.join("\n");
}
/**
* Format individual Dataview values
*/
private formatDataviewValue(value: any): string {
if (value === null || value === undefined) {
return "";
}
// Handle links
if (value && typeof value === "object" && value.path) {
return `[[${value.path}]]`;
}
// Handle arrays
if (Array.isArray(value)) {
return value.map((v) => this.formatDataviewValue(v)).join(", ");
}
return String(value);
}
/**
* Processes context notes, excluding any already handled by custom prompts.
*
@ -108,9 +275,15 @@ export class ContextProcessor {
// 3. If we reach here, parse the file (md, canvas, or other supported type in Plus mode)
let content = await fileParserManager.parseFile(note, vault);
// Special handling for embedded PDFs within markdown (only in Plus mode)
if (note.extension === "md" && isPlusChain(currentChain)) {
content = await this.processEmbeddedPDFs(content, vault, fileParserManager);
// Special handling for markdown files
if (note.extension === "md") {
// Process embedded PDFs (only in Plus mode)
if (isPlusChain(currentChain)) {
content = await this.processEmbeddedPDFs(content, vault, fileParserManager);
}
// Process Dataview blocks (all modes)
content = await this.processDataviewBlocks(content, note.path);
}
// Get file metadata