mirror of
https://github.com/logancyang/obsidian-copilot.git
synced 2026-07-22 07:50:24 +00:00
Support dataview result in active note (#1918)
This commit is contained in:
parent
231d020fea
commit
cea72d61b1
4 changed files with 834 additions and 4 deletions
214
docs/DATAVIEW_INTEGRATION_PLAN.md
Normal file
214
docs/DATAVIEW_INTEGRATION_PLAN.md
Normal file
|
|
@ -0,0 +1,214 @@
|
|||
# Dataview Integration Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Enable Copilot to see the rendered results of Dataview blocks when notes containing them are included as context (active note or context notes).
|
||||
|
||||
## Complexity Assessment
|
||||
|
||||
**Medium Complexity** - This is achievable but requires careful integration with the Dataview plugin API.
|
||||
|
||||
### Key Requirements
|
||||
|
||||
1. **Dataview Plugin Integration**: Access Dataview's public API to execute queries
|
||||
2. **Block Detection & Parsing**: Identify Dataview blocks in markdown (`dataview, `dataviewjs, inline queries)
|
||||
3. **Result Rendering**: Convert Dataview's output (tables, lists, task lists) into LLM-readable text
|
||||
4. **Async Handling**: Dataview queries can be async and may take time to execute
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Dataview Detection & API Access
|
||||
|
||||
**Location**: New utility file or within `contextProcessor.ts`
|
||||
|
||||
**Tasks**:
|
||||
|
||||
- Check if Dataview plugin is installed and enabled via `app.plugins.plugins['dataview']`
|
||||
- Access Dataview's public API
|
||||
- Create graceful fallback if Dataview is not available
|
||||
- Document Dataview API methods we'll use
|
||||
|
||||
**Code Pattern**:
|
||||
|
||||
```typescript
|
||||
const dataviewPlugin = app.plugins.plugins["dataview"];
|
||||
if (!dataviewPlugin) {
|
||||
// Fallback: return content as-is
|
||||
return content;
|
||||
}
|
||||
const dataviewApi = dataviewPlugin.api;
|
||||
```
|
||||
|
||||
### Phase 2: Block Detection
|
||||
|
||||
**Location**: `src/contextProcessor.ts` - new method
|
||||
|
||||
**Method Signature**:
|
||||
|
||||
```typescript
|
||||
async processDataviewBlocks(
|
||||
content: string,
|
||||
sourcePath: string,
|
||||
vault: Vault
|
||||
): Promise<string>
|
||||
```
|
||||
|
||||
**Detect**:
|
||||
|
||||
1. **Codeblock queries**:
|
||||
|
||||
- ` ```dataview` ... ` ``` `
|
||||
- ` ```dataviewjs` ... ` ``` `
|
||||
|
||||
2. **Inline queries**:
|
||||
- `` `= dv.pages()...` ``
|
||||
- `` `$= ...` ``
|
||||
|
||||
**Detection Strategy**:
|
||||
|
||||
- Use regex to find all Dataview blocks
|
||||
- Extract query content and type
|
||||
- Preserve surrounding content structure
|
||||
|
||||
### Phase 3: Query Execution & Result Formatting
|
||||
|
||||
**For each detected Dataview block**:
|
||||
|
||||
1. **Execute the query** using Dataview API
|
||||
2. **Format results** based on output type:
|
||||
- **Tables** → Markdown tables or structured text
|
||||
- **Lists** → Bulleted/numbered lists
|
||||
- **Tasks** → Task lists with [ ] and [x] status
|
||||
- **Single values** → Plain text
|
||||
- **Calendar/Timeline** → Descriptive text representation
|
||||
|
||||
**Output Format**:
|
||||
|
||||
```xml
|
||||
<dataview_block>
|
||||
<query_type>dataview|dataviewjs</query_type>
|
||||
<original_query>
|
||||
[original query code]
|
||||
</original_query>
|
||||
<executed_result>
|
||||
[formatted results]
|
||||
</executed_result>
|
||||
</dataview_block>
|
||||
```
|
||||
|
||||
### Phase 4: Integration into Context Processing
|
||||
|
||||
**Location**: `src/contextProcessor.ts:109` - within `processNote` function
|
||||
|
||||
**Integration Point**:
|
||||
|
||||
```typescript
|
||||
// Current code
|
||||
let content = await fileParserManager.parseFile(note, vault);
|
||||
|
||||
// NEW: Process Dataview blocks if present
|
||||
if (note.extension === "md") {
|
||||
content = await this.processDataviewBlocks(content, note.path, vault);
|
||||
}
|
||||
|
||||
// Special handling for embedded PDFs within markdown (only in Plus mode)
|
||||
if (note.extension === "md" && isPlusChain(currentChain)) {
|
||||
content = await this.processEmbeddedPDFs(content, vault, fileParserManager);
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 5: Error Handling
|
||||
|
||||
**Handle gracefully**:
|
||||
|
||||
- Dataview plugin not installed → Skip processing
|
||||
- Malformed queries → Show error message + original block
|
||||
- Slow queries → Implement timeout (e.g., 5 seconds per query)
|
||||
- Query execution errors → Show error + original block
|
||||
- Empty results → Indicate "No results found"
|
||||
|
||||
**Error Format**:
|
||||
|
||||
```xml
|
||||
<dataview_block>
|
||||
<query_type>dataview</query_type>
|
||||
<original_query>
|
||||
[original query code]
|
||||
</original_query>
|
||||
<error>Query execution failed: [error message]</error>
|
||||
</dataview_block>
|
||||
```
|
||||
|
||||
## Estimated Effort
|
||||
|
||||
- **Core implementation**: 100-150 lines of code
|
||||
- **Testing**: 2-3 hours across different query types
|
||||
- **Documentation**: Update CLAUDE.md with Dataview support notes
|
||||
- **Total time**: 4-6 hours
|
||||
|
||||
## Key Challenges
|
||||
|
||||
### 1. Dataview API Structure
|
||||
|
||||
- Need to understand Dataview's public API methods
|
||||
- Different methods for different query types (DQL vs DataviewJS)
|
||||
- API may differ across Dataview versions
|
||||
|
||||
### 2. Result Formatting
|
||||
|
||||
- Converting complex nested data structures into readable text
|
||||
- DataviewJS can return arbitrary JavaScript objects
|
||||
- Maintaining readability for LLMs
|
||||
|
||||
### 3. Performance
|
||||
|
||||
- Dataview queries can be slow on large vaults
|
||||
- Multiple queries in one note could cause delays
|
||||
- Need timeout mechanisms to prevent blocking
|
||||
|
||||
### 4. Query Context
|
||||
|
||||
- Dataview queries execute in the context of the source note
|
||||
- Need to pass correct file path for `this.file` references
|
||||
- Date-based queries need current date context
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
- [ ] Basic LIST query
|
||||
- [ ] Basic TABLE query
|
||||
- [ ] TASK query with completion status
|
||||
- [ ] DataviewJS query
|
||||
- [ ] Inline queries (`` `= ...` ``)
|
||||
- [ ] Multiple Dataview blocks in one note
|
||||
- [ ] Queries with filters and sorting
|
||||
- [ ] Queries referencing `this.file`
|
||||
- [ ] Empty result sets
|
||||
- [ ] Malformed queries (error handling)
|
||||
- [ ] Performance with slow queries
|
||||
- [ ] Behavior when Dataview plugin is disabled
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Enhanced Context Awareness**: Users can ask questions about dynamically generated content
|
||||
2. **No UI Changes Required**: Works seamlessly with existing Copilot features
|
||||
3. **Transparent Integration**: Users don't need to change their workflow
|
||||
4. **Better Insights**: LLM can understand aggregated data, not just raw notes
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
1. **Query Result Caching**: Cache results for repeated queries in same session
|
||||
2. **Selective Processing**: Allow users to enable/disable Dataview processing via settings
|
||||
3. **Result Summarization**: For very large result sets, provide summaries instead of full output
|
||||
4. **Real-time Updates**: Reflect Dataview query updates when vault changes during chat session
|
||||
|
||||
## Related Files
|
||||
|
||||
- `src/contextProcessor.ts` - Main integration point
|
||||
- `src/core/ContextManager.ts` - Orchestrates context processing
|
||||
- `src/tools/FileParserManager.ts` - File parsing utilities
|
||||
- `docs/CLAUDE.md` - Documentation updates needed
|
||||
|
||||
## References
|
||||
|
||||
- [Dataview Plugin API Documentation](https://blacksmithgu.github.io/obsidian-dataview/)
|
||||
- [Obsidian Plugin API](https://github.com/obsidianmd/obsidian-api)
|
||||
|
|
@ -96,6 +96,7 @@ export const SELECTED_TEXT_TAG = "selected_text";
|
|||
export const VARIABLE_TAG = "variable";
|
||||
export const VARIABLE_NOTE_TAG = "variable_note";
|
||||
export const EMBEDDED_PDF_TAG = "embedded_pdf";
|
||||
export const DATAVIEW_BLOCK_TAG = "dataview_block";
|
||||
export const VAULT_NOTE_TAG = "vault_note";
|
||||
export const RETRIEVED_DOCUMENT_TAG = "retrieved_document";
|
||||
export const EMPTY_INDEX_ERROR_MESSAGE =
|
||||
|
|
|
|||
442
src/contextProcessor.dataview.test.ts
Normal file
442
src/contextProcessor.dataview.test.ts
Normal file
|
|
@ -0,0 +1,442 @@
|
|||
// Mock chainFactory before importing anything else to avoid import errors
|
||||
jest.mock("@/chainFactory", () => ({
|
||||
ChainType: {
|
||||
LLM_CHAIN: "llm_chain",
|
||||
VAULT_QA_CHAIN: "vault_qa",
|
||||
COPILOT_PLUS_CHAIN: "copilot_plus",
|
||||
PROJECT_CHAIN: "project",
|
||||
},
|
||||
}));
|
||||
|
||||
import { ContextProcessor } from "@/contextProcessor";
|
||||
import { DATAVIEW_BLOCK_TAG } from "@/constants";
|
||||
|
||||
// Mock the global app object for Dataview plugin access
|
||||
global.app = {
|
||||
plugins: {
|
||||
plugins: {},
|
||||
},
|
||||
} as any;
|
||||
|
||||
describe("ContextProcessor - Dataview Integration", () => {
|
||||
let contextProcessor: ContextProcessor;
|
||||
let originalConsoleError: typeof console.error;
|
||||
|
||||
beforeEach(() => {
|
||||
contextProcessor = ContextProcessor.getInstance();
|
||||
// Reset plugins for each test
|
||||
(global.app as any).plugins.plugins = {};
|
||||
// Save and mock console.error to suppress expected error messages
|
||||
originalConsoleError = console.error;
|
||||
console.error = jest.fn();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
// Restore original console.error
|
||||
console.error = originalConsoleError;
|
||||
});
|
||||
|
||||
describe("processDataviewBlocks - Plugin Availability", () => {
|
||||
it("should return content unchanged when Dataview plugin is not installed", async () => {
|
||||
const content = "```dataview\nLIST\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
expect(result).toBe(content);
|
||||
});
|
||||
|
||||
it("should return content unchanged when Dataview API is not available", async () => {
|
||||
(global.app as any).plugins.plugins.dataview = {};
|
||||
const content = "```dataview\nLIST\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
expect(result).toBe(content);
|
||||
});
|
||||
});
|
||||
|
||||
describe("processDataviewBlocks - Regex Pattern Matching", () => {
|
||||
beforeEach(() => {
|
||||
// Mock successful Dataview plugin with API
|
||||
(global.app as any).plugins.plugins.dataview = {
|
||||
api: {
|
||||
query: jest.fn().mockResolvedValue({
|
||||
successful: true,
|
||||
value: {
|
||||
type: "list",
|
||||
values: [{ path: "note1.md" }, { path: "note2.md" }],
|
||||
},
|
||||
}),
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
it("should match dataview blocks with exact newline", async () => {
|
||||
const content = "```dataview\nLIST\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
expect(result).toContain(`<${DATAVIEW_BLOCK_TAG}>`);
|
||||
expect(result).toContain("<query_type>dataview</query_type>");
|
||||
});
|
||||
|
||||
it("should match dataview blocks with trailing spaces", async () => {
|
||||
const content = "```dataview \nLIST\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
expect(result).toContain(`<${DATAVIEW_BLOCK_TAG}>`);
|
||||
expect(result).toContain("<query_type>dataview</query_type>");
|
||||
});
|
||||
|
||||
it("should match dataview blocks with tabs", async () => {
|
||||
const content = "```dataview\t\nLIST\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
expect(result).toContain(`<${DATAVIEW_BLOCK_TAG}>`);
|
||||
expect(result).toContain("<query_type>dataview</query_type>");
|
||||
});
|
||||
|
||||
it("should match dataview blocks with Windows CRLF line endings", async () => {
|
||||
const content = "```dataview\r\nLIST\r\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
expect(result).toContain(`<${DATAVIEW_BLOCK_TAG}>`);
|
||||
expect(result).toContain("<query_type>dataview</query_type>");
|
||||
});
|
||||
|
||||
it("should match dataviewjs blocks", async () => {
|
||||
const content = "```dataviewjs\ndv.list()\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
expect(result).toContain(`<${DATAVIEW_BLOCK_TAG}>`);
|
||||
expect(result).toContain("<query_type>dataviewjs</query_type>");
|
||||
});
|
||||
});
|
||||
|
||||
describe("processDataviewBlocks - Multiple Blocks", () => {
|
||||
beforeEach(() => {
|
||||
(global.app as any).plugins.plugins.dataview = {
|
||||
api: {
|
||||
query: jest
|
||||
.fn()
|
||||
.mockResolvedValueOnce({
|
||||
successful: true,
|
||||
value: {
|
||||
type: "list",
|
||||
values: [{ path: "note1.md" }],
|
||||
},
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
successful: true,
|
||||
value: {
|
||||
type: "list",
|
||||
values: [{ path: "note2.md" }],
|
||||
},
|
||||
}),
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
it("should process multiple different dataview blocks", async () => {
|
||||
const content = `
|
||||
# First Query
|
||||
\`\`\`dataview
|
||||
LIST WHERE tag = "#project"
|
||||
\`\`\`
|
||||
|
||||
# Second Query
|
||||
\`\`\`dataview
|
||||
TABLE file.name
|
||||
\`\`\`
|
||||
`;
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
// Should contain two dataview block tags
|
||||
const blockMatches = result.match(new RegExp(`<${DATAVIEW_BLOCK_TAG}>`, "g"));
|
||||
expect(blockMatches).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("should process multiple identical dataview blocks correctly", async () => {
|
||||
const content = `
|
||||
# First Instance
|
||||
\`\`\`dataview
|
||||
LIST
|
||||
\`\`\`
|
||||
|
||||
# Second Instance
|
||||
\`\`\`dataview
|
||||
LIST
|
||||
\`\`\`
|
||||
`;
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
// Should contain two dataview block tags
|
||||
const blockMatches = result.match(new RegExp(`<${DATAVIEW_BLOCK_TAG}>`, "g"));
|
||||
expect(blockMatches).toHaveLength(2);
|
||||
|
||||
// Both should have different results (since mock returns different values)
|
||||
expect(result).toContain("[[note1.md]]");
|
||||
expect(result).toContain("[[note2.md]]");
|
||||
});
|
||||
});
|
||||
|
||||
describe("processDataviewBlocks - Query Execution", () => {
|
||||
it("should include both original query and executed results", async () => {
|
||||
(global.app as any).plugins.plugins.dataview = {
|
||||
api: {
|
||||
query: jest.fn().mockResolvedValue({
|
||||
successful: true,
|
||||
value: {
|
||||
type: "list",
|
||||
values: [{ path: "note1.md" }],
|
||||
},
|
||||
}),
|
||||
},
|
||||
};
|
||||
|
||||
const content = '```dataview\nLIST WHERE contains(tags, "#project")\n```';
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
expect(result).toContain("<original_query>");
|
||||
expect(result).toContain('LIST WHERE contains(tags, "#project")');
|
||||
expect(result).toContain("<executed_result>");
|
||||
expect(result).toContain("[[note1.md]]");
|
||||
});
|
||||
|
||||
it("should handle query timeout gracefully", async () => {
|
||||
(global.app as any).plugins.plugins.dataview = {
|
||||
api: {
|
||||
query: jest.fn().mockImplementation(
|
||||
() =>
|
||||
new Promise((resolve) => {
|
||||
// Never resolve to simulate timeout
|
||||
setTimeout(resolve, 10000);
|
||||
})
|
||||
),
|
||||
},
|
||||
};
|
||||
|
||||
const content = "```dataview\nLIST\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
expect(result).toContain("<error>Query timeout</error>");
|
||||
}, 10000); // Increase test timeout to 10s
|
||||
|
||||
it("should handle query execution errors", async () => {
|
||||
(global.app as any).plugins.plugins.dataview = {
|
||||
api: {
|
||||
query: jest.fn().mockResolvedValue({
|
||||
successful: false,
|
||||
error: "Invalid syntax",
|
||||
}),
|
||||
},
|
||||
};
|
||||
|
||||
const content = "```dataview\nINVALID QUERY\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
expect(result).toContain("<error>Invalid syntax</error>");
|
||||
});
|
||||
|
||||
it("should handle dataviewjs with unsupported message", async () => {
|
||||
(global.app as any).plugins.plugins.dataview = {
|
||||
api: {
|
||||
query: jest.fn(),
|
||||
},
|
||||
};
|
||||
|
||||
const content = "```dataviewjs\ndv.pages()\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
expect(result).toContain("DataviewJS execution not yet supported");
|
||||
});
|
||||
});
|
||||
|
||||
describe("processDataviewBlocks - Result Formatting", () => {
|
||||
it("should format LIST results correctly", async () => {
|
||||
(global.app as any).plugins.plugins.dataview = {
|
||||
api: {
|
||||
query: jest.fn().mockResolvedValue({
|
||||
successful: true,
|
||||
value: {
|
||||
type: "list",
|
||||
values: [{ path: "note1.md" }, { path: "note2.md" }, { path: "note3.md" }],
|
||||
},
|
||||
}),
|
||||
},
|
||||
};
|
||||
|
||||
const content = "```dataview\nLIST\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
expect(result).toContain("- [[note1.md]]");
|
||||
expect(result).toContain("- [[note2.md]]");
|
||||
expect(result).toContain("- [[note3.md]]");
|
||||
});
|
||||
|
||||
it("should format TABLE results correctly", async () => {
|
||||
(global.app as any).plugins.plugins.dataview = {
|
||||
api: {
|
||||
query: jest.fn().mockResolvedValue({
|
||||
successful: true,
|
||||
value: {
|
||||
type: "table",
|
||||
headers: ["File", "Size"],
|
||||
values: [
|
||||
[{ path: "note1.md" }, 100],
|
||||
[{ path: "note2.md" }, 200],
|
||||
],
|
||||
},
|
||||
}),
|
||||
},
|
||||
};
|
||||
|
||||
const content = "```dataview\nTABLE file.size\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
expect(result).toContain("| File | Size |");
|
||||
expect(result).toContain("| --- | --- |");
|
||||
expect(result).toContain("| [[note1.md]] | 100 |");
|
||||
expect(result).toContain("| [[note2.md]] | 200 |");
|
||||
});
|
||||
|
||||
it("should format TASK results correctly", async () => {
|
||||
(global.app as any).plugins.plugins.dataview = {
|
||||
api: {
|
||||
query: jest.fn().mockResolvedValue({
|
||||
successful: true,
|
||||
value: {
|
||||
type: "task",
|
||||
values: [
|
||||
{ text: "Task 1", completed: false },
|
||||
{ text: "Task 2", completed: true },
|
||||
],
|
||||
},
|
||||
}),
|
||||
},
|
||||
};
|
||||
|
||||
const content = "```dataview\nTASK\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
expect(result).toContain("- [ ] Task 1");
|
||||
expect(result).toContain("- [x] Task 2");
|
||||
});
|
||||
|
||||
it("should handle empty results", async () => {
|
||||
(global.app as any).plugins.plugins.dataview = {
|
||||
api: {
|
||||
query: jest.fn().mockResolvedValue({
|
||||
successful: true,
|
||||
value: {
|
||||
type: "list",
|
||||
values: [],
|
||||
},
|
||||
}),
|
||||
},
|
||||
};
|
||||
|
||||
const content = "```dataview\nLIST WHERE false\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
expect(result).toContain("<executed_result>\nNo results\n</executed_result>");
|
||||
});
|
||||
|
||||
it("should handle null values gracefully", async () => {
|
||||
(global.app as any).plugins.plugins.dataview = {
|
||||
api: {
|
||||
query: jest.fn().mockResolvedValue({
|
||||
successful: true,
|
||||
value: {
|
||||
type: "list",
|
||||
values: [null, { path: "note1.md" }, undefined],
|
||||
},
|
||||
}),
|
||||
},
|
||||
};
|
||||
|
||||
const content = "```dataview\nLIST\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
// Should skip null/undefined values
|
||||
expect(result).toContain("- ");
|
||||
expect(result).toContain("- [[note1.md]]");
|
||||
});
|
||||
|
||||
it("should handle arrays in values", async () => {
|
||||
(global.app as any).plugins.plugins.dataview = {
|
||||
api: {
|
||||
query: jest.fn().mockResolvedValue({
|
||||
successful: true,
|
||||
value: {
|
||||
type: "list",
|
||||
values: [[{ path: "note1.md" }, { path: "note2.md" }]],
|
||||
},
|
||||
}),
|
||||
},
|
||||
};
|
||||
|
||||
const content = "```dataview\nLIST\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
expect(result).toContain("[[note1.md]], [[note2.md]]");
|
||||
});
|
||||
});
|
||||
|
||||
describe("processDataviewBlocks - Edge Cases", () => {
|
||||
beforeEach(() => {
|
||||
(global.app as any).plugins.plugins.dataview = {
|
||||
api: {
|
||||
query: jest.fn().mockResolvedValue({
|
||||
successful: true,
|
||||
value: {
|
||||
type: "list",
|
||||
values: [],
|
||||
},
|
||||
}),
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
it("should not process non-dataview code blocks", async () => {
|
||||
const content = "```javascript\nconst x = 1;\n```";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
expect(result).toBe(content);
|
||||
expect(result).not.toContain(`<${DATAVIEW_BLOCK_TAG}>`);
|
||||
});
|
||||
|
||||
it("should handle content with no dataview blocks", async () => {
|
||||
const content = "Just some regular markdown content.";
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
expect(result).toBe(content);
|
||||
});
|
||||
|
||||
it("should preserve surrounding content", async () => {
|
||||
const content = `
|
||||
# Before
|
||||
|
||||
Some text before
|
||||
|
||||
\`\`\`dataview
|
||||
LIST
|
||||
\`\`\`
|
||||
|
||||
Some text after
|
||||
|
||||
# After
|
||||
`;
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
expect(result).toContain("# Before");
|
||||
expect(result).toContain("Some text before");
|
||||
expect(result).toContain("Some text after");
|
||||
expect(result).toContain("# After");
|
||||
expect(result).toContain(`<${DATAVIEW_BLOCK_TAG}>`);
|
||||
});
|
||||
|
||||
it("should handle multiline queries", async () => {
|
||||
const content = `\`\`\`dataview
|
||||
LIST
|
||||
WHERE contains(tags, "#project")
|
||||
AND file.mtime > date(today) - dur(7 days)
|
||||
SORT file.mtime DESC
|
||||
\`\`\``;
|
||||
const result = await contextProcessor.processDataviewBlocks(content, "test.md");
|
||||
|
||||
expect(result).toContain("<original_query>");
|
||||
expect(result).toContain('WHERE contains(tags, "#project")');
|
||||
expect(result).toContain("AND file.mtime > date(today) - dur(7 days)");
|
||||
expect(result).toContain("SORT file.mtime DESC");
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
@ -4,7 +4,12 @@ import { RESTRICTION_MESSAGES } from "@/constants";
|
|||
import { FileParserManager } from "@/tools/FileParserManager";
|
||||
import { isPlusChain } from "@/utils";
|
||||
import { TFile, Vault, Notice } from "obsidian";
|
||||
import { NOTE_CONTEXT_PROMPT_TAG, EMBEDDED_PDF_TAG, SELECTED_TEXT_TAG } from "./constants";
|
||||
import {
|
||||
NOTE_CONTEXT_PROMPT_TAG,
|
||||
EMBEDDED_PDF_TAG,
|
||||
SELECTED_TEXT_TAG,
|
||||
DATAVIEW_BLOCK_TAG,
|
||||
} from "./constants";
|
||||
|
||||
export class ContextProcessor {
|
||||
private static instance: ContextProcessor;
|
||||
|
|
@ -49,6 +54,168 @@ export class ContextProcessor {
|
|||
return content;
|
||||
}
|
||||
|
||||
/**
|
||||
* Process Dataview blocks in content, executing queries and replacing them with structured results
|
||||
*/
|
||||
async processDataviewBlocks(content: string, sourcePath: string): Promise<string> {
|
||||
// Check if Dataview plugin is available
|
||||
const dataviewPlugin = (app as any).plugins?.plugins?.dataview;
|
||||
if (!dataviewPlugin) {
|
||||
return content; // Dataview not installed, return content as-is
|
||||
}
|
||||
|
||||
const dataviewApi = dataviewPlugin.api;
|
||||
if (!dataviewApi) {
|
||||
return content; // API not available
|
||||
}
|
||||
|
||||
// Match dataview and dataviewjs code blocks
|
||||
// Fixed regex: \s* handles trailing spaces and different line endings
|
||||
const blockRegex = /```(dataview|dataviewjs)\s*\n([\s\S]*?)```/g;
|
||||
const matches = [...content.matchAll(blockRegex)];
|
||||
|
||||
// Process matches in reverse order to avoid position shifts when replacing
|
||||
// This also handles multiple identical blocks correctly
|
||||
for (let i = matches.length - 1; i >= 0; i--) {
|
||||
const match = matches[i];
|
||||
const queryType = match[1]; // 'dataview' or 'dataviewjs'
|
||||
const query = match[2].trim();
|
||||
const matchStart = match.index!;
|
||||
const matchEnd = matchStart + match[0].length;
|
||||
|
||||
try {
|
||||
// Execute query with timeout
|
||||
const result = await Promise.race([
|
||||
this.executeDataviewQuery(dataviewApi, query, queryType, sourcePath),
|
||||
new Promise((_, reject) => setTimeout(() => reject(new Error("Query timeout")), 5000)),
|
||||
]);
|
||||
|
||||
// Replace block with structured output using slice (position-based, handles duplicates)
|
||||
const replacement = `\n\n<${DATAVIEW_BLOCK_TAG}>\n<query_type>${queryType}</query_type>\n<original_query>\n${query}\n</original_query>\n<executed_result>\n${result}\n</executed_result>\n</${DATAVIEW_BLOCK_TAG}>\n\n`;
|
||||
content = content.slice(0, matchStart) + replacement + content.slice(matchEnd);
|
||||
} catch (error) {
|
||||
console.error(`Error executing Dataview query:`, error);
|
||||
// On error, include query with error message
|
||||
const replacement = `\n\n<${DATAVIEW_BLOCK_TAG}>\n<query_type>${queryType}</query_type>\n<original_query>\n${query}\n</original_query>\n<error>${error instanceof Error ? error.message : "Query execution failed"}</error>\n</${DATAVIEW_BLOCK_TAG}>\n\n`;
|
||||
content = content.slice(0, matchStart) + replacement + content.slice(matchEnd);
|
||||
}
|
||||
}
|
||||
|
||||
return content;
|
||||
}
|
||||
|
||||
/**
|
||||
* Execute a Dataview query and format the results
|
||||
*/
|
||||
private async executeDataviewQuery(
|
||||
dataviewApi: any,
|
||||
query: string,
|
||||
queryType: string,
|
||||
sourcePath: string
|
||||
): Promise<string> {
|
||||
if (queryType === "dataviewjs") {
|
||||
// DataviewJS requires more complex handling - for now, return a message
|
||||
return "[DataviewJS execution not yet supported - showing original query]";
|
||||
}
|
||||
|
||||
// Parse and execute DQL query
|
||||
const result = await dataviewApi.query(query, sourcePath);
|
||||
|
||||
if (!result.successful) {
|
||||
throw new Error(result.error || "Query failed");
|
||||
}
|
||||
|
||||
// Format results based on type
|
||||
return this.formatDataviewResult(result.value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Format Dataview query results into readable text
|
||||
*/
|
||||
private formatDataviewResult(result: any): string {
|
||||
if (!result) {
|
||||
return "No results";
|
||||
}
|
||||
|
||||
// Handle different result types
|
||||
if (result.type === "list") {
|
||||
return this.formatDataviewList(result.values);
|
||||
} else if (result.type === "table") {
|
||||
return this.formatDataviewTable(result.headers, result.values);
|
||||
} else if (result.type === "task") {
|
||||
return this.formatDataviewTasks(result.values);
|
||||
} else if (Array.isArray(result)) {
|
||||
return result.map((item) => this.formatDataviewValue(item)).join("\n");
|
||||
}
|
||||
|
||||
return String(result);
|
||||
}
|
||||
|
||||
/**
|
||||
* Format Dataview list results
|
||||
*/
|
||||
private formatDataviewList(values: any[]): string {
|
||||
if (!values || values.length === 0) {
|
||||
return "No results";
|
||||
}
|
||||
return values.map((item) => `- ${this.formatDataviewValue(item)}`).join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Format Dataview table results
|
||||
*/
|
||||
private formatDataviewTable(headers: string[], rows: any[][]): string {
|
||||
if (!rows || rows.length === 0) {
|
||||
return "No results";
|
||||
}
|
||||
|
||||
// Create markdown table
|
||||
let table = `| ${headers.join(" | ")} |\n`;
|
||||
table += `| ${headers.map(() => "---").join(" | ")} |\n`;
|
||||
|
||||
for (const row of rows) {
|
||||
table += `| ${row.map((cell) => this.formatDataviewValue(cell)).join(" | ")} |\n`;
|
||||
}
|
||||
|
||||
return table;
|
||||
}
|
||||
|
||||
/**
|
||||
* Format Dataview task results
|
||||
*/
|
||||
private formatDataviewTasks(tasks: any[]): string {
|
||||
if (!tasks || tasks.length === 0) {
|
||||
return "No results";
|
||||
}
|
||||
return tasks
|
||||
.map((task) => {
|
||||
const checkbox = task.completed ? "[x]" : "[ ]";
|
||||
return `- ${checkbox} ${this.formatDataviewValue(task.text || task)}`;
|
||||
})
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Format individual Dataview values
|
||||
*/
|
||||
private formatDataviewValue(value: any): string {
|
||||
if (value === null || value === undefined) {
|
||||
return "";
|
||||
}
|
||||
|
||||
// Handle links
|
||||
if (value && typeof value === "object" && value.path) {
|
||||
return `[[${value.path}]]`;
|
||||
}
|
||||
|
||||
// Handle arrays
|
||||
if (Array.isArray(value)) {
|
||||
return value.map((v) => this.formatDataviewValue(v)).join(", ");
|
||||
}
|
||||
|
||||
return String(value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Processes context notes, excluding any already handled by custom prompts.
|
||||
*
|
||||
|
|
@ -108,9 +275,15 @@ export class ContextProcessor {
|
|||
// 3. If we reach here, parse the file (md, canvas, or other supported type in Plus mode)
|
||||
let content = await fileParserManager.parseFile(note, vault);
|
||||
|
||||
// Special handling for embedded PDFs within markdown (only in Plus mode)
|
||||
if (note.extension === "md" && isPlusChain(currentChain)) {
|
||||
content = await this.processEmbeddedPDFs(content, vault, fileParserManager);
|
||||
// Special handling for markdown files
|
||||
if (note.extension === "md") {
|
||||
// Process embedded PDFs (only in Plus mode)
|
||||
if (isPlusChain(currentChain)) {
|
||||
content = await this.processEmbeddedPDFs(content, vault, fileParserManager);
|
||||
}
|
||||
|
||||
// Process Dataview blocks (all modes)
|
||||
content = await this.processDataviewBlocks(content, note.path);
|
||||
}
|
||||
|
||||
// Get file metadata
|
||||
|
|
|
|||
Loading…
Reference in a new issue