aaronsb_obsidian-mcp-plugin/wiki/Design.md

Obsidian MCP Plugin - Architecture Design

ADR-001: Bases Architecture Modernization

Status: Proposed
Date: 2025-08-26
Requirement IDs: Performance optimization, architectural compliance, extensibility requirements

Context

The current Bases implementation, while functional, exhibits several architectural anti-patterns that limit scalability, maintainability, and performance:

Current Architecture Issues

1. Monolithic BasesAPI Class: 440+ lines handling file parsing, query execution, export formatting, and property extraction
2. Tight Coupling: ExpressionEvaluator directly depends on Obsidian concrete types
3. Performance Bottlenecks: Full vault scans for every query, no caching, synchronous processing
4. Security Concerns: Direct eval() usage in expression evaluation without proper sandboxing
5. Limited Extensibility: Hard-coded export formats and view types

SOLID Violations

• SRP: BasesAPI has multiple responsibilities (file ops, queries, formatting)
• OCP: Adding new view types or export formats requires modifying existing code
• ISP: No separation between different operational concerns
• DIP: Direct dependencies on Obsidian concrete implementations

Decision

Implement a layered architecture with clear separation of concerns following SOLID principles:

Layer 1: Core Interfaces (Domain Layer)

interface IBaseRepository {
  findAll(): Promise<BaseConfig[]>;
  findByPath(path: string): Promise<BaseConfig>;
  create(config: BaseConfig): Promise<void>;
  update(path: string, config: BaseConfig): Promise<void>;
  delete(path: string): Promise<void>;
}

interface IQueryEngine {
  execute(query: BaseQuery): Promise<QueryResult>;
  count(query: BaseQuery): Promise<number>;
  explain(query: BaseQuery): QueryPlan;
}

interface IViewRenderer {
  render(result: QueryResult, config: ViewConfig): Promise<RenderedView>;
  supports(viewType: string): boolean;
}

interface IExportStrategy {
  export(result: QueryResult): Promise<string>;
  supports(format: string): boolean;
}

Layer 2: Service Layer

class BaseService {
  constructor(
    private repository: IBaseRepository,
    private queryEngine: IQueryEngine,
    private viewRenderers: Map<string, IViewRenderer>,
    private exportStrategies: Map<string, IExportStrategy>
  ) {}

  async listBases(): Promise<BaseInfo[]> { /* delegates to repository */ }
  async queryBase(path: string, viewName?: string): Promise<BaseQueryResult> { /* orchestrates query + view */ }
  async exportBase(path: string, format: string): Promise<string> { /* orchestrates query + export */ }
}

Layer 3: Infrastructure Layer

class ObsidianBaseRepository implements IBaseRepository {
  constructor(private api: ObsidianAPI) {}
  // Direct integration with vault operations
}

class IndexedQueryEngine implements IQueryEngine {
  constructor(
    private indexService: IIndexService,
    private evaluator: IExpressionEvaluator
  ) {}
  // Optimized query execution with indexing
}

Consequences

Positive

• Testability: Each component can be unit tested in isolation
• Performance: Dedicated query engine with indexing and caching
• Extensibility: New view types and export formats via plugin pattern
• Security: Sandboxed expression evaluation with controlled context
• Maintainability: Clear separation of concerns following SOLID principles

Negative

• Complexity: More interfaces and classes to manage
• Initial Development Cost: Significant refactoring required
• Learning Curve: Team needs to understand layered architecture

Neutral

• Memory Usage: Slight increase due to additional abstraction layers
• Bundle Size: Minimal impact on plugin size

Alternatives Considered

1. Incremental Refactoring: Gradually improve existing code
   • Rejected: Doesn't address fundamental architectural issues
2. External Query Library: Use existing database/query libraries
   • Rejected: Adds dependencies and doesn't integrate well with Obsidian patterns
3. Event-Driven Architecture: Use pub/sub for loose coupling
   • Deferred: Would add complexity without clear benefits for current use cases

Implementation Notes

Phase 1: Core Architecture (2-3 weeks)

• Extract interfaces and core domain types
• Implement repository pattern for base file operations
• Create basic query engine with filtering support

Phase 2: Performance Optimization (2 weeks)

• Add indexing service for frequently queried properties
• Implement result caching with cache invalidation
• Add query optimization and execution planning

Phase 3: Extensibility (1-2 weeks)

• Plugin system for view renderers and export strategies
• Security hardening of expression evaluation
• Advanced query features (aggregation, joins)

---

ADR-002: Expression Evaluation Security Model

Status: Proposed
Date: 2025-08-26
Requirement IDs: Security hardening, safe expression evaluation

Context

Current expression evaluation uses new Function() with dynamic code generation, creating potential security vulnerabilities. The evaluation context has access to the full Obsidian API and file system.

Decision

Implement a sandboxed expression evaluation system:

interface IExpressionEvaluator {
  evaluate(expression: string, context: EvaluationContext): Promise<any>;
  validate(expression: string): ValidationResult;
}

class SandboxedExpressionEvaluator implements IExpressionEvaluator {
  private allowedGlobals: Set<string>;
  private functionWhitelist: Set<string>;
  
  constructor() {
    this.allowedGlobals = new Set(['Math', 'Date', 'String', 'Number']);
    this.functionWhitelist = new Set(['date', 'now', 'today', 'iff', 'choice']);
  }
  
  async evaluate(expression: string, context: EvaluationContext): Promise<any> {
    // Parse AST and validate against whitelist
    const ast = this.parseExpression(expression);
    this.validateAST(ast);
    
    // Execute in controlled environment
    return this.executeInSandbox(ast, context);
  }
}

Security Boundaries

1. AST-based evaluation: Parse expressions into Abstract Syntax Tree for validation
2. Function whitelist: Only allow pre-approved functions and operations
3. Context isolation: Limit access to necessary properties only
4. Resource limits: Prevent infinite loops and excessive memory usage

Consequences

Positive

• Security: Eliminates code injection vulnerabilities
• Reliability: Prevents expressions from crashing the plugin
• Predictability: Clear boundaries on what expressions can do

Negative

• Performance: AST parsing adds overhead compared to direct eval
• Functionality: Some advanced expressions may not be possible

---

ADR-003: Query Engine Architecture

Status: Proposed
Date: 2025-08-26
Requirement IDs: Performance optimization, scalability

Context

Current query implementation scans all files for every query, leading to O(n) performance that doesn't scale with large vaults. No caching or indexing strategy exists.

Decision

Implement a multi-tier query engine with indexing and caching:

interface IIndexService {
  createIndex(property: string): Promise<void>;
  dropIndex(property: string): Promise<void>;
  getIndexedValues(property: string): Promise<Set<any>>;
  findByIndexedValue(property: string, value: any): Promise<string[]>;
}

class QueryEngine implements IQueryEngine {
  constructor(
    private indexService: IIndexService,
    private cacheService: ICacheService
  ) {}

  async execute(query: BaseQuery): Promise<QueryResult> {
    // 1. Check cache for identical query
    const cacheKey = this.generateCacheKey(query);
    const cached = await this.cacheService.get(cacheKey);
    if (cached) return cached;

    // 2. Optimize query using available indexes
    const optimizedQuery = this.optimizeQuery(query);
    
    // 3. Execute optimized query
    const result = await this.executeOptimizedQuery(optimizedQuery);
    
    // 4. Cache result with TTL
    await this.cacheService.set(cacheKey, result, { ttl: 300000 }); // 5 min
    
    return result;
  }

  private optimizeQuery(query: BaseQuery): OptimizedQuery {
    // Use indexes for equality filters
    // Push down filters to reduce dataset early
    // Optimize sort operations using indexed properties
  }
}

Performance Strategies

1. Property Indexing: Build indexes for frequently queried properties
2. Query Result Caching: Cache expensive query results with invalidation
3. Filter Pushdown: Apply filters as early as possible in query pipeline
4. Incremental Updates: Only re-process changed files on vault updates

Consequences

Positive

• Performance: O(log n) query performance for indexed properties
• Scalability: Handles large vaults efficiently
• Responsiveness: Cached results provide instant responses

Negative

• Memory Usage: Indexes and caches consume additional memory
• Complexity: Index management and cache invalidation logic
• Startup Time: Initial index building may slow plugin initialization

---

ADR-004: Integration with ObsidianAPI Abstraction

Status: Proposed
Date: 2025-08-26
Requirement IDs: Architecture consistency, plugin integration

Context

The plugin maintains an ObsidianAPI abstraction layer for all vault operations. The new Bases architecture must integrate seamlessly with this pattern while maintaining the performance benefits of direct API access.

Decision

Extend the ObsidianAPI abstraction with Bases-specific operations while maintaining the facade pattern:

// ObsidianAPI Extensions
interface IObsidianAPI {
  // Existing methods...
  
  // Bases operations
  getBases(): Promise<BaseInfo[]>;
  getBase(path: string): Promise<BaseConfig>;
  createBase(path: string, config: BaseConfig): Promise<void>;
  updateBase(path: string, config: BaseConfig): Promise<void>;
  deleteBase(path: string): Promise<void>;
  
  // Optimized data access for Bases
  getFileMetadataCache(): MetadataCache;
  getFilesByTag(tag: string): TFile[];
  getFilesByFolder(folder: string): TFile[];
  getFileFrontmatter(file: TFile): Record<string, any>;
}

// Dependency Injection Setup
class BasesModule {
  static create(api: IObsidianAPI): BaseService {
    const repository = new ObsidianBaseRepository(api);
    const indexService = new ObsidianIndexService(api);
    const cacheService = new MemoryCacheService();
    const queryEngine = new IndexedQueryEngine(indexService, cacheService);
    
    const viewRenderers = new Map([
      ['table', new TableViewRenderer()],
      ['cards', new CardViewRenderer()],
      ['list', new ListViewRenderer()]
    ]);
    
    const exportStrategies = new Map([
      ['csv', new CSVExportStrategy()],
      ['json', new JSONExportStrategy()],
      ['markdown', new MarkdownExportStrategy()]
    ]);
    
    return new BaseService(repository, queryEngine, viewRenderers, exportStrategies);
  }
}

Consequences

Positive

• Consistency: Maintains established architectural patterns
• Testability: Easy to mock ObsidianAPI for testing
• Performance: Direct access to Obsidian APIs where needed
• Flexibility: Can switch between HTTP and direct API implementations

Negative

• Additional Abstraction: One more layer between Bases logic and Obsidian
• API Surface: ObsidianAPI interface grows larger

Implementation Notes

• Preserve exact method signatures from current implementation for backward compatibility
• Add performance monitoring to identify bottlenecks during migration
• Implement feature flags to enable gradual rollout of new architecture
• Maintain fallback to current implementation if issues arise

---

Migration Strategy

Phase 1: Foundation (Week 1-2)

1. Extract core interfaces and domain types
2. Implement repository pattern
3. Create basic service layer
4. Maintain existing API compatibility

Phase 2: Performance (Week 3-4)

1. Add indexing service
2. Implement caching layer
3. Optimize query execution
4. Add performance metrics

Phase 3: Extensibility (Week 5-6)

1. Plugin system for views and exports
2. Security hardening
3. Advanced query features
4. Complete testing coverage

Testing Strategy

• Unit Tests: Each service and component in isolation
• Integration Tests: End-to-end query workflows
• Performance Tests: Benchmark against current implementation
• Security Tests: Validate expression evaluation safety

Risk Mitigation

• Feature Flags: Enable gradual rollout
• Fallback Mechanism: Revert to current implementation if needed
• Performance Monitoring: Track metrics during migration
• User Feedback: Beta testing with BRAT users

This architectural redesign addresses all identified issues while maintaining backward compatibility and establishing a foundation for future enhancements.