taskgenius_taskgenius-plugin/docs/file-source-implementation.md

FileSource Implementation Tracking

Overview

This document tracks the implementation progress of the FileSource feature, which enables files to be recognized as tasks based on their metadata properties. The implementation follows the dataflow architecture patterns and integrates seamlessly with existing functionality.

Task Breakdown

Phase 1: Core Infrastructure (Week 1-2)

TypeScript Definitions

• Create src/types/file-source.d.ts
  • FileSourceTaskMetadata interface
  • FileSourceConfiguration interface
  • RecognitionStrategy interface
  • FileSourceStats interface
  • Recognition strategy type definitions

Configuration Management

• Create src/dataflow/sources/FileSourceConfig.ts
  • Configuration validation logic
  • Default configuration constants
  • Configuration update handlers
  • Strategy-specific configuration management

Basic FileSource Implementation

• Create src/dataflow/sources/FileSource.ts
  • Basic source initialization
  • Event subscription to FILE_UPDATED
  • Granular event subscriptions (FILE_METADATA_CHANGED, FILE_CONTENT_CHANGED)
  • Basic file task detection
  • Smart update detection integration (basic implementation)
  • Selective property updates (updateFileTaskProperties)
  • Selective children updates (updateFileTaskChildren)
  • Event emission for file task updates
  • Basic cleanup and destroy methods

Settings Integration

• Update src/common/setting-definition.ts

  • Add FileSourceConfiguration to TaskProgressBarSettings
  • Update DEFAULT_SETTINGS with FileSource defaults
  • Add validation for FileSource settings

• Create settings UI components

  • src/components/settings/FileSourceSettings.ts
  • Recognition strategy configuration panels
  • File task properties configuration
  • Performance settings interface

Unit Tests

• Create src/__tests__/file-source/
  • FileSourceConfig.test.ts - Configuration validation tests
  • FileSource.basic.test.ts - Basic functionality tests
  • FileSourceUpdateDetector.test.ts - Update detection logic tests (Phase 2)
    • Test metadata change detection
    • Test content change detection
    • Test children structure detection
    • Test hash comparison logic
  • RecognitionStrategies.test.ts - Strategy testing framework (Phase 2)

Phase 2: Recognition Strategies (Week 3-4)

Update Detection System

• Create src/dataflow/sources/FileSourceUpdateDetector.ts
  • Update decision logic implementation
  • Change type analysis (metadata vs content)
  • Children structure change detection
  • Previous state caching mechanism
  • Hash-based property comparison
  • Performance metrics collection

Core Parser Implementation

• Create src/dataflow/parsers/FileSourceParser.ts
  • Base recognition strategy framework
  • Metadata extraction utilities
  • File content analysis helpers
  • Strategy result aggregation
  • Selective parsing for updates

Metadata-Based Recognition

• Implement metadata recognition strategy
  • Frontmatter field checking
  • Required vs optional field logic
  • Metadata validation and parsing
  • Default value assignment

Tag-Based Recognition

• Implement tag recognition strategy
  • Tag pattern matching (exact, prefix, contains)
  • Tag extraction from file cache
  • Multiple tag requirement logic
  • Tag-based metadata extraction

Template-Based Recognition

• Implement template recognition strategy
  • Template path matching
  • Template metadata inheritance
  • Template instance detection
  • Template-specific configuration

Path-Based Recognition

• Implement path recognition strategy
  • Path pattern matching (prefix, regex, glob)
  • Path-based metadata assignment
  • Exclude pattern handling
  • Path hierarchy analysis

Parser Entry Point

• Create src/dataflow/parsers/FileSourceEntry.ts
  • Integration with existing parser architecture
  • Strategy orchestration
  • Result merging and conflict resolution
  • Performance optimization

Strategy Testing

• Comprehensive strategy tests
  • MetadataStrategy.test.ts
  • TagStrategy.test.ts
  • TemplateStrategy.test.ts
  • PathStrategy.test.ts
  • Strategy integration tests

Phase 3: Integration & Augmentation (Week 5-6)

File Task Augmentation

• Create src/dataflow/augment/FileSourceAugmentor.ts
  • File task metadata enhancement
  • Child task relationship building
  • Metadata inheritance implementation
  • Project data integration

Repository Integration

• Update src/dataflow/indexer/Repository.ts
  • File task storage and indexing
  • File task query methods
  • Integration with existing task cache
  • Event handling for file task updates

Event System Integration

• Update src/dataflow/events/Events.ts

  • Add FileSource event types
  • Add granular update events (FILE_METADATA_CHANGED, FILE_CONTENT_CHANGED)
  • Add specific file task events (FILE_TASK_PROPERTY_CHANGED, FILE_TASK_CHILDREN_CHANGED)
  • Add INLINE_TASK_CONTENT_CHANGED event
  • Event payload definitions
  • Sequence tracking for file tasks

• Update src/dataflow/Orchestrator.ts

  • FileSource initialization
  • Event routing for file tasks
  • Integration with existing sources
  • Granular event handling logic

Child Task Relationships

• Implement child task management
  • Parent-child relationship tracking
  • Child task discovery within files
  • Relationship updates on file changes
  • Orphaned task cleanup

Metadata Inheritance System

• Implement inheritance logic
  • Configurable inheritance fields
  • Inheritance priority resolution
  • Update propagation to child tasks
  • Inheritance conflict resolution

Integration Tests

• End-to-end integration testing
  • FileSourceDataflow.test.ts
  • FileTaskRelationships.test.ts
  • MetadataInheritance.test.ts
  • EventFlow.test.ts

Phase 4: Performance & Polish (Week 7-8)

Specialized Indexing

• Create src/dataflow/indexer/FileSourceIndex.ts
  • Optimized file task indexing
  • Fast lookup structures
  • Batch update operations
  • Memory-efficient storage

Worker Integration

• Update worker system for file tasks
  • src/dataflow/workers/FileSourceWorker.ts
  • Background file task processing
  • Worker message protocol
  • Performance monitoring

Caching Implementation

• Create src/dataflow/indexer/FileTaskCache.ts
  • File task state caching structure
  • Frontmatter hash implementation
  • Children ID set management
  • Recognition result caching
  • TTL-based cache invalidation
  • Memory usage optimization
  • Cache performance metrics
  • State comparison utilities

Settings UI Enhancement

• Advanced settings components
  • Strategy priority configuration
  • Custom recognition function editor
  • Performance tuning interface
  • Diagnostic tools

Performance Testing

• Performance benchmark suite
  • Large vault testing (1000+ files)
  • Recognition performance metrics
  • Memory usage profiling
  • Worker performance validation
  • Update detection performance tests
    • Rapid file changes scenario
    • Large file with many tasks
    • Metadata-only updates
    • Content-only updates
    • Mixed update patterns
  • PerformanceOptimization.test.ts
    • Test early exit patterns
    • Test selective processing
    • Test cache hit rates
    • Test event deduplication

Documentation

• User documentation
  • Feature overview guide
  • Configuration examples
  • Troubleshooting guide
  • Best practices document

Phase 5: Advanced Features (Week 9-10)

Custom Recognition Functions

• Implement custom function support
  • JavaScript function parsing
  • Sandboxed execution environment
  • Function validation and testing
  • Error handling and logging

Advanced Conflict Resolution

• Sophisticated conflict handling
  • Dual role (project/task) management
  • Metadata priority rules
  • User-defined resolution strategies
  • Conflict notification system

Bulk Operations

• Bulk file task operations
  • Batch file task creation
  • Bulk metadata updates
  • Batch recognition re-evaluation
  • Progress reporting for bulk ops

Export/Import Functionality

• File task data portability
  • Export file task configurations
  • Import recognition strategies
  • Backup and restore functionality
  • Migration tools

Advanced Filtering

• Enhanced query capabilities
  • File task specific filters
  • Recognition strategy filtering
  • Child task aggregation filters
  • Custom filter expressions

File Structure and Components

New Files to Create

src/
├── types/
│   └── file-source.d.ts                    # Core type definitions
├── dataflow/
│   ├── sources/
│   │   ├── FileSource.ts                   # Main FileSource implementation
│   │   ├── FileSourceConfig.ts             # Configuration management
│   │   └── FileSourceUpdateDetector.ts     # Update detection logic
│   ├── parsers/
│   │   ├── FileSourceEntry.ts              # Parser entry point
│   │   └── FileSourceParser.ts             # Recognition strategies
│   ├── augment/
│   │   └── FileSourceAugmentor.ts          # File task augmentation
│   ├── indexer/
│   │   ├── FileSourceIndex.ts              # Specialized indexing
│   │   └── FileTaskCache.ts                # File task state caching
│   └── workers/
│       └── FileSourceWorker.ts             # Background processing
├── components/
│   └── settings/
│       └── FileSourceSettings.ts           # Settings UI
└── __tests__/
    └── file-source/
        ├── FileSourceConfig.test.ts
        ├── FileSource.basic.test.ts
        ├── FileSourceUpdateDetector.test.ts
        ├── RecognitionStrategies.test.ts
        ├── MetadataStrategy.test.ts
        ├── TagStrategy.test.ts
        ├── TemplateStrategy.test.ts
        ├── PathStrategy.test.ts
        ├── FileSourceDataflow.test.ts
        ├── FileTaskRelationships.test.ts
        ├── MetadataInheritance.test.ts
        ├── EventFlow.test.ts
        └── PerformanceOptimization.test.ts

Files to Modify

src/
├── common/
│   └── setting-definition.ts               # Add FileSource configuration
├── dataflow/
│   ├── events/
│   │   └── Events.ts                       # Add FileSource events and granular update events
│   ├── sources/
│   │   └── ObsidianSource.ts              # Emit granular events (FILE_METADATA_CHANGED, FILE_CONTENT_CHANGED)
│   ├── indexer/
│   │   └── Repository.ts                   # File task integration
│   ├── Orchestrator.ts                     # FileSource initialization
│   └── createDataflow.ts                   # Factory updates
├── components/
│   └── settings/
│       └── TaskProgressBarSettingTab.ts    # Settings integration
└── index.ts                                # Plugin initialization

Dependencies and Integration Points

Internal Dependencies

• Dataflow Architecture: Core event system, repository, orchestrator
• Project System: Resolver, ProjectConfigManager, ProjectDataCache
• Task System: Task types, metadata, parsing infrastructure
• Settings System: Setting definitions, UI components, validation
• Worker System: TaskWorkerManager, worker orchestration

External Dependencies

• Obsidian API: Vault, MetadataCache, TFile, FileSystemAdapter
• File System: Path manipulation, file watching, metadata extraction
• Performance: Worker threads, caching, batch processing

Integration Points

1. Event System: FILE_UPDATED → FileSource → FILE_TASK_UPDATED
2. Repository: File task storage alongside regular tasks
3. QueryAPI: Extended queries for file tasks
4. Views: File tasks displayed in all task views
5. Project System: Dual role management and conflict resolution

Progress Tracking Sections

Week 1-2: Core Infrastructure

Status: ✅ Completed Target Completion: 2025-08-20 Completion: 20/25 tasks (80% - Core functionality complete)

Priority Tasks:

• Core type definitions
• Basic FileSource implementation
• Settings integration
• Initial unit tests

Implementation Notes:

• Core FileSource infrastructure is fully functional
• Metadata and tag-based recognition strategies implemented
• Comprehensive configuration system with validation
• Full test coverage for core components
• TypeScript compilation passing without errors
• Ready for Phase 2 implementation

Week 3-4: Recognition Strategies

Status: ⏳ Not Started Target Completion: [Date] Completion: 0/18 tasks

Priority Tasks:

• Metadata-based recognition
• Tag-based recognition
• Strategy testing framework
• Parser entry point

Week 5-6: Integration & Augmentation

Status: ⏳ Not Started Target Completion: [Date] Completion: 0/22 tasks

Priority Tasks:

• Repository integration
• Event system integration
• Child task relationships
• Metadata inheritance

Week 7-8: Performance & Polish

Status: ⏳ Not Started Target Completion: [Date] Completion: 0/18 tasks

Priority Tasks:

• Worker integration
• Caching implementation
• Performance testing
• User documentation

Week 9-10: Advanced Features

Status: ⏳ Not Started Target Completion: [Date] Completion: 0/15 tasks

Priority Tasks:

• Custom recognition functions
• Bulk operations
• Advanced filtering
• Export/import functionality

Risk Assessment

High Risk Items

• Performance Impact: Large vaults with many files could impact startup time
  • Mitigation: Worker-based processing, selective file scanning, caching
• Memory Usage: File tasks add to memory footprint
  • Mitigation: Efficient indexing, lazy loading, configurable limits
• Conflict Resolution: Dual role files may create complex scenarios
  • Mitigation: Clear precedence rules, user configuration options

Medium Risk Items

• Configuration Complexity: Many options may overwhelm users
  • Mitigation: Sensible defaults, configuration wizard, presets
• Event Loop Performance: Additional event processing overhead
  • Mitigation: Debounced events, batch processing, sequence optimization

Low Risk Items

• Backward Compatibility: New feature shouldn't break existing functionality
  • Mitigation: Feature flag, disabled by default, comprehensive testing

Success Metrics

Functional Metrics

• All recognition strategies working correctly
• File tasks appear in all relevant views
• Child task relationships functioning
• Metadata inheritance working
• Settings interface complete and intuitive

Performance Metrics

• File task detection under 100ms for typical files
• Memory usage increase under 10% for equivalent functionality
• No noticeable impact on plugin startup time
• Worker processing provides measurable performance benefits

Quality Metrics

• 90%+ test coverage for FileSource components
• Zero regression in existing functionality
• User documentation complete and clear
• Configuration examples working as documented

Notes and Decisions

Architecture Decisions

• FileSource as separate source: Maintains clean separation from ObsidianSource
• Event-driven integration: Leverages existing event system for consistency
• Strategy pattern: Allows flexible recognition approaches
• Metadata extension: Extends existing task metadata rather than creating new types

Configuration Decisions

• Disabled by default: Conservative rollout approach
• Multiple strategies: Accommodates different user workflows
• Extensive configuration: Provides flexibility while maintaining usability

Performance Decisions

• Worker integration: Maintains UI responsiveness
• Caching strategy: Balances memory usage with performance
• Selective processing: Avoid processing irrelevant files

---

Last Updated: [Current Date] Document Version: 1.0 Next Review: [Date]