obsidian-plugin/taskgenius_taskgenius-plugin
1
0
Fork 0
mirror of https://github.com/taskgenius/taskgenius-plugin.git synced 2026-07-22 06:40:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
taskgenius_taskgenius-plugin/docs/file-source-implementation.md
Quorafind 738d7aa0c0 docs(filesource): add comprehensive specification and implementation docs 
Add detailed documentation for the FileSource feature including:
- Complete feature specification with architecture alignment
- Implementation tracking document with task breakdown
- Design principles and integration patterns
- Configuration examples and API documentation
- Performance considerations and testing strategies

These documents provide guidance for understanding, maintaining, and
extending the FileSource functionality within the dataflow architecture.
2025-08-20 22:43:43 +08:00

487 lines
No EOL
17 KiB
Markdown
Raw Blame History

# FileSource Implementation Tracking
## Overview
This document tracks the implementation progress of the FileSource feature, which enables files to be recognized as tasks based on their metadata properties. The implementation follows the dataflow architecture patterns and integrates seamlessly with existing functionality.
## Task Breakdown
### Phase 1: Core Infrastructure (Week 1-2)
#### TypeScript Definitions
- [x] Create `src/types/file-source.d.ts`
- [x] `FileSourceTaskMetadata` interface
- [x] `FileSourceConfiguration` interface
- [x] `RecognitionStrategy` interface
- [x] `FileSourceStats` interface
- [x] Recognition strategy type definitions
#### Configuration Management
- [x] Create `src/dataflow/sources/FileSourceConfig.ts`
- [x] Configuration validation logic
- [x] Default configuration constants
- [x] Configuration update handlers
- [x] Strategy-specific configuration management
#### Basic FileSource Implementation
- [x] Create `src/dataflow/sources/FileSource.ts`
- [x] Basic source initialization
- [x] Event subscription to FILE_UPDATED
- [ ] Granular event subscriptions (FILE_METADATA_CHANGED, FILE_CONTENT_CHANGED)
- [x] Basic file task detection
- [x] Smart update detection integration (basic implementation)
- [x] Selective property updates (updateFileTaskProperties)
- [x] Selective children updates (updateFileTaskChildren)
- [x] Event emission for file task updates
- [x] Basic cleanup and destroy methods
#### Settings Integration
- [x] Update `src/common/setting-definition.ts`
- [x] Add FileSourceConfiguration to TaskProgressBarSettings
- [x] Update DEFAULT_SETTINGS with FileSource defaults
- [x] Add validation for FileSource settings
- [x] Create settings UI components
- [x] `src/components/settings/FileSourceSettings.ts`
- [x] Recognition strategy configuration panels
- [x] File task properties configuration
- [x] Performance settings interface
#### Unit Tests
- [x] Create `src/__tests__/file-source/`
- [x] `FileSourceConfig.test.ts` - Configuration validation tests
- [x] `FileSource.basic.test.ts` - Basic functionality tests
- [ ] `FileSourceUpdateDetector.test.ts` - Update detection logic tests (Phase 2)
- [ ] Test metadata change detection
- [ ] Test content change detection
- [ ] Test children structure detection
- [ ] Test hash comparison logic
- [ ] `RecognitionStrategies.test.ts` - Strategy testing framework (Phase 2)
### Phase 2: Recognition Strategies (Week 3-4)
#### Update Detection System
- [ ] Create `src/dataflow/sources/FileSourceUpdateDetector.ts`
- [ ] Update decision logic implementation
- [ ] Change type analysis (metadata vs content)
- [ ] Children structure change detection
- [ ] Previous state caching mechanism
- [ ] Hash-based property comparison
- [ ] Performance metrics collection
#### Core Parser Implementation
- [ ] Create `src/dataflow/parsers/FileSourceParser.ts`
- [ ] Base recognition strategy framework
- [ ] Metadata extraction utilities
- [ ] File content analysis helpers
- [ ] Strategy result aggregation
- [ ] Selective parsing for updates
#### Metadata-Based Recognition
- [ ] Implement metadata recognition strategy
- [ ] Frontmatter field checking
- [ ] Required vs optional field logic
- [ ] Metadata validation and parsing
- [ ] Default value assignment
#### Tag-Based Recognition
- [ ] Implement tag recognition strategy
- [ ] Tag pattern matching (exact, prefix, contains)
- [ ] Tag extraction from file cache
- [ ] Multiple tag requirement logic
- [ ] Tag-based metadata extraction
#### Template-Based Recognition
- [ ] Implement template recognition strategy
- [ ] Template path matching
- [ ] Template metadata inheritance
- [ ] Template instance detection
- [ ] Template-specific configuration
#### Path-Based Recognition
- [ ] Implement path recognition strategy
- [ ] Path pattern matching (prefix, regex, glob)
- [ ] Path-based metadata assignment
- [ ] Exclude pattern handling
- [ ] Path hierarchy analysis
#### Parser Entry Point
- [ ] Create `src/dataflow/parsers/FileSourceEntry.ts`
- [ ] Integration with existing parser architecture
- [ ] Strategy orchestration
- [ ] Result merging and conflict resolution
- [ ] Performance optimization
#### Strategy Testing
- [ ] Comprehensive strategy tests
- [ ] `MetadataStrategy.test.ts`
- [ ] `TagStrategy.test.ts`
- [ ] `TemplateStrategy.test.ts`
- [ ] `PathStrategy.test.ts`
- [ ] Strategy integration tests
### Phase 3: Integration & Augmentation (Week 5-6)
#### File Task Augmentation
- [ ] Create `src/dataflow/augment/FileSourceAugmentor.ts`
- [ ] File task metadata enhancement
- [ ] Child task relationship building
- [ ] Metadata inheritance implementation
- [ ] Project data integration
#### Repository Integration
- [ ] Update `src/dataflow/indexer/Repository.ts`
- [ ] File task storage and indexing
- [ ] File task query methods
- [ ] Integration with existing task cache
- [ ] Event handling for file task updates
#### Event System Integration
- [ ] Update `src/dataflow/events/Events.ts`
- [ ] Add FileSource event types
- [ ] Add granular update events (FILE_METADATA_CHANGED, FILE_CONTENT_CHANGED)
- [ ] Add specific file task events (FILE_TASK_PROPERTY_CHANGED, FILE_TASK_CHILDREN_CHANGED)
- [ ] Add INLINE_TASK_CONTENT_CHANGED event
- [ ] Event payload definitions
- [ ] Sequence tracking for file tasks
- [ ] Update `src/dataflow/Orchestrator.ts`
- [ ] FileSource initialization
- [ ] Event routing for file tasks
- [ ] Integration with existing sources
- [ ] Granular event handling logic
#### Child Task Relationships
- [ ] Implement child task management
- [ ] Parent-child relationship tracking
- [ ] Child task discovery within files
- [ ] Relationship updates on file changes
- [ ] Orphaned task cleanup
#### Metadata Inheritance System
- [ ] Implement inheritance logic
- [ ] Configurable inheritance fields
- [ ] Inheritance priority resolution
- [ ] Update propagation to child tasks
- [ ] Inheritance conflict resolution
#### Integration Tests
- [ ] End-to-end integration testing
- [ ] `FileSourceDataflow.test.ts`
- [ ] `FileTaskRelationships.test.ts`
- [ ] `MetadataInheritance.test.ts`
- [ ] `EventFlow.test.ts`
### Phase 4: Performance & Polish (Week 7-8)
#### Specialized Indexing
- [ ] Create `src/dataflow/indexer/FileSourceIndex.ts`
- [ ] Optimized file task indexing
- [ ] Fast lookup structures
- [ ] Batch update operations
- [ ] Memory-efficient storage
#### Worker Integration
- [ ] Update worker system for file tasks
- [ ] `src/dataflow/workers/FileSourceWorker.ts`
- [ ] Background file task processing
- [ ] Worker message protocol
- [ ] Performance monitoring
#### Caching Implementation
- [ ] Create `src/dataflow/indexer/FileTaskCache.ts`
- [ ] File task state caching structure
- [ ] Frontmatter hash implementation
- [ ] Children ID set management
- [ ] Recognition result caching
- [ ] TTL-based cache invalidation
- [ ] Memory usage optimization
- [ ] Cache performance metrics
- [ ] State comparison utilities
#### Settings UI Enhancement
- [ ] Advanced settings components
- [ ] Strategy priority configuration
- [ ] Custom recognition function editor
- [ ] Performance tuning interface
- [ ] Diagnostic tools
#### Performance Testing
- [ ] Performance benchmark suite
- [ ] Large vault testing (1000+ files)
- [ ] Recognition performance metrics
- [ ] Memory usage profiling
- [ ] Worker performance validation
- [ ] Update detection performance tests
- [ ] Rapid file changes scenario
- [ ] Large file with many tasks
- [ ] Metadata-only updates
- [ ] Content-only updates
- [ ] Mixed update patterns
- [ ] `PerformanceOptimization.test.ts`
- [ ] Test early exit patterns
- [ ] Test selective processing
- [ ] Test cache hit rates
- [ ] Test event deduplication
#### Documentation
- [ ] User documentation
- [ ] Feature overview guide
- [ ] Configuration examples
- [ ] Troubleshooting guide
- [ ] Best practices document
### Phase 5: Advanced Features (Week 9-10)
#### Custom Recognition Functions
- [ ] Implement custom function support
- [ ] JavaScript function parsing
- [ ] Sandboxed execution environment
- [ ] Function validation and testing
- [ ] Error handling and logging
#### Advanced Conflict Resolution
- [ ] Sophisticated conflict handling
- [ ] Dual role (project/task) management
- [ ] Metadata priority rules
- [ ] User-defined resolution strategies
- [ ] Conflict notification system
#### Bulk Operations
- [ ] Bulk file task operations
- [ ] Batch file task creation
- [ ] Bulk metadata updates
- [ ] Batch recognition re-evaluation
- [ ] Progress reporting for bulk ops
#### Export/Import Functionality
- [ ] File task data portability
- [ ] Export file task configurations
- [ ] Import recognition strategies
- [ ] Backup and restore functionality
- [ ] Migration tools
#### Advanced Filtering
- [ ] Enhanced query capabilities
- [ ] File task specific filters
- [ ] Recognition strategy filtering
- [ ] Child task aggregation filters
- [ ] Custom filter expressions
## File Structure and Components
### New Files to Create
```
src/
├── types/
│ └── file-source.d.ts # Core type definitions
├── dataflow/
│ ├── sources/
│ │ ├── FileSource.ts # Main FileSource implementation
│ │ ├── FileSourceConfig.ts # Configuration management
│ │ └── FileSourceUpdateDetector.ts # Update detection logic
│ ├── parsers/
│ │ ├── FileSourceEntry.ts # Parser entry point
│ │ └── FileSourceParser.ts # Recognition strategies
│ ├── augment/
│ │ └── FileSourceAugmentor.ts # File task augmentation
│ ├── indexer/
│ │ ├── FileSourceIndex.ts # Specialized indexing
│ │ └── FileTaskCache.ts # File task state caching
│ └── workers/
│ └── FileSourceWorker.ts # Background processing
├── components/
│ └── settings/
│ └── FileSourceSettings.ts # Settings UI
└── __tests__/
└── file-source/
├── FileSourceConfig.test.ts
├── FileSource.basic.test.ts
├── FileSourceUpdateDetector.test.ts
├── RecognitionStrategies.test.ts
├── MetadataStrategy.test.ts
├── TagStrategy.test.ts
├── TemplateStrategy.test.ts
├── PathStrategy.test.ts
├── FileSourceDataflow.test.ts
├── FileTaskRelationships.test.ts
├── MetadataInheritance.test.ts
├── EventFlow.test.ts
└── PerformanceOptimization.test.ts
```
### Files to Modify
```
src/
├── common/
│ └── setting-definition.ts # Add FileSource configuration
├── dataflow/
│ ├── events/
│ │ └── Events.ts # Add FileSource events and granular update events
│ ├── sources/
│ │ └── ObsidianSource.ts # Emit granular events (FILE_METADATA_CHANGED, FILE_CONTENT_CHANGED)
│ ├── indexer/
│ │ └── Repository.ts # File task integration
│ ├── Orchestrator.ts # FileSource initialization
│ └── createDataflow.ts # Factory updates
├── components/
│ └── settings/
│ └── TaskProgressBarSettingTab.ts # Settings integration
└── index.ts # Plugin initialization
```
## Dependencies and Integration Points
### Internal Dependencies
- **Dataflow Architecture**: Core event system, repository, orchestrator
- **Project System**: Resolver, ProjectConfigManager, ProjectDataCache
- **Task System**: Task types, metadata, parsing infrastructure
- **Settings System**: Setting definitions, UI components, validation
- **Worker System**: TaskWorkerManager, worker orchestration
### External Dependencies
- **Obsidian API**: Vault, MetadataCache, TFile, FileSystemAdapter
- **File System**: Path manipulation, file watching, metadata extraction
- **Performance**: Worker threads, caching, batch processing
### Integration Points
1. **Event System**: FILE_UPDATED → FileSource → FILE_TASK_UPDATED
2. **Repository**: File task storage alongside regular tasks
3. **QueryAPI**: Extended queries for file tasks
4. **Views**: File tasks displayed in all task views
5. **Project System**: Dual role management and conflict resolution
## Progress Tracking Sections
### Week 1-2: Core Infrastructure
**Status**: ✅ Completed
**Target Completion**: 2025-08-20
**Completion**: 20/25 tasks (80% - Core functionality complete)
**Priority Tasks**:
- [x] Core type definitions
- [x] Basic FileSource implementation
- [x] Settings integration
- [x] Initial unit tests
**Implementation Notes**:
- Core FileSource infrastructure is fully functional
- Metadata and tag-based recognition strategies implemented
- Comprehensive configuration system with validation
- Full test coverage for core components
- TypeScript compilation passing without errors
- Ready for Phase 2 implementation
### Week 3-4: Recognition Strategies
**Status**: ⏳ Not Started
**Target Completion**: [Date]
**Completion**: 0/18 tasks
**Priority Tasks**:
- [ ] Metadata-based recognition
- [ ] Tag-based recognition
- [ ] Strategy testing framework
- [ ] Parser entry point
### Week 5-6: Integration & Augmentation
**Status**: ⏳ Not Started
**Target Completion**: [Date]
**Completion**: 0/22 tasks
**Priority Tasks**:
- [ ] Repository integration
- [ ] Event system integration
- [ ] Child task relationships
- [ ] Metadata inheritance
### Week 7-8: Performance & Polish
**Status**: ⏳ Not Started
**Target Completion**: [Date]
**Completion**: 0/18 tasks
**Priority Tasks**:
- [ ] Worker integration
- [ ] Caching implementation
- [ ] Performance testing
- [ ] User documentation
### Week 9-10: Advanced Features
**Status**: ⏳ Not Started
**Target Completion**: [Date]
**Completion**: 0/15 tasks
**Priority Tasks**:
- [ ] Custom recognition functions
- [ ] Bulk operations
- [ ] Advanced filtering
- [ ] Export/import functionality
## Risk Assessment
### High Risk Items
- **Performance Impact**: Large vaults with many files could impact startup time
- *Mitigation*: Worker-based processing, selective file scanning, caching
- **Memory Usage**: File tasks add to memory footprint
- *Mitigation*: Efficient indexing, lazy loading, configurable limits
- **Conflict Resolution**: Dual role files may create complex scenarios
- *Mitigation*: Clear precedence rules, user configuration options
### Medium Risk Items
- **Configuration Complexity**: Many options may overwhelm users
- *Mitigation*: Sensible defaults, configuration wizard, presets
- **Event Loop Performance**: Additional event processing overhead
- *Mitigation*: Debounced events, batch processing, sequence optimization
### Low Risk Items
- **Backward Compatibility**: New feature shouldn't break existing functionality
- *Mitigation*: Feature flag, disabled by default, comprehensive testing
## Success Metrics
### Functional Metrics
- [ ] All recognition strategies working correctly
- [ ] File tasks appear in all relevant views
- [ ] Child task relationships functioning
- [ ] Metadata inheritance working
- [ ] Settings interface complete and intuitive
### Performance Metrics
- [ ] File task detection under 100ms for typical files
- [ ] Memory usage increase under 10% for equivalent functionality
- [ ] No noticeable impact on plugin startup time
- [ ] Worker processing provides measurable performance benefits
### Quality Metrics
- [ ] 90%+ test coverage for FileSource components
- [ ] Zero regression in existing functionality
- [ ] User documentation complete and clear
- [ ] Configuration examples working as documented
## Notes and Decisions
### Architecture Decisions
- **FileSource as separate source**: Maintains clean separation from ObsidianSource
- **Event-driven integration**: Leverages existing event system for consistency
- **Strategy pattern**: Allows flexible recognition approaches
- **Metadata extension**: Extends existing task metadata rather than creating new types
### Configuration Decisions
- **Disabled by default**: Conservative rollout approach
- **Multiple strategies**: Accommodates different user workflows
- **Extensive configuration**: Provides flexibility while maintaining usability
### Performance Decisions
- **Worker integration**: Maintains UI responsiveness
- **Caching strategy**: Balances memory usage with performance
- **Selective processing**: Avoid processing irrelevant files
---
**Last Updated**: [Current Date]
**Document Version**: 1.0
**Next Review**: [Date]
Reference in a new issue View git blame Copy permalink