obsidian-plugin/jacobinwwey_obsidian-NotEMD
1
0
Fork 0
mirror of https://github.com/jacobinwwey/obsidian-NotEMD.git synced 2026-07-22 12:40:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

test(local-kb): execute repo-owned research docs through fixtures

Browse source
This commit is contained in:
Jacobinwwey 2026-06-09 10:45:41 -05:00
parent faa783e8b3
commit 2ed3d018ff
11 changed files with 133 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
docs/brainstorms/2026-05-20-unified-follow-through-matrix.md
View file

@ -55,6 +55,7 @@ Current Batch-C alignment note:
2. The regex / file selection / local-KB / chapter split lane has therefore moved from “maintainer examples still need alignment” to “helper/docs/tests are aligned to the current fixture lane’s real note/query diversity, including reproducible docs-vault failure-state probes for `no-paths` and `no-candidate-files`.” When that lane grows again, prioritize failure-state, low-signal query, and noisy-corpus evidence before adding task count.
3. This batch does not change the MiniSearch retrieval algorithm, chapter-split write contract, packaging boundary, or provider control plane. It reduces drift between test evidence, maintainer-reproducible examples, and progress documentation.
4. The local-KB decision, execution-quality lens, and chapter-split comparison conclusions are now also mirrored into tracked `docs/brainstorms/2026-06-09-*` records instead of remaining only under `.trellis/tasks/.../research/`.
5. Those repo-owned `2026-06-09` records are now also part of executable retrieval evidence through the offline fixture and maintainer inspect examples, not only documentation truth.
## 4. Reconfirmed Current-Main Register

1
docs/brainstorms/2026-05-20-unified-follow-through-matrix.zh-CN.md
View file

@ -55,6 +55,7 @@ canonical: true
2. Regex / 文件筛选 / local-KB / chapter split 轨道的当前进展从“需要 maintainer 示例对齐”推进为“helper/docs/tests 已对齐当前 fixture lane 的真实 note/query 多样性，并补上了可直接复现 `no-paths` / `no-candidate-files` 的 docs-vault failure-state 示例”；后续再扩大时，应先补失败态、低信号 query 与 noisy-corpus 证据，而不是扩任务数量。
3. 这批更新没有改变 MiniSearch retrieval algorithm、chapter split 写入契约、packaging 边界或 provider control-plane；它只降低测试证据、维护者可执行示例与进度文档之间的漂移风险。
4. local-KB 的方案决策、执行质量视角以及 chapter-split 对比结论，现在也已镜像到受跟踪的 `docs/brainstorms/2026-06-09-*` 文档，而不再只留在 `.trellis/tasks/.../research/` 下。
5. 这些 repo-owned 的 `2026-06-09` 文档现在也已经进入可执行的检索证据面：离线 fixture 与 maintainer inspect 示例会直接沿真实 retrieval 路径探测它们，而不只是把它们当作文档真值。
## 4. 当前主线已确认 register

2
docs/brainstorms/2026-05-28-mainline-progress-audit-and-next-level-direction.md
View file

@ -248,6 +248,8 @@ This is no longer only a principle statement. The repo now carries tracked mirro
2. `docs/brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md`
3. `docs/brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md`
Those mirrors are now also part of executable Stage-C evidence instead of remaining docs-only truth: the local-KB offline fixture and maintainer inspect examples now probe them through real retrieval paths.
## 3. Deep Comparison Against Earlier Plan Language
### 3.1 What the 2026-05-25 audit now understates

2
docs/brainstorms/2026-05-28-mainline-progress-audit-and-next-level-direction.zh-CN.md
View file

@ -248,6 +248,8 @@ canonical: true
2. `docs/brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.zh-CN.md`
3. `docs/brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.zh-CN.md`
这些镜像现在也已经进入可执行的 Stage-C 证据面，而不再只是 docs-only 真值：local-KB 的离线 fixture 与 maintainer inspect 示例会沿真实检索路径探测它们。
## 3. 相对先前方案语言的深度对比
### 3.1 2026-05-25 审计现在低估了什么

2
docs/maintainer/notemd-cli-capability-matrix.md
View file

@ -99,6 +99,8 @@ Boundary:
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"researchSummarize","query":"task-scoped retrieval behavior","knowledgePaths":["maintainer"]}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"researchSummarize","query":"chapter split TOC managed artifacts guarded reruns","knowledgePaths":["chapter-split-toc.md","chapter-split-toc.zh-CN.md"]}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"researchSummarize","query":"real-note query diversity beyond chapter split showcase","knowledgePaths":["brainstorms","maintainer"],"topK":2,"slidingWindowSize":1}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"researchSummarize","query":"MiniSearch ragas RAGPerf execution chain maintainer-only offline evidence","knowledgePaths":["brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md"],"topK":2,"slidingWindowSize":1,"maxSnippetChars":640}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"researchSummarize","query":"managed-artifact kpm markdown-toc active-file scoped stable block refs","knowledgePaths":["brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md"],"topK":2,"slidingWindowSize":1,"maxSnippetChars":640}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"batchGenerateFromTitles","sourcePath":"brainstorms/2026-05-28-mainline-progress-audit-and-next-level-direction.md","knowledgePaths":["brainstorms","maintainer"],"topK":2,"slidingWindowSize":1}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"diagramGeneration","sourcePath":"index.zh-CN.md","knowledgePaths":["brainstorms","maintainer"],"topK":2,"slidingWindowSize":1}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"researchSummarize","query":"missing path coverage","knowledgePaths":[]}' --pretty`

2
docs/maintainer/notemd-cli-capability-matrix.zh-CN.md
View file

@ -99,6 +99,8 @@
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"researchSummarize","query":"task-scoped retrieval behavior","knowledgePaths":["maintainer"]}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"researchSummarize","query":"chapter split TOC managed artifacts guarded reruns","knowledgePaths":["chapter-split-toc.md","chapter-split-toc.zh-CN.md"]}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"researchSummarize","query":"real-note query diversity beyond chapter split showcase","knowledgePaths":["brainstorms","maintainer"],"topK":2,"slidingWindowSize":1}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"researchSummarize","query":"MiniSearch ragas RAGPerf execution chain maintainer-only offline evidence","knowledgePaths":["brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md"],"topK":2,"slidingWindowSize":1,"maxSnippetChars":640}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"researchSummarize","query":"managed-artifact kpm markdown-toc active-file scoped stable block refs","knowledgePaths":["brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md"],"topK":2,"slidingWindowSize":1,"maxSnippetChars":640}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"batchGenerateFromTitles","sourcePath":"brainstorms/2026-05-28-mainline-progress-audit-and-next-level-direction.md","knowledgePaths":["brainstorms","maintainer"],"topK":2,"slidingWindowSize":1}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"diagramGeneration","sourcePath":"index.zh-CN.md","knowledgePaths":["brainstorms","maintainer"],"topK":2,"slidingWindowSize":1}' --pretty`
- `npm run cli:invoke -- --vault docs --operation local-knowledge.inspect --input-json '{"taskScope":"researchSummarize","query":"missing path coverage","knowledgePaths":[]}' --pretty`

2
scripts/lib/maintainer-cli-operation-help.js
View file

@ -33,6 +33,8 @@ const OPERATION_HELP = {
'{"taskScope":"researchSummarize","query":"task-scoped retrieval behavior","knowledgePaths":["maintainer"]}',
'{"taskScope":"researchSummarize","query":"chapter split TOC managed artifacts guarded reruns","knowledgePaths":["chapter-split-toc.md","chapter-split-toc.zh-CN.md"]}',
'{"taskScope":"researchSummarize","query":"real-note query diversity beyond chapter split showcase","knowledgePaths":["brainstorms","maintainer"],"topK":2,"slidingWindowSize":1}',
'{"taskScope":"researchSummarize","query":"MiniSearch ragas RAGPerf execution chain maintainer-only offline evidence","knowledgePaths":["brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md"],"topK":2,"slidingWindowSize":1,"maxSnippetChars":640}',
'{"taskScope":"researchSummarize","query":"managed-artifact kpm markdown-toc active-file scoped stable block refs","knowledgePaths":["brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md"],"topK":2,"slidingWindowSize":1,"maxSnippetChars":640}',
'{"taskScope":"batchGenerateFromTitles","sourcePath":"brainstorms/2026-05-28-mainline-progress-audit-and-next-level-direction.md","knowledgePaths":["brainstorms","maintainer"],"topK":2,"slidingWindowSize":1}',
'{"taskScope":"diagramGeneration","sourcePath":"index.zh-CN.md","knowledgePaths":["brainstorms","maintainer"],"topK":2,"slidingWindowSize":1}',
'{"taskScope":"researchSummarize","query":"missing path coverage","knowledgePaths":[]}',

8
src/tests/cliPublicSurfaceDocsAlignment.test.ts
View file

@ -128,6 +128,14 @@ describe('CLI public surface docs alignment', () => {
expect(matrixZh).toContain('"query":"real-note query diversity beyond chapter split showcase"');
expect(matrix).toContain('"knowledgePaths":["brainstorms","maintainer"]');
expect(matrixZh).toContain('"knowledgePaths":["brainstorms","maintainer"]');
expect(matrix).toContain('"query":"MiniSearch ragas RAGPerf execution chain maintainer-only offline evidence"');
expect(matrixZh).toContain('"query":"MiniSearch ragas RAGPerf execution chain maintainer-only offline evidence"');
expect(matrix).toContain('"knowledgePaths":["brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md"],"topK":2,"slidingWindowSize":1,"maxSnippetChars":640');
expect(matrixZh).toContain('"knowledgePaths":["brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md"],"topK":2,"slidingWindowSize":1,"maxSnippetChars":640');
expect(matrix).toContain('"query":"managed-artifact kpm markdown-toc active-file scoped stable block refs"');
expect(matrixZh).toContain('"query":"managed-artifact kpm markdown-toc active-file scoped stable block refs"');
expect(matrix).toContain('"knowledgePaths":["brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md"],"topK":2,"slidingWindowSize":1,"maxSnippetChars":640');
expect(matrixZh).toContain('"knowledgePaths":["brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md"],"topK":2,"slidingWindowSize":1,"maxSnippetChars":640');
expect(matrix).toContain('"sourcePath":"brainstorms/2026-05-28-mainline-progress-audit-and-next-level-direction.md"');
expect(matrixZh).toContain('"sourcePath":"brainstorms/2026-05-28-mainline-progress-audit-and-next-level-direction.md"');
expect(matrix).toContain('"query":"missing path coverage"');

4
src/tests/invokeMaintainerCliOperationScript.test.ts
View file

@ -47,6 +47,10 @@ describe('invoke maintainer CLI operation script', () => {
expect(output).toContain('"knowledgePaths":["chapter-split-toc.md","chapter-split-toc.zh-CN.md"]');
expect(output).toContain('"query":"real-note query diversity beyond chapter split showcase"');
expect(output).toContain('"knowledgePaths":["brainstorms","maintainer"]');
expect(output).toContain('"query":"MiniSearch ragas RAGPerf execution chain maintainer-only offline evidence"');
expect(output).toContain('"knowledgePaths":["brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md"],"topK":2,"slidingWindowSize":1,"maxSnippetChars":640');
expect(output).toContain('"query":"managed-artifact kpm markdown-toc active-file scoped stable block refs"');
expect(output).toContain('"knowledgePaths":["brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md"],"topK":2,"slidingWindowSize":1,"maxSnippetChars":640');
expect(output).toContain('"sourcePath":"brainstorms/2026-05-28-mainline-progress-audit-and-next-level-direction.md"');
expect(output).toContain('"query":"missing path coverage"');
expect(output).toContain('"knowledgePaths":[]');

106
src/tests/localKnowledgeEvaluationFixture.test.ts
View file

@ -255,6 +255,38 @@ const FIXTURE_FILES: FixtureFile[] = [
'Batch title inspection against brainstorm notes should still find cross-folder operator guidance after current-file exclusion removes the source note itself.'
].join('\n')
},
{
path: 'brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md',
markdown: [
'# Local KB RAG Quality And Execution Truth',
'Repo-owned mirror for the shipped local retrieval execution chain and evaluation boundary.',
'',
'## Execution Chain',
'The shipped retrieval path is intentionally narrow: plugin-native, local-only, in-process, server-free, GPU-free, embedding-free.',
'',
'## Evaluation References',
'MiniSearch remains the bounded runtime fit while ragas and RAGPerf stay evaluation references.',
'',
'## Guidance',
'The right near-term move is to borrow the evaluation mindset off the hot path, keep local-knowledge.inspect maintainer-only, and keep npm run verify:local-kb-fixtures as offline evidence.'
].join('\n')
},
{
path: 'brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md',
markdown: [
'# Chapter Split Knowledge Management And TOC Comparison Truth',
'Repo-owned mirror for the chapter-split comparison conclusions.',
'',
'## Managed Artifacts',
'This is a managed-artifact write contract, not just TOC insertion, and kpm plus markdown-toc remain comparison references.',
'',
'## Scope',
'Keep chapter split active-file scoped on current main.',
'',
'## Guidance',
'Deterministic TOC front matter, stable block refs, and manifest-backed reruns should stay explicit in docs and tests.'
].join('\n')
},
{
path: 'maintainer/CLI Surface.md',
markdown: [
@ -501,6 +533,42 @@ const EVALUATION_CASES: EvaluationCase[] = [
'recall proxy, precision pressure, noisy corpus behavior, latency telemetry',
'local-only, server-free, cloud-free, and GPU-free'
]
},
{
id: 'repo-owned-rag-truth-doc',
query: 'MiniSearch ragas RAGPerf execution chain maintainer-only offline evidence',
configuredKnowledgePaths: 'brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md',
expectedSourcePaths: ['brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md'],
topK: 2,
slidingWindowSize: 1,
maxSnippetChars: 640,
minExpectedPathRecall: 1,
minReturnedHitCount: 1,
minMatchedSectionCount: 1,
minExpandedSectionCount: 2,
expectedContextFragments: [
'plugin-native, local-only, in-process, server-free, GPU-free, embedding-free',
'MiniSearch remains the bounded runtime fit while ragas and RAGPerf stay evaluation references.',
'borrow the evaluation mindset off the hot path'
]
},
{
id: 'repo-owned-chapter-split-comparison-doc',
query: 'managed-artifact kpm markdown-toc active-file scoped stable block refs',
configuredKnowledgePaths: 'brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md',
expectedSourcePaths: ['brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md'],
topK: 2,
slidingWindowSize: 1,
maxSnippetChars: 640,
minExpectedPathRecall: 1,
minReturnedHitCount: 1,
minMatchedSectionCount: 1,
minExpandedSectionCount: 2,
expectedContextFragments: [
'managed-artifact write contract',
'kpm plus markdown-toc remain comparison references',
'Keep chapter split active-file scoped on current main.'
]
}
];
@ -1003,6 +1071,8 @@ describe('local knowledge offline evaluation fixture', () => {
researchScoped,
chapterSplitShowcase,
researchCrossFolder,
repoOwnedRagTruth,
repoOwnedChapterSplitTruth,
batchCrossFolder,
diagramCrossFolder,
noPathsExample,
@ -1048,6 +1118,34 @@ describe('local knowledge offline evaluation fixture', () => {
])
);
expect(repoOwnedRagTruth.queryDerivation).toBe('explicit');
expect(repoOwnedRagTruth.effectivePathSource).toBe('override');
expect(repoOwnedRagTruth.effectiveConfiguredPaths).toEqual([
'brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md'
]);
expect(repoOwnedRagTruth.retrieverBuildStatus).toBe('ready');
expect(repoOwnedRagTruth.candidateFilePaths).toEqual([
'brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md'
]);
expect(repoOwnedRagTruth.retrieval.sourcePaths).toEqual([
'brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md'
]);
expect(repoOwnedRagTruth.context).toContain('ragas and RAGPerf stay evaluation references');
expect(repoOwnedChapterSplitTruth.queryDerivation).toBe('explicit');
expect(repoOwnedChapterSplitTruth.effectivePathSource).toBe('override');
expect(repoOwnedChapterSplitTruth.effectiveConfiguredPaths).toEqual([
'brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md'
]);
expect(repoOwnedChapterSplitTruth.retrieverBuildStatus).toBe('ready');
expect(repoOwnedChapterSplitTruth.candidateFilePaths).toEqual([
'brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md'
]);
expect(repoOwnedChapterSplitTruth.retrieval.sourcePaths).toEqual([
'brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md'
]);
expect(repoOwnedChapterSplitTruth.context).toContain('Keep chapter split active-file scoped on current main.');
expect(batchCrossFolder.queryDerivation).toBe('basename');
expect(batchCrossFolder.effectivePathSource).toBe('override');
expect(batchCrossFolder.effectiveConfiguredPaths).toEqual(['brainstorms', 'maintainer']);
@ -1071,12 +1169,18 @@ describe('local knowledge offline evaluation fixture', () => {
expect(diagramCrossFolder.retrieverBuildStatus).toBe('ready');
expect(diagramCrossFolder.retrieval.returnedHitCount).toBeGreaterThanOrEqual(1);
expect(diagramCrossFolder.retrieval.sourcePaths).toEqual(
expect.arrayContaining(['maintainer/CLI Surface.md'])
expect.arrayContaining([
'brainstorms/2026-05-28-mainline-progress-audit-and-next-level-direction.md',
'brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md'
])
);
expect(diagramCrossFolder.context).toContain('ragas and RAGPerf stay evaluation references');
expect(diagramCrossFolder.candidateFilePaths).toEqual(
expect.arrayContaining([
'brainstorms/2026-05-28-mainline-progress-audit-and-next-level-direction.md',
'brainstorms/2026-05-29-stage-c-quality-follow-through.md',
'brainstorms/2026-06-09-local-kb-rag-quality-and-execution-truth.md',
'brainstorms/2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md',
'maintainer/CLI Surface.md'
])
);

4
src/tests/mainlineProgressAuditContract.test.ts
View file

@ -170,7 +170,11 @@ describe('mainline progress audit contract', () => {
expect(progressDocZh).toContain('2026-06-09-local-kb-rag-quality-and-execution-truth.zh-CN.md');
expect(progressDoc).toContain('2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.md');
expect(progressDocZh).toContain('2026-06-09-chapter-split-knowledge-management-and-toc-comparison-truth.zh-CN.md');
expect(progressDoc).toContain('offline fixture and maintainer inspect examples now probe them through real retrieval paths');
expect(progressDocZh).toContain('离线 fixture 与 maintainer inspect 示例会沿真实检索路径探测它们');
expect(matrixDoc).toContain('docs/brainstorms/2026-06-09-*');
expect(matrixDocZh).toContain('docs/brainstorms/2026-06-09-*');
expect(matrixDoc).toContain('part of executable retrieval evidence through the offline fixture and maintainer inspect examples');
expect(matrixDocZh).toContain('进入可执行的检索证据面');
});
});