mirror of
https://github.com/sotashimozono/obsidian-remote-ssh.git
synced 2026-07-22 06:52:07 +00:00
6772f60b65
Pure-module aggregator that turns the span-duration samples emitted by
PerfTracer (M0) + the M2 instrumentation into per-bucket percentile
stats. M6 (microbench) will feed it; M10's CI gate will diff its
NDJSON against the `perf-baseline` branch and post the Markdown table
as a PR comment.
Why now: M4 is purely additive (no production code touched, no
existing tests modified) and is the prerequisite for M6 — without it
the bench has no way to turn raw span records into the p50/p95/p99
numbers the regression detector compares against. Lands in parallel
with M3 (Go daemon spans + cid correlator) since the two are
orthogonal: M3 is Go-only, M4 is TS-only and consumer-only.
Surface (`plugin/tests/integration/helpers/perfAggregator.ts`):
- Keyed by the bench-matrix tuple `(span, transport, sizeBytes)`:
* span — 'S.adp' | 'S.rpc' | 'S.app' | 'S.e2e' | …
* transport — 'rpc' | 'sftp'
* sizeBytes — fixture file size (0 when N/A, e.g. remove/rename)
- `record(span, transport, sizeBytes, durMs)` — push one sample.
NaN / Infinity / negative values are silently dropped (a perf
collector should never poison its own dataset on bad input). 0 is
allowed (sub-resolution timer, not invalid).
- `percentiles(span, transport, sizeBytes, opts?) → PerfStats | null`
— null when the bucket was never seen. Returns p50/p95/p99/mean/
stddev/n, plus a `filtered` count when `opts.filterOutliers` is on.
p* are linear-interpolated (R-7 / numpy.percentile default).
- `toNDJSON(opts?) → string` — one JSON object per bucket, trailing
newline. Matches the format `PerfTracer.flushNDJSON` writes so the
two streams can be `cat`'d into one file for offline analysis.
- `toMarkdownTable(opts?) → string` — header + separator + per-bucket
row with width-aware numeric formatting (3 dec for sub-ms, 2 dec
for sub-100ms, 1 dec otherwise). For PR comments / CI logs.
- `buckets() → PerfBucket[]` — introspection, lists every (span,
transport, sizeBytes) seen with its raw count.
- `size()` — distinct-bucket count.
Tukey 1.5×IQR outlier filter is opt-in per call so a single dataset
can be reported both raw (transparency in CI logs) and filtered (the
gate that compares against baseline). Skipped automatically for
series with fewer than 4 samples — IQR isn't meaningful below that
and dropping points would zero out short bench runs.
Tests (13, all green; `tests/PerfAggregator.test.ts`):
- Empty aggregator: size=0, percentiles()=null, NDJSON/Markdown empty.
- Bucket isolation across (span, transport, sizeBytes).
- record() drops NaN / -1 / Infinity but keeps 0.
- Single-sample bucket: every percentile equals the sample, stddev=0.
- 1..10 dataset: p50=5.5, p95=9.55, p99=9.91 (numpy reference values),
population stddev ≈ 2.872.
- Constant series: stddev=0.
- Tukey filter: 1..10 + 1000 → outlier dropped, mean returns to 5.5,
filtered=1.
- Tukey skipped for n<4 (no points dropped from a 3-sample series).
- toNDJSON: one JSON per line, trailing newline, parseable, includes
the `filtered` count when filtering is on.
- toMarkdownTable: header + separator + per-bucket row, numeric
formatting widens by magnitude.
- buckets(): listing matches insertion order with raw counts.
Suite results:
- `npx vitest run` — 34 files / 454 tests green (was 441, +13 new)
- `npx tsc --noEmit -p tsconfig.json` — clean
Pure module: zero production-side code modified. Path is under
`tests/integration/helpers/` per the plan because the only consumer
is the M6 bench, but the module is integration-free so its unit
tests live in the standard `tests/` directory and run in the unit
suite (where the project's coverage and PR-gate workflows pick them
up).
Roadmap context: M4 + M3 are the two parallel streams that close out
the producer ↔ consumer contract for Phase C. After both land, M6
wires PerfTracer + M3 cid threading + PerfAggregator into a single
bench file and writes NDJSON results to disk, completing the MVP
slice (Phase C.0–6 of the plan).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
177 lines
6.7 KiB
TypeScript
177 lines
6.7 KiB
TypeScript
import { describe, it, expect } from 'vitest';
|
||
import { PerfAggregator } from './integration/helpers/perfAggregator';
|
||
|
||
/**
|
||
* Pure-module unit coverage for the bench-side aggregator. The path
|
||
* lives under `tests/integration/helpers/` (consumer is the M6 bench)
|
||
* but the module itself is integration-free, so it gets unit tests
|
||
* here in the standard `tests/` directory.
|
||
*/
|
||
|
||
describe('PerfAggregator — keying & isolation', () => {
|
||
it('starts empty', () => {
|
||
const a = new PerfAggregator();
|
||
expect(a.size()).toBe(0);
|
||
expect(a.percentiles('S.adp', 'rpc', 1024)).toBeNull();
|
||
expect(a.toNDJSON()).toBe('');
|
||
expect(a.toMarkdownTable()).toBe('');
|
||
});
|
||
|
||
it('isolates samples across (span, transport, sizeBytes) tuples', () => {
|
||
const a = new PerfAggregator();
|
||
a.record('S.adp', 'rpc', 1024, 1.0);
|
||
a.record('S.adp', 'rpc', 1024, 2.0);
|
||
a.record('S.adp', 'rpc', 100_000, 5.0); // different size → different bucket
|
||
a.record('S.adp', 'sftp', 1024, 7.0); // different transport
|
||
a.record('S.rpc', 'rpc', 1024, 0.5); // different span
|
||
|
||
expect(a.size()).toBe(4);
|
||
expect(a.percentiles('S.adp', 'rpc', 1024)?.n).toBe(2);
|
||
expect(a.percentiles('S.adp', 'rpc', 100_000)?.n).toBe(1);
|
||
expect(a.percentiles('S.adp', 'sftp', 1024)?.n).toBe(1);
|
||
expect(a.percentiles('S.rpc', 'rpc', 1024)?.n).toBe(1);
|
||
});
|
||
|
||
it('drops non-finite or negative samples', () => {
|
||
const a = new PerfAggregator();
|
||
a.record('S.adp', 'rpc', 1024, NaN);
|
||
a.record('S.adp', 'rpc', 1024, -1);
|
||
a.record('S.adp', 'rpc', 1024, Infinity);
|
||
expect(a.size()).toBe(0);
|
||
a.record('S.adp', 'rpc', 1024, 0); // 0 is allowed (sub-resolution)
|
||
expect(a.percentiles('S.adp', 'rpc', 1024)?.n).toBe(1);
|
||
});
|
||
});
|
||
|
||
describe('PerfAggregator — percentile math', () => {
|
||
it('single-sample bucket: every percentile equals that sample', () => {
|
||
const a = new PerfAggregator();
|
||
a.record('S.adp', 'rpc', 1024, 7.0);
|
||
const s = a.percentiles('S.adp', 'rpc', 1024);
|
||
expect(s).not.toBeNull();
|
||
expect(s!.p50).toBe(7);
|
||
expect(s!.p95).toBe(7);
|
||
expect(s!.p99).toBe(7);
|
||
expect(s!.mean).toBe(7);
|
||
expect(s!.stddev).toBe(0);
|
||
expect(s!.n).toBe(1);
|
||
});
|
||
|
||
it('matches the R-7 / numpy linear-interpolated percentile on 1..10', () => {
|
||
const a = new PerfAggregator();
|
||
for (let i = 1; i <= 10; i++) a.record('S.adp', 'rpc', 1024, i);
|
||
const s = a.percentiles('S.adp', 'rpc', 1024)!;
|
||
// numpy: percentile([1..10], [50, 95, 99]) → [5.5, 9.55, 9.91]
|
||
expect(s.p50).toBeCloseTo(5.5, 5);
|
||
expect(s.p95).toBeCloseTo(9.55, 5);
|
||
expect(s.p99).toBeCloseTo(9.91, 5);
|
||
expect(s.mean).toBeCloseTo(5.5, 5);
|
||
// population stddev of 1..10 ≈ 2.872
|
||
expect(s.stddev).toBeCloseTo(2.8722813, 5);
|
||
expect(s.n).toBe(10);
|
||
expect(s.filtered).toBe(0); // no outlier filter applied
|
||
});
|
||
|
||
it('mean / stddev / n on a constant series', () => {
|
||
const a = new PerfAggregator();
|
||
for (let i = 0; i < 5; i++) a.record('S.adp', 'rpc', 1024, 4.0);
|
||
const s = a.percentiles('S.adp', 'rpc', 1024)!;
|
||
expect(s.mean).toBe(4);
|
||
expect(s.stddev).toBe(0);
|
||
expect(s.n).toBe(5);
|
||
});
|
||
});
|
||
|
||
describe('PerfAggregator — Tukey 1.5×IQR outlier filtering', () => {
|
||
it('removes the obvious outlier from 1..10 + 1000', () => {
|
||
const a = new PerfAggregator();
|
||
for (let i = 1; i <= 10; i++) a.record('S.adp', 'rpc', 1024, i);
|
||
a.record('S.adp', 'rpc', 1024, 1000);
|
||
|
||
const raw = a.percentiles('S.adp', 'rpc', 1024)!;
|
||
expect(raw.n).toBe(11);
|
||
// mean of 1..10 + 1000 = 95.45..., dragged way up by the outlier
|
||
expect(raw.mean).toBeGreaterThan(90);
|
||
|
||
const filt = a.percentiles('S.adp', 'rpc', 1024, { filterOutliers: true })!;
|
||
// Q1≈3.25, Q3≈8.5 (after sort), IQR≈5.25, hi-fence ≈ 16.4 → 1000 dropped
|
||
expect(filt.n).toBe(10);
|
||
expect(filt.filtered).toBe(1);
|
||
expect(filt.mean).toBeCloseTo(5.5, 5);
|
||
});
|
||
|
||
it('skips the filter on series with fewer than 4 samples (IQR is meaningless)', () => {
|
||
const a = new PerfAggregator();
|
||
a.record('S.adp', 'rpc', 1024, 1);
|
||
a.record('S.adp', 'rpc', 1024, 1);
|
||
a.record('S.adp', 'rpc', 1024, 1000); // would be a clear outlier with more samples
|
||
const filt = a.percentiles('S.adp', 'rpc', 1024, { filterOutliers: true })!;
|
||
expect(filt.n).toBe(3);
|
||
expect(filt.filtered).toBe(0);
|
||
});
|
||
});
|
||
|
||
describe('PerfAggregator — output formats', () => {
|
||
it('toNDJSON: one JSON object per bucket, trailing newline', () => {
|
||
const a = new PerfAggregator();
|
||
a.record('S.adp', 'rpc', 1024, 1.0);
|
||
a.record('S.adp', 'rpc', 1024, 3.0);
|
||
a.record('S.rpc', 'rpc', 1024, 0.5);
|
||
|
||
const text = a.toNDJSON();
|
||
expect(text.endsWith('\n')).toBe(true);
|
||
const lines = text.trimEnd().split('\n');
|
||
expect(lines).toHaveLength(2);
|
||
const parsed = lines.map((l) => JSON.parse(l));
|
||
expect(parsed[0]).toMatchObject({ span: 'S.adp', transport: 'rpc', sizeBytes: 1024, n: 2 });
|
||
expect(parsed[0].p50).toBeCloseTo(2.0, 5);
|
||
expect(parsed[1]).toMatchObject({ span: 'S.rpc', transport: 'rpc', sizeBytes: 1024, n: 1 });
|
||
});
|
||
|
||
it('toNDJSON honours filterOutliers and reports the dropped count', () => {
|
||
const a = new PerfAggregator();
|
||
for (let i = 1; i <= 10; i++) a.record('S.adp', 'rpc', 1024, i);
|
||
a.record('S.adp', 'rpc', 1024, 1000);
|
||
|
||
const filtered = JSON.parse(a.toNDJSON({ filterOutliers: true }).trim());
|
||
expect(filtered.n).toBe(10);
|
||
expect(filtered.filtered).toBe(1);
|
||
});
|
||
|
||
it('toMarkdownTable: header + separator + per-bucket row', () => {
|
||
const a = new PerfAggregator();
|
||
a.record('S.adp', 'rpc', 1024, 1.0);
|
||
a.record('S.adp', 'rpc', 1024, 3.0);
|
||
|
||
const md = a.toMarkdownTable();
|
||
const lines = md.split('\n');
|
||
expect(lines[0]).toContain('| span | transport |');
|
||
expect(lines[1]).toMatch(/\|---/); // separator
|
||
expect(lines[2]).toContain('| S.adp | rpc | 1024 | 2 |');
|
||
});
|
||
|
||
it('toMarkdownTable: numeric formatting widens by magnitude', () => {
|
||
const a = new PerfAggregator();
|
||
a.record('A', 'rpc', 0, 0.123456); // sub-millisecond → 3 decimals
|
||
a.record('B', 'rpc', 0, 12.345); // sub-100ms → 2 decimals
|
||
a.record('C', 'rpc', 0, 1234.5); // ≥100ms → 1 decimal
|
||
const md = a.toMarkdownTable();
|
||
expect(md).toContain('0.123');
|
||
expect(md).toContain('12.35');
|
||
expect(md).toContain('1234.5');
|
||
});
|
||
});
|
||
|
||
describe('PerfAggregator — buckets() introspection', () => {
|
||
it('lists every (span, transport, sizeBytes) seen, with raw counts', () => {
|
||
const a = new PerfAggregator();
|
||
a.record('S.adp', 'rpc', 1024, 1);
|
||
a.record('S.adp', 'rpc', 1024, 2);
|
||
a.record('S.rpc', 'sftp', 0, 5);
|
||
const bs = a.buckets();
|
||
expect(bs).toEqual([
|
||
{ span: 'S.adp', transport: 'rpc', sizeBytes: 1024, n: 2 },
|
||
{ span: 'S.rpc', transport: 'sftp', sizeBytes: 0, n: 1 },
|
||
]);
|
||
});
|
||
});
|