mirror of
https://github.com/sotashimozono/obsidian-remote-ssh.git
synced 2026-07-22 17:10:32 +00:00
Closes the second half of the Phase C MVP, but with a structurally different cadence than the original M10 design (PR #106 v1). The bench (`plugin/tests/integration/perf.sync.bench.test.ts`) takes ~6 minutes per run — 96 % of the integration job's wall time, all of it in the SSH-tunneled RPC RTT (~41 ms × ~3000 calls). Running it on every PR was wrong on two counts: 1. **Feedback latency**: every PR adds 6 minutes to CI for a comment-only gate. Souta correctly called out that the bench is overkill for the per-PR feedback loop. 2. **Bench purpose**: the bench is a *trend tracker* and an *on-demand perf microscope*. Trends want a stable cadence (nightly), not noise from every PR; a microscope wants to be pulled out when there's a hypothesis to test, not run on every keystroke. So this PR splits the workflow: - **integration.yml** (PR + push to main): runs everything EXCEPT the bench. Drops back to ~30 s wall time. - **bench.yml** (new, nightly cron + workflow_dispatch + push-to-main): runs ONLY the bench. Updates `perf-baseline` branch on nightly + main-push. workflow_dispatch with an optional `pr_number` input posts the diff comment to the named PR — preserves the on-demand "hey what does this branch look like perf-wise" capability without paying the 6-minute tax on every push. What ships: - `plugin/scripts/perf/compare.mjs` (new, pure) Comparison core. Reads two NDJSON datasets, joins by `(span, transport, sizeBytes)`, computes p95 deltas, classifies each bucket as regressed / improved / unchanged / new / removed against per-span tolerance (default 25 %; S.fs widened to 40 % because shared-CI disk variance is high; S.note to 50 % for fsnotify debounce noise). Renders a Markdown table for the PR comment with arrows (🔴↑/🟢↓/−/🆕/➖), bolds regressed rows, appends a pass/fail summary + a footer with the baseline / head SHA pair + run URL. Status-ordered output (regressions first) so reviewers see the bad news without scrolling. - `plugin/scripts/perf/run-compare.mjs` (new, CLI) Picks the latest NDJSON in `plugin/perf-results/`, fetches the baseline at `<workspace>/perf-baseline/baseline.ndjson`, calls `compare()`, posts via `gh pr comment`. **Upserts** the comment by sentinel header so a long-running PR with multiple workflow_dispatch invocations gets one rolling comment instead of a per-trigger spam stack. Exits 0 unless `PERF_GATE=1` AND there's a regression past tolerance. - `plugin/scripts/perf/run-update-baseline.mjs` (new, CLI) Bootstraps the orphan `perf-baseline` branch on first run; on subsequent triggers, copies the latest NDJSON onto it as `baseline.ndjson` + writes a sidecar `baseline.sha`. Skips the commit when the baseline bytes haven't changed (no-op pushes on idle main). Uses `git worktree` to isolate the baseline checkout from the main checkout. - `.github/workflows/bench.yml` (new) Triggers: nightly cron (UTC 19:00 = JST 04:00) + manual `workflow_dispatch` + push to main. Manual triggers can pass an optional `pr_number` input to comment the diff to a PR (the on-demand spot-check path). Permissions: `contents: write` (baseline branch push) + `pull-requests: write` (PR comment). - `plugin/package.json` - `test:integration` — now excludes the bench file (`--exclude **/perf.sync.bench.test.ts`) so PR runs of the integration job stay fast. - `test:integration:bench` (new) — runs ONLY the bench file (used by bench.yml). - `plugin/tests/PerfCompare.test.ts` (new, 19 tests) Pure-module unit coverage for the comparison logic — runs in the fast unit suite via vitest's ESM loader (no .mjs build step needed; ~280 ms). Covered: parseNDJSON: empty, well-formed, malformed (line number in error), missing required fields. compare status semantics: new / removed / unchanged (within ±5 % band) / regressed (past tolerance) / improved (clear drop). Per-span tolerance override (S.fs 40 %). Drops to 'new' when baseline p95 is 0 (can't compute %). ordering: regressions surface first (status-grouped), then improvements, unchanged, new, removed, each lex-sorted. isolation: buckets across (span, transport, sizeBytes) tuples stay separate. formatMarkdown: header / table / pass-summary on no- regressions, ❌ + bolded rows on regressions, "no buckets" message on empty input, gate-enabled hint surface. round-trip: NDJSON in → table out, end-to-end smoke. - `.github/workflows/integration.yml` Unchanged from main — this PR deliberately does NOT add perf- related steps to the integration workflow. integration.yml stays the home of functional integration tests; bench.yml is the home of perf bench + baseline. The split keeps each workflow's purpose discoverable from its name. Architecture summary: PR push → integration.yml (no bench, ~30 s) push to main → integration.yml (no bench) → bench.yml (writes perf-baseline) nightly cron → bench.yml (refreshes perf-baseline) workflow_dispatch→ bench.yml (no baseline write; optional PR comment) Verification: - `npx vitest run` — 37 files / 501 tests green (was 482, +19 new) - `npx tsc --noEmit -p tsconfig.json` — clean - `npm run test:integration` (locally if Docker present) — bench excluded via `--exclude` flag; only the functional integration tests run. - workflow YAML: validated against the existing repo's structural patterns. manifest/package/versions bumped 0.4.46 → 0.4.48 (one ahead of the merged PR #105 / M9 at 0.4.47). Roadmap context: - M0 → M9 (Phase C MVP) ✓ - **M10 (this)** wires the gate but off the PR critical path. - Issue #110 (PERF_GATE flip) needs an update to track "flip on schedule + workflow_dispatch in bench.yml" rather than the old "flip in integration.yml" wording. - Issue #109 (E2E expansion) and #108 (modify case revival) are unaffected; they unlock more bench cells, not bench cadence. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
131 lines
5.3 KiB
JavaScript
131 lines
5.3 KiB
JavaScript
#!/usr/bin/env node
|
|
// CLI for the M10 perf gate's baseline maintenance.
|
|
//
|
|
// On `push` to `main` (or a nightly cron), copy the latest head NDJSON
|
|
// from `plugin/perf-results/` onto the orphan `perf-baseline` branch
|
|
// as `baseline.ndjson`, write a sidecar `baseline.sha` pointing at
|
|
// the head commit, and push. PR runs of run-compare.mjs check this
|
|
// branch out into `perf-baseline/` and diff against it.
|
|
//
|
|
// Bootstraps the orphan branch on first run (no `perf-baseline` ref
|
|
// on the remote yet → create one from a clean tree).
|
|
//
|
|
// Env contract (set by the workflow):
|
|
// - PERF_COMMIT_SHA head commit SHA recorded into the sidecar
|
|
// - PERF_RESULTS_DIR override `plugin/perf-results/` (rare)
|
|
// - PERF_BASELINE_REF override `perf-baseline` (rare)
|
|
|
|
import * as fs from 'node:fs';
|
|
import * as path from 'node:path';
|
|
import * as os from 'node:os';
|
|
import { spawnSync } from 'node:child_process';
|
|
import { fileURLToPath } from 'node:url';
|
|
|
|
const here = path.dirname(fileURLToPath(import.meta.url));
|
|
const pluginRoot = path.resolve(here, '..', '..');
|
|
const repoRoot = path.resolve(pluginRoot, '..');
|
|
|
|
const RESULTS_DIR = process.env.PERF_RESULTS_DIR ?? path.join(pluginRoot, 'perf-results');
|
|
const BASELINE_REF = process.env.PERF_BASELINE_REF ?? 'perf-baseline';
|
|
const COMMIT_SHA = process.env.PERF_COMMIT_SHA ?? gitOutput(['rev-parse', 'HEAD']).trim();
|
|
|
|
function main() {
|
|
const headFile = pickLatestNDJSON(RESULTS_DIR);
|
|
if (!headFile) {
|
|
console.log('[perf-baseline] no NDJSON in perf-results/ — bench was skipped or failed; not updating baseline.');
|
|
return 0;
|
|
}
|
|
console.log(`[perf-baseline] head: ${path.basename(headFile)} → ${BASELINE_REF}/baseline.ndjson`);
|
|
|
|
const headText = fs.readFileSync(headFile, 'utf8');
|
|
const tmpRepo = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-baseline-'));
|
|
|
|
try {
|
|
const remoteUrl = gitOutput(['config', '--get', 'remote.origin.url']).trim();
|
|
if (!remoteUrl) throw new Error('perf-baseline: cannot read origin url');
|
|
|
|
// Bootstrap the worktree. fetch the branch if it exists; otherwise
|
|
// initialise an orphan branch with a clean tree.
|
|
runIn(repoRoot, ['git', 'fetch', '--depth=1', 'origin', BASELINE_REF], { allowFail: true });
|
|
const branchExists = gitOutput(['ls-remote', '--heads', 'origin', BASELINE_REF]).trim().length > 0;
|
|
|
|
if (branchExists) {
|
|
runIn(repoRoot, ['git', 'worktree', 'add', '--no-checkout', tmpRepo, `origin/${BASELINE_REF}`]);
|
|
runIn(tmpRepo, ['git', 'checkout', '-B', BASELINE_REF, `origin/${BASELINE_REF}`]);
|
|
} else {
|
|
console.log(`[perf-baseline] bootstrapping orphan branch '${BASELINE_REF}' (first run)`);
|
|
runIn(repoRoot, ['git', 'worktree', 'add', '--detach', tmpRepo, 'HEAD']);
|
|
runIn(tmpRepo, ['git', 'checkout', '--orphan', BASELINE_REF]);
|
|
runIn(tmpRepo, ['git', 'rm', '-rf', '--quiet', '.'], { allowFail: true });
|
|
}
|
|
|
|
fs.writeFileSync(path.join(tmpRepo, 'baseline.ndjson'), headText, 'utf8');
|
|
fs.writeFileSync(path.join(tmpRepo, 'baseline.sha'), COMMIT_SHA + '\n', 'utf8');
|
|
fs.writeFileSync(
|
|
path.join(tmpRepo, 'README.md'),
|
|
[
|
|
'# Phase C perf baseline',
|
|
'',
|
|
'This orphan branch holds the rolling baseline NDJSON used by',
|
|
'the `perf-compare` step in `.github/workflows/integration.yml`.',
|
|
'PR runs diff their fresh bench output against `baseline.ndjson`',
|
|
'and post a Markdown comment on the PR.',
|
|
'',
|
|
'`baseline.ndjson` is rewritten by every push to `main` (or by a',
|
|
'nightly cron) — do not commit changes here directly.',
|
|
'',
|
|
`Last updated from \`${COMMIT_SHA}\`.`,
|
|
'',
|
|
].join('\n'),
|
|
'utf8',
|
|
);
|
|
|
|
runIn(tmpRepo, ['git', 'add', 'baseline.ndjson', 'baseline.sha', 'README.md']);
|
|
|
|
// No-op when the new bytes match what's already on the branch.
|
|
const status = gitOutputIn(tmpRepo, ['status', '--porcelain']).trim();
|
|
if (status.length === 0) {
|
|
console.log('[perf-baseline] no change in baseline contents — skipping commit');
|
|
return 0;
|
|
}
|
|
|
|
runIn(tmpRepo, ['git', '-c', 'user.email=actions@github.com', '-c', 'user.name=github-actions[bot]',
|
|
'commit', '-m', `perf: update baseline from ${COMMIT_SHA.slice(0, 7)}`]);
|
|
runIn(tmpRepo, ['git', 'push', 'origin', `HEAD:${BASELINE_REF}`]);
|
|
console.log(`[perf-baseline] pushed updated baseline to origin/${BASELINE_REF}`);
|
|
return 0;
|
|
} finally {
|
|
try { runIn(repoRoot, ['git', 'worktree', 'remove', '--force', tmpRepo], { allowFail: true }); }
|
|
catch { /* best effort */ }
|
|
}
|
|
}
|
|
|
|
function pickLatestNDJSON(dir) {
|
|
if (!fs.existsSync(dir)) return null;
|
|
const entries = fs.readdirSync(dir).filter((f) => f.endsWith('.ndjson'));
|
|
if (entries.length === 0) return null;
|
|
const sorted = entries
|
|
.map((f) => ({ f, mtime: fs.statSync(path.join(dir, f)).mtimeMs }))
|
|
.sort((a, b) => b.mtime - a.mtime);
|
|
return path.join(dir, sorted[0].f);
|
|
}
|
|
|
|
function runIn(cwd, argv, opts = {}) {
|
|
const r = spawnSync(argv[0], argv.slice(1), { cwd, stdio: 'inherit' });
|
|
if (r.status !== 0 && !opts.allowFail) {
|
|
throw new Error(`${argv.join(' ')} (in ${cwd}) exited ${r.status}`);
|
|
}
|
|
return r.status ?? 0;
|
|
}
|
|
|
|
function gitOutput(args) {
|
|
return gitOutputIn(repoRoot, args);
|
|
}
|
|
|
|
function gitOutputIn(cwd, args) {
|
|
const r = spawnSync('git', args, { cwd, encoding: 'utf8' });
|
|
if (r.status !== 0) return '';
|
|
return r.stdout;
|
|
}
|
|
|
|
process.exit(main());
|