sotashimozono_obsidian-remo.../plugin/scripts/perf/run-update-baseline.mjs
Souta 6476372550 ci(plugin): Phase C.M10 — perf-bench off the PR critical path; nightly + workflow_dispatch (0.4.48)
Closes the second half of the Phase C MVP, but with a structurally
different cadence than the original M10 design (PR #106 v1).

The bench (`plugin/tests/integration/perf.sync.bench.test.ts`)
takes ~6 minutes per run — 96 % of the integration job's wall
time, all of it in the SSH-tunneled RPC RTT (~41 ms × ~3000 calls).
Running it on every PR was wrong on two counts:

  1. **Feedback latency**: every PR adds 6 minutes to CI for a
     comment-only gate. Souta correctly called out that the bench
     is overkill for the per-PR feedback loop.
  2. **Bench purpose**: the bench is a *trend tracker* and an
     *on-demand perf microscope*. Trends want a stable cadence
     (nightly), not noise from every PR; a microscope wants to be
     pulled out when there's a hypothesis to test, not run on
     every keystroke.

So this PR splits the workflow:

  - **integration.yml** (PR + push to main): runs everything
    EXCEPT the bench. Drops back to ~30 s wall time.
  - **bench.yml** (new, nightly cron + workflow_dispatch +
    push-to-main): runs ONLY the bench. Updates `perf-baseline`
    branch on nightly + main-push. workflow_dispatch with an
    optional `pr_number` input posts the diff comment to the
    named PR — preserves the on-demand "hey what does this
    branch look like perf-wise" capability without paying the
    6-minute tax on every push.

What ships:

- `plugin/scripts/perf/compare.mjs` (new, pure)

  Comparison core. Reads two NDJSON datasets, joins by
  `(span, transport, sizeBytes)`, computes p95 deltas, classifies
  each bucket as regressed / improved / unchanged / new / removed
  against per-span tolerance (default 25 %; S.fs widened to 40 %
  because shared-CI disk variance is high; S.note to 50 % for
  fsnotify debounce noise). Renders a Markdown table for the PR
  comment with arrows (🔴↑/🟢↓/−/🆕/), bolds regressed rows,
  appends a pass/fail summary + a footer with the baseline / head
  SHA pair + run URL. Status-ordered output (regressions first)
  so reviewers see the bad news without scrolling.

- `plugin/scripts/perf/run-compare.mjs` (new, CLI)

  Picks the latest NDJSON in `plugin/perf-results/`, fetches the
  baseline at `<workspace>/perf-baseline/baseline.ndjson`, calls
  `compare()`, posts via `gh pr comment`. **Upserts** the comment
  by sentinel header so a long-running PR with multiple
  workflow_dispatch invocations gets one rolling comment instead
  of a per-trigger spam stack. Exits 0 unless `PERF_GATE=1` AND
  there's a regression past tolerance.

- `plugin/scripts/perf/run-update-baseline.mjs` (new, CLI)

  Bootstraps the orphan `perf-baseline` branch on first run; on
  subsequent triggers, copies the latest NDJSON onto it as
  `baseline.ndjson` + writes a sidecar `baseline.sha`. Skips the
  commit when the baseline bytes haven't changed (no-op pushes
  on idle main). Uses `git worktree` to isolate the baseline
  checkout from the main checkout.

- `.github/workflows/bench.yml` (new)

  Triggers: nightly cron (UTC 19:00 = JST 04:00) + manual
  `workflow_dispatch` + push to main. Manual triggers can pass an
  optional `pr_number` input to comment the diff to a PR (the
  on-demand spot-check path). Permissions:
  `contents: write` (baseline branch push) +
  `pull-requests: write` (PR comment).

- `plugin/package.json`

  - `test:integration` — now excludes the bench file
    (`--exclude **/perf.sync.bench.test.ts`) so PR runs of the
    integration job stay fast.
  - `test:integration:bench` (new) — runs ONLY the bench file
    (used by bench.yml).

- `plugin/tests/PerfCompare.test.ts` (new, 19 tests)

  Pure-module unit coverage for the comparison logic — runs in
  the fast unit suite via vitest's ESM loader (no .mjs build step
  needed; ~280 ms).

  Covered:
    parseNDJSON: empty, well-formed, malformed (line number in
      error), missing required fields.
    compare status semantics: new / removed / unchanged (within
      ±5 % band) / regressed (past tolerance) / improved (clear
      drop). Per-span tolerance override (S.fs 40 %). Drops to
      'new' when baseline p95 is 0 (can't compute %).
    ordering: regressions surface first (status-grouped), then
      improvements, unchanged, new, removed, each lex-sorted.
    isolation: buckets across (span, transport, sizeBytes) tuples
      stay separate.
    formatMarkdown: header / table / pass-summary on no-
      regressions,  + bolded rows on regressions, "no buckets"
      message on empty input, gate-enabled hint surface.
    round-trip: NDJSON in → table out, end-to-end smoke.

- `.github/workflows/integration.yml`

  Unchanged from main — this PR deliberately does NOT add perf-
  related steps to the integration workflow. integration.yml
  stays the home of functional integration tests; bench.yml is
  the home of perf bench + baseline. The split keeps each
  workflow's purpose discoverable from its name.

Architecture summary:

  PR push          → integration.yml (no bench, ~30 s)
  push to main     → integration.yml (no bench)
                   → bench.yml (writes perf-baseline)
  nightly cron     → bench.yml (refreshes perf-baseline)
  workflow_dispatch→ bench.yml (no baseline write; optional PR comment)

Verification:

- `npx vitest run` — 37 files / 501 tests green (was 482, +19 new)
- `npx tsc --noEmit -p tsconfig.json` — clean
- `npm run test:integration` (locally if Docker present) — bench
  excluded via `--exclude` flag; only the functional integration
  tests run.
- workflow YAML: validated against the existing repo's structural
  patterns.

manifest/package/versions bumped 0.4.46 → 0.4.48 (one ahead of
the merged PR #105 / M9 at 0.4.47).

Roadmap context:
- M0 → M9 (Phase C MVP) ✓
- **M10 (this)** wires the gate but off the PR critical path.
- Issue #110 (PERF_GATE flip) needs an update to track "flip on
  schedule + workflow_dispatch in bench.yml" rather than the old
  "flip in integration.yml" wording.
- Issue #109 (E2E expansion) and #108 (modify case revival) are
  unaffected; they unlock more bench cells, not bench cadence.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 08:39:01 +09:00

131 lines
5.3 KiB
JavaScript

#!/usr/bin/env node
// CLI for the M10 perf gate's baseline maintenance.
//
// On `push` to `main` (or a nightly cron), copy the latest head NDJSON
// from `plugin/perf-results/` onto the orphan `perf-baseline` branch
// as `baseline.ndjson`, write a sidecar `baseline.sha` pointing at
// the head commit, and push. PR runs of run-compare.mjs check this
// branch out into `perf-baseline/` and diff against it.
//
// Bootstraps the orphan branch on first run (no `perf-baseline` ref
// on the remote yet → create one from a clean tree).
//
// Env contract (set by the workflow):
// - PERF_COMMIT_SHA head commit SHA recorded into the sidecar
// - PERF_RESULTS_DIR override `plugin/perf-results/` (rare)
// - PERF_BASELINE_REF override `perf-baseline` (rare)
import * as fs from 'node:fs';
import * as path from 'node:path';
import * as os from 'node:os';
import { spawnSync } from 'node:child_process';
import { fileURLToPath } from 'node:url';
const here = path.dirname(fileURLToPath(import.meta.url));
const pluginRoot = path.resolve(here, '..', '..');
const repoRoot = path.resolve(pluginRoot, '..');
const RESULTS_DIR = process.env.PERF_RESULTS_DIR ?? path.join(pluginRoot, 'perf-results');
const BASELINE_REF = process.env.PERF_BASELINE_REF ?? 'perf-baseline';
const COMMIT_SHA = process.env.PERF_COMMIT_SHA ?? gitOutput(['rev-parse', 'HEAD']).trim();
function main() {
const headFile = pickLatestNDJSON(RESULTS_DIR);
if (!headFile) {
console.log('[perf-baseline] no NDJSON in perf-results/ — bench was skipped or failed; not updating baseline.');
return 0;
}
console.log(`[perf-baseline] head: ${path.basename(headFile)}${BASELINE_REF}/baseline.ndjson`);
const headText = fs.readFileSync(headFile, 'utf8');
const tmpRepo = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-baseline-'));
try {
const remoteUrl = gitOutput(['config', '--get', 'remote.origin.url']).trim();
if (!remoteUrl) throw new Error('perf-baseline: cannot read origin url');
// Bootstrap the worktree. fetch the branch if it exists; otherwise
// initialise an orphan branch with a clean tree.
runIn(repoRoot, ['git', 'fetch', '--depth=1', 'origin', BASELINE_REF], { allowFail: true });
const branchExists = gitOutput(['ls-remote', '--heads', 'origin', BASELINE_REF]).trim().length > 0;
if (branchExists) {
runIn(repoRoot, ['git', 'worktree', 'add', '--no-checkout', tmpRepo, `origin/${BASELINE_REF}`]);
runIn(tmpRepo, ['git', 'checkout', '-B', BASELINE_REF, `origin/${BASELINE_REF}`]);
} else {
console.log(`[perf-baseline] bootstrapping orphan branch '${BASELINE_REF}' (first run)`);
runIn(repoRoot, ['git', 'worktree', 'add', '--detach', tmpRepo, 'HEAD']);
runIn(tmpRepo, ['git', 'checkout', '--orphan', BASELINE_REF]);
runIn(tmpRepo, ['git', 'rm', '-rf', '--quiet', '.'], { allowFail: true });
}
fs.writeFileSync(path.join(tmpRepo, 'baseline.ndjson'), headText, 'utf8');
fs.writeFileSync(path.join(tmpRepo, 'baseline.sha'), COMMIT_SHA + '\n', 'utf8');
fs.writeFileSync(
path.join(tmpRepo, 'README.md'),
[
'# Phase C perf baseline',
'',
'This orphan branch holds the rolling baseline NDJSON used by',
'the `perf-compare` step in `.github/workflows/integration.yml`.',
'PR runs diff their fresh bench output against `baseline.ndjson`',
'and post a Markdown comment on the PR.',
'',
'`baseline.ndjson` is rewritten by every push to `main` (or by a',
'nightly cron) — do not commit changes here directly.',
'',
`Last updated from \`${COMMIT_SHA}\`.`,
'',
].join('\n'),
'utf8',
);
runIn(tmpRepo, ['git', 'add', 'baseline.ndjson', 'baseline.sha', 'README.md']);
// No-op when the new bytes match what's already on the branch.
const status = gitOutputIn(tmpRepo, ['status', '--porcelain']).trim();
if (status.length === 0) {
console.log('[perf-baseline] no change in baseline contents — skipping commit');
return 0;
}
runIn(tmpRepo, ['git', '-c', 'user.email=actions@github.com', '-c', 'user.name=github-actions[bot]',
'commit', '-m', `perf: update baseline from ${COMMIT_SHA.slice(0, 7)}`]);
runIn(tmpRepo, ['git', 'push', 'origin', `HEAD:${BASELINE_REF}`]);
console.log(`[perf-baseline] pushed updated baseline to origin/${BASELINE_REF}`);
return 0;
} finally {
try { runIn(repoRoot, ['git', 'worktree', 'remove', '--force', tmpRepo], { allowFail: true }); }
catch { /* best effort */ }
}
}
function pickLatestNDJSON(dir) {
if (!fs.existsSync(dir)) return null;
const entries = fs.readdirSync(dir).filter((f) => f.endsWith('.ndjson'));
if (entries.length === 0) return null;
const sorted = entries
.map((f) => ({ f, mtime: fs.statSync(path.join(dir, f)).mtimeMs }))
.sort((a, b) => b.mtime - a.mtime);
return path.join(dir, sorted[0].f);
}
function runIn(cwd, argv, opts = {}) {
const r = spawnSync(argv[0], argv.slice(1), { cwd, stdio: 'inherit' });
if (r.status !== 0 && !opts.allowFail) {
throw new Error(`${argv.join(' ')} (in ${cwd}) exited ${r.status}`);
}
return r.status ?? 0;
}
function gitOutput(args) {
return gitOutputIn(repoRoot, args);
}
function gitOutputIn(cwd, args) {
const r = spawnSync('git', args, { cwd, encoding: 'utf8' });
if (r.status !== 0) return '';
return r.stdout;
}
process.exit(main());