The lazy connect populate (`lazyFolderLoad`, on by default) walks only the
root's IMMEDIATE children, and `LazyFolderLoader` deepens a folder only when
the user CLICKS it open. Everything below depth 1 was therefore absent from
`vault.fileMap` entirely — and Obsidian resolves links, embeds, search, graph,
backlinks and Quick Switcher only against `fileMap`. So a `[[link]]` into a
never-expanded subfolder silently did not resolve, and an image embedded from
one did not render. A vault that registered 12 of 30,000 files reported
"connected" and looked fine.
Fix the product, not the tests: add `BackgroundIndexer`, which completes the
model in the background AFTER connect, so connect stays as fast as #448/#449
made it.
- Fast path (daemon `fs.walk`): one recursive walk (paginated server-side, so
cheap), fed to `buildChunked` depth-by-depth — parents always land in an
earlier call than their children, and cancellation gets frequent cut points.
- Fallback (SFTP): breadth-first, ONE FOLDER AT A TIME. The fallback costs one
`adapter.list` per directory, so a dir-heavy tree must trickle, not stall.
- Yields between every unit (`requestIdleCallback`, else a 0 ms timer) and uses
a 100-entry chunk — smaller than the connect populate's 500 — so the UI stays
responsive while the user works.
- Safe to race the click-driven `LazyFolderLoader` and the live
`FsChangeListener`: `VaultModelBuilder`'s inserts are idempotent, so a
double-insert is a counted skip, never a duplicate. Completed folders are
`markLoaded()` so a later expand click doesn't re-walk them.
- Never reads file CONTENT — registering the path is all metadataCache needs.
- Cancelled on disconnect (alongside `lazyLoader`) and restarted on reconnect.
A cancelled pass marks nothing, so it degrades exactly back to lazy rather
than leaving a folder that claims to be loaded but is empty.
Progress is surfaced honestly: status-bar text while indexing, one Notice on
completion (suppressed when nothing was missing), `logger.info` every 500
entries. `lazyFolderLoad: false` still does the full eager walk at connect and
skips the background pass entirely — its `fileMap` is already complete.
Also adds `BulkWalker.hasFastPath()` (purely additive) so a caller can CHOOSE
a traversal strategy up-front; `walk()` only reports which path it took after
the expensive part is already done.
Turns green (no test touched):
e2e/links-metadata.spec.ts → "5 — a wiki link into a SUBFOLDER resolves"
e2e/images.spec.ts → "embedded image inside an UNEXPANDED subfolder renders"
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two product defects the new E2E matrix pinned red. Both fixed; both tests go
green off the product, not off a weakened assertion.
1. STALE READ (cache-pressure.spec). `readBuffer` revalidated a cached read on
`s.mtime === cached.mtime` ALONE and never compared the size — though the
very same `stat` returns it, and uses it two lines later to size the
transfer. SFTP reports mtime at 1-SECOND resolution
(`SftpClient`: `mtime: stats.mtime * 1000`), so two edits inside one
wall-clock second collapse onto the same value: the user keeps reading a
version that no longer exists on the server, then saves it back over the
real one. Not theoretical — `reflect.spec.ts` sleeps 1.1 s between remote
edits to stay green, which is this bug wearing a workaround.
Now revalidates on mtime AND size. A same-second edit that preserves the
byte length is still invisible (that needs a content hash or a server-side
change counter); this closes the common case for free. Verified the guard is
load-bearing: disabling the size check makes the new unit test fail.
2. UNINSTALL NEVER PROPAGATED (plugin-code-roundtrip.spec). Both
pullCommunityPlugins and pushCommunityPlugins used a monotonic UNION, so
`community-plugins.json` could only ever GROW: a local uninstall never
reached the remote, and the next connect's pull RESURRECTED it. The user
could not uninstall anything, ever.
A union cannot tell "I never had it" from "I removed it", so `mergePluginIds`
is now a 3-WAY merge against a per-device BASE — the list as it stood at the
end of the last successful sync on this device. Base lives at
`~/.obsidian-remote/state/<profile-id>/community-plugins.base.json`,
deliberately OUTSIDE every vault so writeThroughConfig never mirrors it and
PathMapper never redirects it.
- No base yet -> degrades exactly to the old union (first-run fallback:
removals can't be inferred, but nothing is lost).
- Only the PUSH commits a base, and only once both sides hold the converged
list — a base written by preSpawnPull's pull would make the later connect
read `base \ remote` as a remote removal of this device's own additions.
Never committing on pull also makes pre-spawn + connect idempotent.
- An unreadable or ABSENT remote contributes NO removals — "absent" must
never be read as "everything was uninstalled elsewhere".
- remote-ssh is never removable from either side.
- Tie-break is ADD WINS: re-uninstalling is one click; silently losing a
plugin you just installed is invisible data loss.
- A first bootstrap DISCARDS a stale base: deleting the shadow vault and
reconnecting is the documented recovery step, and without this the first
push would have stripped the user's entire enabled-plugin list off the
remote.
Full suite: 1238 passed (was 1227), no regressions.
sync.spec's create/edit/delete went red once the Settings overlay was closed.
The overlay was not what made them work; removing it just made the whole run
faster and exposed two latent faults the spec had carried all along.
1. THE REAL RACE. `beforeAll` used `connectAndOpenShadow`, which returns as
soon as the PLUGIN LOADS — the SSH connect, the ADAPTER PATCH and
populateVaultFromRemote all land later, at layout-ready. A write fired
before the patch goes to the shadow vault's LOCAL directory and never
reaches the remote. That is the same race restart-settings.spec hit, and
the same pre-connect window #342/#429 lives in. With the overlay up the run
was slow enough that the connect had usually landed first; without it, it
hadn't. Now: capture the shadow path via the building blocks and block on
`waitForShadowVaultLoaded` before any test runs.
2. SILENT SKIPS. The rename was `if (await inlineTitle.isVisible(...))` — when
false it was skipped without a word, leaving the note as `Untitled.md`, and
the test then failed on `remote.exists('e2e-test-<stamp>.md')` with a
message that named nothing. `delete` had the same shape around its confirm
dialog. Note that Playwright's isVisible() does NOT hit-test (an element
under an overlay still reports visible) while click() does — so overlay
presence silently flipped which steps worked. Both branches are gone; every
stage is now asserted and dumps the active file / fileMap / open modals on
failure.
Drive path is now `app.vault.create/modify/delete` via page.evaluate. In the
shadow window `app.vault.adapter` IS the patched adapter, so the write still
traverses every line of plugin code a keystroke would have reached; the only
thing dropped is Obsidian's own contenteditable→debounce→vault.modify step,
which is not the product under test. The REMOTE assertions — independent SFTP
ground truth via RemoteVerifier — are unchanged, and two are stronger:
`edit` now asserts the original text is GONE (a stray append used to satisfy
it), and `delete` now asserts the file was actually THERE first (it could
previously pass vacuously). Fixed waitForTimeout sleeps replaced with
expect.poll against the remote.
Root cause of the click-driven failures, and it is suite-wide.
`launchObsidian` -> `ensurePluginLoaded` -> `dismissTrustDialog` clicks "Trust
author and enable plugins". Obsidian's OWN reaction to that gesture is to open
its Settings dialog on the Community-plugins tab — a `.modal-container.mod-dim`
full-window overlay. Playwright cannot click through it:
locator.click: Timeout 15000ms exceeded.
- element is visible, enabled and stable
- <div data-setting-id="plugins" ...> from <div class="modal-container
mod-dim"> subtree intercepts pointer events
Every spec has been running under that overlay, in every window, since the
trust-dismiss path was added. It went unnoticed because the probes that do NOT
click are immune — page.evaluate over vault/metadataCache, and Node-side
fetches — so the model-level tests passed and only click-driven tests were hurt,
and those merely HUNG (Playwright's default actionTimeout is 0 = wait forever),
which read as "the product is slow", not "the harness is blocked".
`demo.spec.ts` is the single place in the repo that ever knew: it presses Esc,
with a comment saying exactly this. Every other spec ate it.
Fixed at the source: `launchObsidian` now calls the new `dismissBlockingModals`
once, for everyone. It LOGS what it closed, so a modal that IS meaningful (a
host-key prompt, PendingPluginsModal) shows up in the CI log rather than being
silently swallowed. Paired with `actionTimeout: 30_000` (previous commit), a
covered element now fails fast with Playwright's own intercept diagnostic — the
message that solved this — instead of burning the test timeout in silence.
This is why `images › embedded image in a root note renders` regressed to red
and why `links-metadata › 4 — clicking a wiki link opens the target` failed:
neither note was ever opened; the file-row click was intercepted. Both should
now go green. The genuine product reds are untouched.
The first CI run mixed genuine product defects with three HARNESS faults.
A red that isn't a real defect is as corrosive as a green that isn't real,
so separate them. No assertion was weakened.
1. UNBOUNDED ACTIONS (the systemic one). playwright.config.ts set no
`actionTimeout`, so Playwright's default of 0 — wait forever — applied to
every click. Obsidian's File Explorer under Xvfb can leave a node ATTACHED
BUT NEVER ACTIONABLE (hover popovers swallow pointer events, rows paint
lazily), so a bare `.click()` blocked until the 180 s test timeout and
reported nothing. Three tests burned 3.0 min each with no message, reading
as "the product hung" when the harness was simply waiting. Now bounded at
30 s, and the specs additionally dump vault-model / DOM diagnostics on
expiry, so a stuck locator explains itself instead of eating the clock.
2. RENDERER CSP BLOCKED THE BRIDGE PROBE. images' four bridge tests failed
with a contentless `TypeError: Failed to fetch`. Obsidian's renderer CSP
forbids page-JS `fetch()` to `http://127.0.0.1:<port>` (connect-src) —
NOT a ResourceBridge defect. Proof: in the same run the root-note `<img>`
embed test PASSED on the very same URL, because img-src is allowed. The
measurement was wrong, not the product. Bytes and headers are now fetched
from the NODE test process (no CSP), with a real PNG IHDR parse replacing
createImageBitmap; `<img>` rendering is still asserted in-page, which is
the actual user path. Every assertion kept, including "the 2048px source
must come back at the 1024 cap AND smaller than the original" — the check
that catches ResourceBridge silently falling back to the full binary.
3. CROSS-SPEC CONTAMINATION — caused by a real defect. sync.spec's
`create — new note appears on remote`, previously green, went red. The new
specs seed fixture plugins into the remote's community-plugins.json, and
because `pushCommunityPlugins` is a monotonic UNION (the very defect
plugin-code-roundtrip test 6 pins), the ids can never be removed — so every
later spec's shadow vault pulled them in at connect. New helpers/remote-reset.ts
force-restores the remote's community-plugins.json to its exact original
bytes and purges fixture plugin dirs (including the per-device copies under
.obsidian/user/<clientId>/) in afterAll regardless of outcome, plus a
defensive pre-clean in beforeAll. That a test harness MUST do this is itself
evidence of the defect's blast radius — it is recorded in the docblock.
Test 6 is byte-for-byte unchanged and stays red.
Also fixed a FALSE GREEN: links test 8 (Quick Switcher finds a note in an
unexpanded subfolder) searched a folder that test 6 expands earlier in the same
session, so its green proved nothing. It now uses an independent `-sub2/`
fixture that no other test touches. Expected to go honestly red — the same
user-visible harm as test 5.
Tests encode CORRECT behaviour. Assertions were NOT weakened to make them
pass. Several are expected RED — each one is a real defect to fix, not a
test to soften. No test.skip, no test.fixme.
New helpers:
remote-verifier: mkdirp / rmrf / readBinaryFile / setMtime
remote-fixtures: makePng (a REAL decodable RGBA PNG with random pixels —
2048x1536 lands at 10.3 MB, CRC + zlib verified; the old makeOnePngOf was
a 1x1 zero-padded blob that could never support a dimension assertion),
makeSvg, makePdf, makeFakePlugin (a genuinely loadable plugin; version is
a parameter because code convergence is version-ordered), seedPluginOnRemote
obsidian: expandFolderInExplorer — clicks the product's own lazy-load hook
(.nav-folder-title[data-path]) so a deferred folder load can be driven
images.spec.ts (9) — getResourcePath/ResourceBridge/webview. Bridge-level
fetch assertions run BEFORE any preview render so a renderer crash still
leaves a verdict on the bytes served. Asserts the 2048px source is actually
downscaled to the 1024 cap AND is smaller than the original — the byte check
is what catches ResourceBridge silently falling back to the full binary when
the thumbnail RPC fails (it swallows that today). Range requests, SVG,
PDF, and stale-generation-after-remote-replace covered.
EXPECTED RED: an image embedded from an unexpanded subfolder.
fs-visibility.spec.ts (10) — the OPEN HALF OF #429. A plugin doing
fs.writeFileSync(join(adapter.basePath, 'note.md')) must reach the remote
vault; a remote note must be readable via raw fs; getFullPath/getFilePath
must resolve. Two vehicles: in-page window.require('fs'), and a REAL fake
plugin whose onload() does the write — a byte-for-byte reproduction of the
field report.
EXPECTED RED (7 of 10). These cannot be fixed by weakening the assertions.
The only correct fixes are (i) mirror the note tree with a local->remote
watcher, or (ii) patch getFullPath/getFilePath over a real FS-backed view.
links-metadata.spec.ts (8) — metadataCache over the virtual vault.
EXPECTED RED: a wiki link into a subfolder does not resolve, and Quick
Switcher cannot reach a note in an unexpanded folder — because lazy folder
loading walks the ROOT ONLY (main.ts: walk('', !lazy)), so those files are
absent from vault.fileMap entirely and metadataCache resolves only against
fileMap. Test 6 is the discriminator: it tells us whether the fix is an
eager/background full index or metadata invalidation on lazy insert.
cache-pressure.spec.ts (10) — LRU eviction and TTL correctness. Eviction is
PROVEN via readCache.stats(), never inferred; the probe itself is a
precondition test that hard-fails rather than continue unproven.
EXPECTED RED — NEW DEFECT: SftpDataAdapter revalidates a cached read on
mtime EQUALITY ALONE and never compares the remote size, though stat()
already returns it. SFTP mtime is 1-second resolution, so two edits in the
same wall-clock second serve a STALE copy. (That is why reflect.spec.ts has
to sleep 1.1 s to make its own test pass.) Deliberately placed in an SFTP
suite: on RPC the daemon's fs.changed push invalidates the entry and MASKS
the defect, so asserting it there would be flaky-red instead of honest-red.
plugin-code-roundtrip.spec.ts (6) — plugin CODE across the device boundary:
pull, load-on-next-start, push, data.json stays out of the code set, and
no-downgrade version-ordered convergence.
EXPECTED RED — DEFECT: uninstalling a plugin does not propagate.
pushCommunityPlugins computes a monotonic UNION, so a removal can never
reach the remote, and the next connect's pull RESURRECTS it. Needs tombstone
/ real removal semantics in the product.
The first CI run of restart-settings.spec.ts failed with a local-disk
ENOENT:
ENOENT: no such file or directory, open
'.../.obsidian-remote/vaults/E2E Test--vault (4)/.obsidian/plugins/e2e-settings-probe/data.json'
That is the STOCK FileSystemAdapter, not ours: `launchObsidian` returns
once the plugin has LOADED, but the SSH connect + adapter patch only land
later, at layout-ready. The spec wrote immediately and hit the unpatched
adapter, which does not mkdir -p and so died on the missing parent dir.
Fixed by waiting on `waitForShadowVaultLoaded` (the existing oracle the
connect-* specs use: getMarkdownFiles()>=1 AND a file-explorer leaf) after
landing in the shadow window.
The failure is worth recording rather than just fixing: it is a live
demonstration of the exact window #342/#429 lives in — at startup the vault
adapter is NOT ours yet, which is precisely why a plugin's loadData() reads
the local shadow disk and why that disk has to already be warm.
The post-restart assertions keep that property deliberately: the local-disk
check runs BEFORE waiting for the connect (that is the pre-connect window a
plugin's loadData() actually sees), and only then do we wait for the session
to fully reconnect and verify the remote copy was not clobbered by a
defaults re-save.
A green E2E run did not mean the product worked. Three separate ways the
suite could pass while broken, plus the one scenario it never covered.
1. GREEN-BY-SKIPPING. `sync.spec.ts` and `reflect.spec.ts` still used the
old skip gate:
try { obsidian = await connectAndOpenShadow(...); connected = true; }
catch (e) { test.skip(true, `connectAndOpenShadow failed: ${e}`); }
A THROWN connect — i.e. a genuinely broken plugin — was swallowed into a
skip, and CI went green. Same for `if (!connected) test.skip(...)` and for
sshd being down. `helpers/sshd.ts` already exists to prevent exactly this
and even documents why ("that is how 1.0.49 shipped broken"); the
connect-* specs were migrated to `assertSshdReachable()` and these two
were left behind. Migrated now: both hard-fail, and the connect is no
longer wrapped in try/catch. A broken connect is RED.
2. THE RECORDER RAN AS A TEST. `demo.spec.ts` asserts nothing — it drives the
UI to emit PNG frames for the README GIF — but `testMatch: '**/*.spec.ts'`
pulled it into the behavioural "Obsidian E2E smoke" job, where it burns
minutes and can only go red for reasons unrelated to the product. Now
`testIgnore`d by default; `demo-capture.yml` sets E2E_DEMO=1 to opt back
in, so the GIF still regenerates.
3. NOTHING EVER RESTARTED OBSIDIAN. #342/#429 is an ORDERING bug: Obsidian
loads plugins and calls `Plugin.loadData()` — reading data.json off the
LOCAL shadow disk — BEFORE remote-ssh connects and patches the adapter. It
is only observable across a real quit + relaunch. No spec did that.
`tests/integration/restart-roundtrip.e2e.test.ts` is not it: despite the
name it is a vitest that never starts Obsidian — it calls ShadowVaultBootstrap
against a tmpdir and asserts the bootstrap's own output. It proves the pull
works; it never proves Obsidian READS it.
New `e2e/restart-settings.spec.ts` closes that: connect → write a plugin's
data.json through the PATCHED adapter (byte-for-byte what saveData() does)
→ assert the remote stored it PER-DEVICE and not at the shared identity
path → assert the write-through landed it on LOCAL disk → QUIT Obsidian →
RELAUNCH on the SAME vault → assert the settings are still on disk (what
loadData() reads at the next startup) and that the remote copy was not
clobbered by a defaults re-save.
Against a pre-1.1.7 build this fails at the local-disk assertion: the file
is never created, because the write only ever reached the remote. That is
the regression 1.1.7 fixed and, until now, nothing verified end-to-end.
The harness already supported the restart — `connectAndOpenShadow` has always
done quit+relaunch (onto the shadow vault); the same-vault case is three lines.
It was simply never written.
Strip the -beta.N suffix so release.yml accepts the version landing on
main. Syncs plugin + root manifest.json / manifest-beta.json /
versions.json (minAppVersion 1.5.0). Version-only -- no code change.
Ships the #342 / #429 fix: third-party plugin settings now survive a
restart (config write-through to the local shadow disk) and are
per-device on the remote, so two machines can no longer collide on one
settings file.
Review of the write-through found two real defects it had introduced.
1. SYMLINK CLOBBER (the #455 bug class, reintroduced). writeThroughConfig
mirrored every `<configDir>/**` write, and `fs.writeFileSync` FOLLOWS
symlinks. `ShadowVaultBootstrap.installPlugin` symlinks remote-ssh's own
main.js / manifest.json / styles.css in the shadow vault back to the
SOURCE vault's real files, so any adapter write to those paths (e.g.
updating the plugin from inside the shadow window) would have written
straight through the link and overwritten the source vault's install —
bricking it, exactly as #455 described.
Now: lstat the target and refuse to mirror a symlink, and realpath the
parent so a symlinked ANCESTOR (a stale whole-dir plugin symlink) cannot
redirect the write out of the shadow root either. Skipping costs nothing —
the remote write already succeeded, and for plugin CODE the local copy is
owned by the pull/push binary round-trip, not by this cache.
Pinned by a regression test that fails loudly without the guard: with it
disabled the source vault's real file is observably overwritten
('SOURCE-REAL-FILE' -> 'NEW-BUNDLE-BYTES').
2. PATH TRAVERSAL. The containment guard was a `startsWith(configDir + '/')`
on the RAW string, applied before `..` was resolved — so
`.obsidian/../evil.txt` passed the check and `path.join` then resolved it
out of the shadow root (on win32, backslash separators were a second way
through). The vault path reaches the adapter from Obsidian and from other
plugins; it is not trusted input.
Now: resolve first, then verify the resolved path is the shadow root or
inside it. Pinned by a test that asserts nothing lands outside the root.
Full suite green: 80 files, 1225 passed.
The previous commit regenerated package-lock.json with
`npm install --package-lock-only` on Windows, which pruned the
platform-specific optional deps (@emnapi/*) from the tree. `npm ci` then
refused to install (lock out of sync with package.json), failing every
CI job before it ran.
Restore the lockfile's dependency tree verbatim from next and hand-edit
only the two version fields, so the diff is 1.1.7-beta.0 -> 1.1.7-beta.1
and nothing else.
Plugin settings were lost on every restart of the shadow vault — and the
real settings on the remote were destroyed in the process.
Root cause is an ordering problem the connect-time pull cannot fix.
Obsidian loads community plugins during startup, and each plugin's
`onload()` calls `Plugin.loadData()` -> reads
`<configDir>/plugins/<id>/data.json` — BEFORE remote-ssh has connected
over SSH and patched the adapter. That read therefore always hits the
real FileSystemAdapter, i.e. the LOCAL shadow disk. We cannot get in
front of it: the SSH connect is async and lands at layout-ready.
Nothing ever wrote that local copy. `saveData()` went through the
patched adapter straight to the remote, so the local file stayed empty
and every restart booted the plugin on DEFAULTS — which the plugin then
saved back, overwriting the good settings on the remote too. The plugin
CODE survived (main.js/manifest.json/styles.css are round-tripped by
PLUGIN_BINARY_FILES), which is exactly the reported asymmetry: "the
plugin is still installed, but its settings are gone".
Two changes:
1. SftpDataAdapter: write-through. A successful `<configDir>/**` write is
now mirrored onto the local shadow disk with the same bytes that
landed on the remote. The local disk becomes a warm cache, so the
startup read sees the settings this device last saved no matter how
the shadow window was launched (Connect from the source window, or
reopening the vault directly — the latter never runs preSpawnPull).
Best-effort: a mirror failure never fails the remote write. The NOTE
tree stays virtual — only the configDir is mirrored.
2. PathMapper: `plugins/*/data.json` is now per-device, redirected into
`<configDir>/user/<clientId>/`. `saveData()` fires on every settings
change, so a shared remote path would reintroduce the perpetual
write-conflict that the four core config files were redirected to
escape (b0e2773). Deliberately `plugins/*/data.json` and NOT
`plugins` — the plugin's CODE stays SHARED at the identity path so a
plugin installed on one machine still loads on every other. This
required teaching the pattern matcher a `*` segment (exactly one
path segment, never across `/`), and making crossing-point detection
wildcard-aware so listing a plugin's own dir merges in this client's
private data.json.
Multi-device is robust by construction: two devices can no longer
collide on one remote settings file.
Tests: new PathMapper cases pinning settings-private/code-shared and the
`*`-does-not-over-match boundary; new SftpDataAdapter cases pinning the
mirror, the per-device remote path, that notes are NOT mirrored, and that
a mirror failure does not fail the write. Full suite green (1223 passed).
Strip the -beta.N suffix so release.yml accepts the version landing on
main. Syncs plugin + root manifest.json / manifest-beta.json /
versions.json (minAppVersion 1.5.0). Version-only -- no code change.
Extract preSpawnPull's inline remote-path resolution into a pure
`preSpawnRemotePath` (PathMapper.toRemote -> remote-base join) so the
#450 CRITICAL redirect is unit-testable without standing up a standalone
SftpClient. Behavior-preserving -- main.ts calls the extracted function.
The four per-device config files (app/appearance/core-plugins/hotkeys)
must resolve to `<configDir>/user/<clientId>/...`; joining the bare
shared identity path clobbered per-device config on every shadow-window
spawn. New tests pin: per-client redirect for the four files, identity
for shared files (community-plugins.json, plugin binaries), "." base
join, and cross-client isolation.
6-dimension multi-agent review surfaced one CRITICAL and several correctness/
clarity issues in the per-device-config + hide-dotfiles change:
CRITICAL — preSpawnPull bypassed PathMapper (main.ts). The pre-spawn config pull
built `toRemote` by joining the remote base only, so it read the OLD shared
identity path `<configDir>/app.json` and clobbered the freshly-redirected
per-device config on EVERY shadow-window spawn — silently undoing the fix. Now it
applies the same `PathMapper.toRemote` the shadow's adapter uses (from
`resolveClientId(settings)` + configDir), so the four config files pull from this
client's `user/<id>/` subtree; identity for non-private paths keeps
community-plugins.json / plugin binaries shared, unchanged.
MEDIUM — cross-client leak + configDir hardcode (BulkWalker). The dotfile filter
excepted a hard-coded `.obsidian`, which (a) drifted from the configurable
`app.vault.configDir` and (b) let every other client's `.obsidian/user/<id>/*`
private state (incl the 4 newly-redirected files) materialize as editable files
in the File Explorer. Dropped the exception — config is loaded off the local
shadow disk, never from the walked model, so hiding the whole `.obsidian` subtree
costs nothing and isolates foreign clients. `isHiddenPath` simplifies to the
codebase's `startsWith('.')` / `.some()` idiom.
HIGH — misleading "0 files" Notice (main.ts + BulkWalker). An all-dotfile remote
filtered to zero entries and fired "0 files found — check remotePath", the wrong
remediation. BulkWalkResult now carries `hiddenCount`; populate distinguishes
"0 visible, N hidden" with an accurate message and logs the count.
LOW — user-facing Notices + a comment said "shared-config" for files that are now
per-device; corrected. (Constant/method rename of SHARED_OBSIDIAN_CONFIG_FILES,
and a legacy→per-client one-time settings migration, are deferred — see PR notes.)
Tests: BulkWalker dotfile test updated (config dir now dropped) + fallback-path
filter test + hiddenCount assertions; PathMapper cross-machine isolation test;
stale restart-roundtrip comment corrected. tsc/lint clean; vitest 1206 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dogfooding follow-up to the lazy loading in this PR. Two real-machine problems:
1. `.obsidian` config kept raising write-conflicts, blocking work. Root cause
(read from the code, not guessed): a conflict fires when the read cache's
mtime ≠ the remote file's mtime (SftpDataAdapter.writeBuffer). The daemon
compares mtime as UnixMilli on both sides, so a single device's sequential
note edits can't perpetually conflict. What CAN: app.json / appearance.json /
core-plugins.json / hotkeys.json were the only config files NOT redirected
per-client — they sat at the shared identity path, so two sessions (or one
device across reconnects) both wrote the SAME remote file and every settings
save tripped PreconditionFailed on the other's mtime.
Fix = make them per-device: add the four to DEFAULT_PRIVATE_PATTERN_BASENAMES
so PathMapper redirects each into this client's `<configDir>/user/<id>/`
subtree (exactly what workspace.json/graph/cache already do). No shared path
→ the conflict is impossible by construction. The existing shared-config
round-trip keeps working — now on the per-client path, so each machine gets
its own remote backup + cross-session persistence. This is the user's own
ask: "config should be settable per accessing device".
2. `.julia` (and other dot-dirs) cluttered + slowed the tree. Now:
- BulkWalker drops dot-prefixed entries from every walk result (full walk AND
each lazy per-folder deepen), keeping only the vault config dir — matching
Obsidian's own default of hiding dot-names. A dot-DIR hides its whole
subtree, not just its row.
- DEFAULT_WALK_IGNORE_DIRS gains `.julia .cargo .rustup .npm .conda .gem` so
the daemon prunes these huge caches server-side (never walked/transferred),
the perf half of "hide `.julia` by default".
tsc / lint clean; vitest 1204 passed (+ per-device PathMapper assertions and a
BulkWalker dotfile-filter test; config round-trip integration tests still green
because seed-write and pull-read go through the same PathMapper). Ships beta.17.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dogfooding: even chunked (#448/beta.15), a deep dir-heavy vault (work/Vault with
tens of thousands of dirs nested deep) is too heavy to walk + materialise in
full at connect. Load ONE folder level at a time instead.
A real-Obsidian spike settled the two runtime unknowns: (U2) a childless
TFolder still renders an expand arrow (mod-collapsible), so an unloaded folder
is openable; (U1) File Explorer renders from the in-memory model and does NOT
call adapter.list on expand, so deepen-on-expand is driven by a DOM
.nav-folder-title click hook (the element carries the folder's data-path).
- BulkWalker.walk gains a `recursive` param; recursive:false returns just a
folder's immediate children (fs.walk / one-level adapter.list), honouring
walkIgnoreDirs per level.
- LazyFolderLoader: idempotent, deduped per-folder deepen (walk one level ->
buildChunked). Unit tests cover recursive-false walk, idempotency,
concurrent-dedup, markLoaded, retry-on-failure, reset.
- populateVaultFromRemote: with `lazyFolderLoad` (default true) it walks only
the root level, wires the loader, marks root loaded, and installs a
capture-phase .nav-folder-title click hook that deepens on first expand.
`lazyFolderLoad:false` restores the full eager walk. Loader dropped on
disconnect.
ignore (reduce N) and lazy (limit depth) compose; the live fs.changed watch
still works against a partial tree via ensureParents. Graph/search see only
loaded branches for now (accepted; revisit later). tsc/lint/vitest green
(1202). Ships as beta.16.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dogfooding beta.14 (RPC daemon finally up): a profile pointed at a large
remote vault (~/work with tens of thousands of files) froze Obsidian for tens
of seconds on connect and the File Explorer stayed empty. Root cause:
VaultModelBuilder.build materialised every walked entry and fired a
vault.trigger('create') for each in ONE synchronous JS tick — 30k+ inserts +
30k+ events block the main thread.
- VaultModelBuilder: extract the per-entry insert into `insertBuildEntry`
(shared by buildSync); add `buildChunked(entries, chunkSize=500)` that
processes the folders-first list in chunks and yields to the event loop
between them, so the tree fills in progressively and the window stays
responsive. Ordering is preserved (parents before children) and inserts are
idempotent, so a live fs.changed insert landing between chunks can't
double-add.
- populateVaultFromRemote uses buildChunked.
Structure-only caching (persist the walked tree to skip the cold re-walk on
reconnect) and lazy per-folder loading are follow-ups; this alone removes the
freeze. tsc/lint/vitest green (1196). Ships as beta.15.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses /review-pr findings on #447 and the E2E smoke failure they caused.
- F1 (code-reviewer + silent-failure): the connect fast-path trusted
existsSync alone, re-opening the "deploy an unverified binary" gap one layer
above DaemonDownloader's fixed sha-check. Now records `daemonBinarySha` and
re-hashes the cached file on the fast path; a post-write-corrupted binary
fails the check and is re-fetched (network-free). Version + sha must both
match to reuse.
- F2 (silent-failure): a saveSettings() failure after a successful download
was caught by the generic "download failed -> SFTP" handler and misreported.
Marker persistence is now its own non-fatal try — the verified binary is
still returned; a persist failure just re-verifies next connect.
- F3 (comment-analyzer): corrected the dev-install marker comment (a stale
binary is refreshed only when its sha differs, not "always on a bump").
- F4/F5 (pr-test): new ensureDaemonBinary.test.ts covers the locateDaemonBinary
.dev-daemon gate and the fast-path reuse/skip (sha match, sha mismatch,
version mismatch).
E2E fix: the new locateDaemonBinary `.dev-daemon` gate broke the Obsidian smoke
run — E2E stages the daemon via `build:server` (no dev-install), so the staged
binary was unmarked -> gated out -> no RPC -> the live-reflect specs failed.
build-server.mjs now writes the `.dev-daemon` marker; the E2E vault-scaffold
copies server-bin/ verbatim so it rides along. (Release daemons are built via
the Makefile, never marked — correct: users' downloads stay subject to the sha
refresh.)
tsc/lint/vitest green (1193). Still beta.14 (same PR #447).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dogfooding beta.13: the static-daemon fix shipped, but users still ran the
old dynamically-linked binary — the cache was keyed by ARCH only
(cacheHit = existsSync), so a plugin upgrade never re-downloaded the daemon.
And locateDaemonBinary returned any staged binary at the dev filename, so a
prior *download* masqueraded as a dev build and skipped the refresh entirely.
Now (single cached file, no per-version pileup):
- DaemonDownloader validates the cache by SHA against the release manifest,
not mere existence: reuse iff the cached bytes still match the expected sha,
else re-download (atomic overwrite = old dropped). A daemon that didn't
change across a bump is NOT re-downloaded; a changed/stale one is.
- ensureDaemonBinary fast-paths on `settings.daemonBinaryVersion === version`
(cached file present) -> no GitHub round-trip, works offline; on mismatch it
runs the sha re-check and records the new version after provisioning.
- locateDaemonBinary now requires a `.dev-daemon` marker (written by
dev-install) to treat a staged binary as a genuine dev build; an unmarked
download is re-validated instead of trusted. This is what makes an existing
store/BRAT user's stale binary auto-replace seamlessly (no manual delete).
Implements the deferred #406 "re-verify cached binary sha before reuse".
tsc/lint/vitest green (1188). Ships as beta.14.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dogfooding root cause: the released linux/amd64 daemon crashed on the remote
with `libc.so.6: version GLIBC_2.34 not found` (arch was correct — x86-64 on
x86_64). `make -C server cross` runs on ubuntu-latest (glibc 2.35+); Go builds
the host-native linux/amd64 target with CGO auto-ENABLED, so it links
dynamically against the runner's newer glibc and dies on older hosts. The
daemon never wrote its token, RPC timed out, and connect fell back to SFTP.
- server/Makefile: `export CGO_ENABLED := 0` -> fully static binaries, no libc
dependency, run on any Linux glibc. (arm64/darwin were already CGO-off as
cross targets; this makes amd64 consistent. Mirrors build-server.mjs.)
- release.yml: assert the linux/amd64 binary is not dynamically linked after
build, so a CGO regression fails the release loudly instead of shipping a
daemon that only starts on new distros.
Ships as beta.13; the daemon (RPC + fast walk / live watch / thumbnails) will
come up on older-glibc hosts like the reporter's.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dogfooding beta.11: the server-bin self-heal let the daemon deploy, but it
then failed to write its token in time ("ServerDeployer: daemon did not
write .../token within 5000ms"). connectProfile only fell back to SFTP for
DaemonUnavailableError (arch / declined download); every OTHER RPC-startup
failure (deploy / token timeout / handshake) hard-failed the connect, so
runAutoConnect saw a non-CONNECTED state and SKIPPED populate — the vault
stayed EMPTY (the "remote files invisible" report) even though SFTP would
have worked.
Now any daemon-start failure degrades to SFTP (same live SSH channel; only
daemon-only features are lost), reaching CONNECTED so populate runs.
Preserved: DaemonVerificationError (checksum / bad manifest) still fails
LOUD and does NOT downgrade — silently falling back could mask a tampered
binary (#406). The existing verification-loud test still passes; a new test
pins the token-timeout -> SFTP path.
tsc/lint/vitest green (1187). Still beta.11 (same PR #444).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dogfooding beta.10 surfaced the real cause behind the R3 ENOENT: a fresh
`Panza--work` shadow whose `server-bin` was a junction to a since-deleted
`<uuid>` vault. installPlugin used to propagate `server-bin` as a junction
from the source plugin dir; when that target vault was deleted the junction
dangled, so ensureDaemonBinary's `mkdir server-bin` threw ENOENT and the
connect silently fell back to SFTP ("daemon binary unavailable").
- ensureDaemonBinary: on mkdir failure, unlink a stale reparse point at
server-bin (never following into / deleting its target) and recreate a
real per-shadow dir, THEN continue the download. Only degrade to SFTP if
the repair itself fails. Auto-heals existing broken shadows on reconnect.
- installPlugin: only mirror a REAL source server-bin dir (a local dev
build). A junction/symlink source is no longer propagated — the daemon
binary is per-arch, downloaded per-shadow at connect time, not a shared
build artifact like main.js. Stops the breakage at its root.
Tests: server-bin mirrored for a real source dir; NOT propagated for a
junction/symlink source. tsc/lint/vitest green (1186). Still beta.10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the R1/R2/R3 shadow-vault fix, from a 6-agent review.
- resolveLayout no longer reports a spurious migrated=true on every
reconnect for a profile parked in a collision ` (2)` dir: the
early-return now compares `found` to the collision-resolved target,
not the bare `<name>--<tail>`, so it stops a no-op same-path rename +
re-showing the one-time "restart Obsidian" notice. (code-reviewer)
- findShadowByProfileId / uniqueVaultDir no longer silently swallow a
transient read error on this profile's OWN data.json (AV / Dropbox
lock / mid-write), which could fork a duplicate ` (2)` shadow and
strand the real one. Extracted readShadowProfileId(): ENOENT stays
silent (not a shadow), an existing-but-unreadable/malformed data.json
is logged; also narrows a non-string id to "no match".
(silent-failure-hunter, code-simplifier)
- ObsidianRegistry.isOpen now fails SAFE: an unreadable obsidian.json is
treated as OPEN (defer the rename) rather than "not open" (proceed and
risk renaming a live vault — the corruption R2 prevents), and logs.
(silent-failure-hunter)
- Stale comments corrected: class layout diagram, BootstrapResult.migrated
and layoutFor/findShadowByProfileId docs; sanitisePathTail's dead `~`
guard made live so a bare `~` remotePath falls back to `vault`.
(comment-analyzer)
Tests: ObsidianRegistry.isOpen direct unit tests (open/closed/absent/
unregistered/unreadable-defer); collision-reconnect asserts migrated=false;
sanitisePathTail `~` + backslash edges. tsc/lint/vitest green (1184).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BRAT beta.9 field testing on a host holding several independent vaults
surfaced three shadow-vault issues:
R1 Dir name was `<name>--<id8>` (opaque, collapses a host's folders
together). Now `<name>--<remotePath-tail>` (`Panza--work` vs
`Panza--dev`). Identity moves off the dir name: resolveLayout finds
the shadow by data.json `autoConnectProfileId`, so a rename or a
display-name collision never strands config. Colliding names get a
` (2)` dir (uniqueVaultDir) so two vaults never merge one config.
R2 Migration renamed the dir even while the vault was open, corrupting
the live junction/handles on Windows — the root cause of the
reported breakage (dangling plugin dir -> daemon ENOENT -> SFTP).
resolveLayout now defers the rename when ObsidianRegistry.isOpen()
reports the vault open; it migrates on the next closed bootstrap.
R3 A corrupted shadow plugin dir made ensureDaemonBinary throw a raw
ENOENT and silently degrade to SFTP. It now probes server-bin up
front and, if broken, logs + notices how to recover (delete the
vault dir under ~/.obsidian-remote/vaults and reconnect).
Rename and updatePath stay in separate try blocks (#438 review): a
moved dir commits even if the registry update throws. Tests rewritten
around find-by-id, open-safety, collision, split-failure and legacy
`<uuid>` migration. Bumps 1.1.6-beta.10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings #438's friendly-vault-name + legacy migration into the Phase B
branch so #439 merges cleanly into next after #438. Source files
(main.ts, ShadowVaultBootstrap.ts) auto-merged (disjoint regions);
resolved version files to beta.9 (> next's beta.7) and kept BOTH test
suites (plugin-binary round-trip + legacy→friendly migration).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adversarial review found a config-orphaning bug: in resolveLayout the
legacy→friendly migration ran `fs.renameSync` and `registry.updatePath`
in one try/catch. If the rename SUCCEEDED but updatePath then threw
(obsidian.json briefly locked on Windows, full disk), the catch fell
back to `layoutForDir(legacyDir)` — but the config had already physically
moved to the friendly dir. bootstrap then recreated legacyDir empty and
opened a BLANK vault while the real config sat orphaned (and the old
deep-link id broke).
Split the try: once renameSync succeeds the migration is committed and we
ALWAYS return the friendly dir (migrated:true). updatePath is secondary —
a failure is logged and self-heals via bootstrap's later register(). Only
a rename failure (the move itself) falls back to the legacy dir.
Also sort readdir in findPriorFriendlyDir so a (degenerate) multi-match is
deterministic rather than readdir-order dependent.
Test: registry.updatePath throws after a successful rename → session uses
the friendly dir, config carried, legacy not recreated. tsc + eslint
clean; unit suite 1156 pass.
Refs #429#342
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two real defects an independent review found in Phase B:
BLOCKER — plugin-binary clobber/oscillation. The push diff-gate was
string-equality and the pull was "local authoritative", so a machine on
an OLDER plugin version would overwrite the remote's newer copy (and the
two sides ping-ponged forever). Now VERSION-ORDERED by manifest
`version`: pull only when the remote is strictly newer (or absent
locally), push only when local is strictly newer (or remote lacks it).
Converges, never downgrades. + parseManifestVersion/versionGt helpers.
MAJOR — pre-spawn pull unhandled rejection. When the budget fired first,
`finally` disconnected the client and the in-flight read then threw
"not connected", settling the pull promise a second time after the race
had lost → unhandledrejection in the renderer. The pull now runs as a
standalone promise with a mop-up `.catch` (real-error diagnostics still
flow through the withTimeout race into the existing catch).
Also: withTimeout typed its timer id as `number` (Obsidian renderer DOM)
— `window.setTimeout`'s @types/node overload was leaking `Timeout` and
failing `tsc`. And a clear warning that the round-trip is a UTF-8 TEXT
channel (no binary assets).
Tests rewritten to the version-ordered contract: upgrade, anti-downgrade
(both directions), no-manifest skips, convergence-no-ping-pong, per-plugin
error isolation. Integration test 3 → real-SSH downgrade guard.
tsc + eslint clean; unit suite 1166 pass.
Clean per review: B3 standalone SftpClient does not patch the source
vault, no double-connect leak, B1/B2 ordering correct, backward compat OK.
Refs #429#342
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase B/3. Until now the shadow window booted on its STALE local
`.obsidian/` and only pulled the remote config (#342), plugin list
(#434) and binaries (B/2) AFTER Obsidian had already read settings at
startup — so a setting changed on another machine was a session late.
openShadowVaultFor now runs a best-effort, time-boxed pre-spawn pull
between bootstrap and spawn (via a new ShadowVaultManager onBootstrapped
hook). preSpawnPull builds a STANDALONE SftpClient that does NOT patch
the source window's vault adapter (so the user's real vault is never
hijacked), pulls shared config + plugin list + binaries into the shadow
dir, then disconnects. The window then boots on canonical config — no
mid-session reload.
Safety:
- non-interactive by design: the kbd-interactive + host-key callbacks
REJECT instead of prompting, so a 2FA / unknown-host profile falls
through to the shadow window (which prompts exactly once) — no double
prompt, no surprise modal in the source window
- time-boxed (connectTimeoutMs + 8s) via new withTimeout util; a slow
link falls back to spawn-and-catch-up
- the manager swallows any hook throw, so the spawn NEVER blocks
- disconnects in finally; first connect to an unknown host falls back
and the next (host now trusted) gets the canonical boot
Tests: withTimeout (4) + ShadowVaultManager hook order/fallback (2).
The actual SSH connect + canonical-boot effect needs a manual Obsidian
smoke (can't run headless) — the fallback path IS unit-tested.
Refs #429#342
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase B/2. The enabled-plugins LIST round-trips (#434) and the
marketplace installer fetches binaries for registry plugins (#439,
Phase B/1) — but a BRAT / sideloaded plugin isn't on the marketplace,
so its code never reached another machine.
ShadowVaultBootstrap.pull/pushPluginBinaries round-trip the plugin
*code* (manifest.json / main.js / styles.css; never data.json) through
the remote vault's canonical `.obsidian/plugins/<id>/`. Wired into
runAutoConnect AFTER the marketplace installer so the plugins it
fetched live aren't re-pulled (pull only stages what's still missing =
the non-marketplace ones, preserving the installer's live load and
acting as a fallback when a marketplace fetch fails). The push makes
the remote `.obsidian/plugins/` a complete vault. A pulled binary loads
on the next vault open (Obsidian scans the plugins dir at startup).
- pull never overwrites a local file (local install authoritative)
- push is diff-gated (no churn) and skips remote-ssh (self-managed)
- per-file SSH errors are swallowed, never abort the batch
Tests: 7 unit (in-memory fakes) + 3 integration (real docker sshd:
cross-session round-trip, push→fresh-pull, local-authoritative).
Refs #429#342
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase B, step 1. The marketplace installer ran in prepareForAutoConnect
(before connect) and not at all on a reconnect, so a plugin the remote
community-plugins pull adds to the list had no binary staged and didn't
load until the next Obsidian restart — kazink's "plugins not kept
between sessions".
runAutoConnect now re-runs installMissingShadowPlugins right after
pull/pushCommunityPlugins, when the list is current. enablePluginAndSave
loads a marketplace plugin live (no restart); the call is idempotent so
already-installed ids are skipped, and it now also covers the reconnect
path (which never hit prepareForAutoConnect). installMissingShadowPlugins
is made public for this second call site.
Remaining Phase B (separate): binary round-trip for BRAT / non-
marketplace plugins (their binary isn't on the remote), and a pre-spawn
pull so app.json is canonical at boot without a reload.
Refs #429#342
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shadow vault directory is now named `<profile-name>--<short-id>`
(e.g. `My Homelab--a1b2c3d4`) instead of a bare UUID, so Obsidian — which
shows a vault by its directory basename — displays a recognisable name.
The profile id (a UUID) stays the stable backend key.
- `layoutFor(profile)` derives the friendly dir; `resolveLayout` picks
the dir to use and migrates as needed.
- Transitional migration (removable after a few releases): an existing
legacy `<uuid>` dir is renamed once to the friendly name, carrying its
config/plugins, and obsidian.json is repointed (same vault id) via the
new `ObsidianRegistry.updatePath`. The rename runs during bootstrap —
before the shadow window opens — so the vault is never open while it
moves; on failure it falls back to the legacy dir (no data loss).
- A profile rename reuses the existing `--<short-id>` dir as-is rather
than churning the name (the id is the key).
- The renamed/new path isn't in the running Obsidian's cached vault
list, so the connect flow surfaces the one-time restart notice for
`migrated` too (same as a newly-registered vault).
Tests: friendly naming, empty-name fallback, collision-safety, escape
safety, migration (rename + config carried + obsidian.json repointed),
idempotent re-bootstrap, fresh profile, and rename-failure fallback.
Refs #429#342
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Independent integration spec (vitest + docker sshd, no Obsidian UI) that
exercises the real bootstrap + pull/push round-trip against a live SSH
server — the empirical guard the adversarial review found missing:
- community plugins enabled on the remote round-trip into a fresh shadow
(keeping remote-ssh)
- a plugin enabled in session 1 persists into a fresh session 2 (via the
remote as the canonical store)
- push unions with the remote and never drops a plugin enabled elsewhere
(the #437 clobber-guard, verified over real SSH, not just fakes)
- a settings change stays consistent across write → push → fresh-pull
The remote vault's `.obsidian/` is the single source of truth; a fresh
shadow vault stands in for a new session / machine.
Refs #429#342
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
