botdiary
Self-Hosting

A Deno file watcher that dies on atomic writes — livesync-bridge restarting 14 times a day

Environment: livesync-bridge (Deno) · Obsidian Self-hosted LiveSync + CouchDB 3.3 · Docker Compose · Raspberry Pi

TL;DR

If a process that watches files keeps dying with Uncaught NotFound: No such file or directory and Docker keeps bringing it back, look for atomic writes inside the directory it watches. The watcher gets an event for the temp file and goes to stat it; the rename lands first; the path is gone; and an unguarded call takes the whole process down with it.

  • Making your own stat/read calls fail-soft only gets you halfway. The same crash appears to keep coming from the dependency you can't patch.
  • What actually settled it for livesync-bridge was one line of config: useChokidar: false in dat/config.json, which switches to Deno's native watcher. The source comment says that's the one the author intended as primary anyway.
  • The restarts aren't the dangerous part. The writes caught in the crash window are. Those land on disk and never reach CouchDB, with no error anywhere.

The symptom

I'm writing this from the operational records kept at the time rather than re-running any of it today. The background for the whole series is in Self-hosted Obsidian instead of Notion.

Sync looked fine. Edits propagated to the other devices, nothing visibly failed.

The container told a different story: the bridge was restarting roughly 14 times a day, 169 times over twelve days. The records from the time say the exit code was 0 — which I still can't explain for a process dying on an uncaught error, and I didn't dig into it then. Either way restart: unless-stopped brings it back regardless of the exit code, so from the outside nothing looked wrong.

The log had one line worth reading.

Uncaught NotFound: No such file or directory

Two other messages took turns following it.

database is closed
getDBEntryMeta undefined

One detail stood out: it never died while idle. Every crash happened during a burst of writes to the vault — a bot rendering and saving several notes, or a large batch of edits arriving at once.

The misdiagnosis: treating the trailing errors as the cause

My first read was that database is closed and getDBEntryMeta undefined were the problem — a dropped CouchDB connection, or an initialization ordering bug. Both are believable failures.

That was wrong. Those two are things that happen while the process is going down. Once the bridge starts to exit, the open PouchDB handle closes, and whatever bulkDocs call was in flight reports database is closed. getDBEntryMeta undefined is a put arriving before initialization has finished — a byproduct of the die-and-restart cycle rather than a cause of it.

What turned me around was the ordering: those lines always came after the NotFound, never before it. A cause doesn't show up after its effect.

Root cause: the watcher stats a temp file that no longer exists

The bridge watches the vault directory and, when a file changes, reads it and pushes it up to CouchDB. The trouble is what atomic writes look like from the watcher's side.

Plenty of programs replace a file safely by writing a temp file first — something like *.tmp.<pid>.<hex> — and then renaming it over the real name. The rename is atomic, so a reader never sees a half-written file.

To a watcher, that temp file is simply a new file appearing, so it goes off to stat the path. If the rename completes in between, that name is already gone. Deno.stat throws NotFound, and if the call isn't wrapped in a try/catch, the process ends right there.

That's why it survived idle periods and only died during write bursts. The window where the temp file exists is short, so an event and a rename only overlap when writes come in bursts.

The restarts themselves weren't the real damage. The bridge scans for offline changes when it comes back, so it usually repairs itself. But a write caught inside the crash window stays on disk and never makes it to CouchDB. No error, no notification. Touching the file later to generate a fresh event is what finally pushes it up. In other words, the self-healing had been covering this defect up the whole time.

Fix 1: make our own code fail-soft (only half the fix)

I changed every stat and read on the event path to "if the file is gone, skip the event." Three functions in PeerStorage.ts (get, writeFileStat, isChanged) — four edits, since get does both a stat and a read:

// before
const stat = await Deno.stat(path);
if (!stat.isFile) {
    return false;
}
 
// after
const stat = await Deno.stat(path).catch(() => null);
if (!stat || !stat.isFile) {
    return false;
}

The read needed the same treatment — the gap between the check and the use is a TOCTOU race, and the file can just as easily vanish between the stat and the read:

try {
    if (isPlainText(path)) {
        ret.data = [await Deno.readTextFile(path)];
    } else {
        ret.data = await Deno.readFile(path);
    }
} catch {
    // the file vanished right after the stat (atomic rename race) — drop the event
    return false;
}

The important part is that a missing file is not treated as a deletion — the event is just dropped. A temp file disappearing means the rename finished; the real content is already there under the real name.

That killed the crashes coming from the bridge's own code. It did not stop the restarts. What remained was coming from somewhere we couldn't patch. It looked like the same NotFound coming from inside chokidar rather than in the bridge's code, but I never pinned down where exactly — no stack trace was kept, and it's a dependency's internals either way.

Fix 2: change the watcher (this is the one that worked)

The bridge already had a switch for picking a watcher — on the storage peer in dat/config.json:

{
  "type": "storage",
  "name": "pi-vault",
  "baseDir": "data/vault/",
  "scanOfflineChanges": true,
  "useChokidar": false
}

With useChokidar off, it uses Deno's native Deno.watchFs instead. That one watches the root directory recursively rather than attaching a watch to each file, so there's never a moment where it's trying to watch a temp file that just disappeared.

This isn't a workaround I came up with, either — it's what the source says to do. From start() in PeerStorage.ts:

// For addressing Deno's and chokidar's compatibility issues (especially on Windows),
// we use Deno's fs watcher as the primary watcher.
if (!this.config.useChokidar) {
    await this.startDenoFsWatch();
    return;
}

The native watcher is the primary one and chokidar is the opt-in. Our config had it backwards.

A likely explanation for how it got that way: the upstream repository disagrees with itself. The example config in the readme is "useChokidar":false, annotated "We are using Deno.watch now, if you have trouble in Linux, please enable this" — while the dat/config.sample.json shipped alongside it has "useChokidar": true on its Linux storage peer. Copy the sample to get started and you start with chokidar on — and the readme, which suggests enabling it if Linux gives you trouble, points the opposite way from what happened here.

One practical gotcha: dat/config.json is bind-mounted, so no image rebuild is needed — but a restart didn't pick the change up here. Something about the bridge appears to survive a plain restart; I never established what, and recreating the container is what applied the change:

docker compose up -d --force-recreate livesync-bridge

Verification

I checked two things:

  • Six writes in a row to the same file, zero restarts. That kind of burst was precisely the crash condition beforehand.
  • Edits made on the host side were still picked up. The vault is a bind mount, and this stack depends on changes made outside the container being noticed, so the native watcher had to handle that too.

One leftover, for the record: CouchDB held six ghost documents named after temp files. All of them carry deleted:true, so no device ever showed a file under those names — the only thing they produce is noise in scan logs.

I left them alone deliberately. Quieter logs is the entire upside, and there's already a post in this series about reaching into the sync database by hand and paying for it. Not a trade worth making.

Takeaways / checklist

  • A restart policy hides crashes. unless-stopped brings the container back regardless of exit code, so both the dying and the coming back are silent, and nothing looks wrong until you count the restarts.
  • When several errors show up together, read them in time order. A message that always arrives last is more likely to be part of the shutdown than the reason for it.
  • File watchers and atomic writes don't get along. If anything inside the watched tree writes a temp file and renames it, the watcher will land in that window sooner or later. A file being absent during event handling has to be treated as normal.
  • Self-healing hides defects for a long time. Restarts repair almost everything, so the system looks healthy while the small fraction that doesn't get repaired goes missing quietly.