I’m still investigating what’s causing it and where it exactly stalls but it could be related to upgrading to Meteor 3.5 from 3.4.1 - has anyone else using passwordless and is experiencing the following:
Login never reached data-ready — and the client onLogin hook runs after data-ready, so it never even ran (loginSuccessful was never called)
? Will update this thread when I get more information from instruments.
I really think this will be fixed on this reported fix that has been merged to Meteor 3.5.1.
meteor:release-3.5.1 ← TimHeckel:fix/changestream-fence-queue-deadlock
opened 07:18PM - 10 Jul 26 UTC
## Summary
Meteor 3.5's `ChangeStreamObserveDriver` can deadlock a collection's… `ObserveMultiplexer` queue against a DDP write fence. Once wedged, the method's `updated` message is never sent (the client's method callback never fires — it sits at `gotResult: true, dataVisible: false` forever), and every subsequent method that writes the same collection hangs on the dead multiplexer too. The wedge is silent: the `"Meteor: change stream catching up took too long"` watchdog never fires because the wait is not in `_waitUntilCaughtUp`.
The canonical victim is **login**, because a login method both writes `users` (the token) and causes new `users` observers to be created mid-method (`setUserId` reruns the userData publications). In our app (3.4 → 3.5 upgrade), `Meteor.loginWithPasswordAnd2faCode` hung deterministically — the client stayed in `loggingIn: true` forever and users were bounced back to the login screen on reload. Plain `loginWithPassword` sometimes survived, because the deadlock is race-dependent (see below).
## Mechanism
1. A method writes to a collection (e.g. the login token write to `users`). The driver's `listenAll` callback in `_startListening` registers the fence-sync `onBeforeFire` handler on the method's write fence.
2. Still inside the method, new observers on that collection are created (login: `setUserId` reruns user publications). Each new driver's `_sendInitialAdds` runs with the method's fence current, so it parks an extra write: `this._writesToCommitWhenReady.push(fence.beginWrite())`.
3. When the new driver finishes its snapshot, `_flushWritesToCommit()` commits those parked writes **inside a multiplexer queue task**: `await this._multiplexer.onFlush(async () => { for (const write of writes) await write.committed(); })`.
4. If one of those commits is the **last outstanding write on the already-armed fence**, `committed()` awaits `WriteFence._maybeFire()`, which awaits all `onBeforeFire` callbacks…
5. …and the fence-sync handler from step 1 calls `await driver._multiplexer.onFlush(...)` — i.e. `queue.runTask(...)` — **on the same multiplexer queue that is currently blocked inside the step-3 task**. The new task is queued behind the running task; the running task is awaiting the new task. Circular wait, forever.
Because step 4 requires the parked commit to be the last write on an *armed* fence, the bug is timing-dependent — which is why simple logins sometimes pass while 2FA logins (extra writes/hooks shift the ordering) hit it every time.
Diagnostics that confirmed this (instrumented `WriteFence.beginWrite`/`arm` with caller stacks + a watchdog on multiplexer queue tasks):
```
[FENCE-DEBUG] fence stuck >5s (via arm). outstanding: 2
pending writers:
- packages/mongo/changestream_observe_driver.js:107 (onBeforeFire beginWrite)
[MPLEX-TRACE] task RUNNING >5s (runTask): async () => { if (!this._ready()) throw ...; await cb(); } <- onFlush cb awaiting committed()
[CS-TRACE] onFlush STUCK 5s <driverId> queueLen: 1 running: true active: 0 <- fence's onFlush queued behind it, never runs
```
## Fix
In `_flushWritesToCommit`, start each `committed()` in a microtask inside the `onFlush` callback instead of awaiting it. The ordering guarantee that `onFlush` exists to provide is preserved — commits still begin only after the flush point, so clients receive `added`/`changed` before the fence's `updated` — but the queue task completes without holding the queue across the fence fire. The fence's own `onBeforeFire` → `onFlush` task can then run normally and the fire completes.
## Repro sketch
Any app shape where a method writes a collection **and** triggers creation of new observers on that collection mid-method:
- accounts-password (+ accounts-2fa makes it deterministic in our experience): client calls login → token write → `setUserId` → userData publications create fresh `users` observers → hang.
- Should also be reproducible without accounts: a method that writes doc X to collection C and, via `Meteor.server.publish_handlers` rerun / a new subscription racing the method, causes a first-ever observer on C to initialize during the method.
Verified in our application (MongoDB 6.0 replica set): before the patch, 2FA login hung indefinitely on every attempt; with this one-line-of-behavior change, login completes instantly, repeatedly, with no stuck-fence warnings under a 5s fence watchdog, and reactivity behaves normally afterward.
Possibly related: #14521 — a wedged multiplexer queue also stops delivering `changed`/`removed` to subscribed clients, which matches that issue's "server state correct, client Minimongo stale" symptom.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_011Ry6SkCcLCsYwnhTQkERJZ
We are about to release Meteor 3.5.1 with the fix, lets confirm is fixed when beta is available. I’ll let you know.
Update: Added a regression test
meteor:release-3.5.1 ← meteor:change-stream-queued-deadlock-test-regression
opened 03:07PM - 23 Jul 26 UTC
Context: https://github.com/meteor/meteor/pull/14564 and https://forums.meteor.c… om/t/is-anyone-else-having-login-problems-with-passwordless-after-entering-tokens-with-version-3-5/64705
- With the PR #14564 fix (https://github.com/meteor/meteor/pull/14564/changes): passes
- After reverting _flushWritesToCommit to await commits inside onFlush: fails, because the fence callback cannot re-enter the multiplexer queue
It’s confirmed, this is the bug that has broken my passwordless login.
As a temporary fix, I have added:
"packages": { "mongo": { "reactivity": ["oplog","polling"] } }
to the METEOR_SETTINGS file on my cloud provider. That has finally fixed things. I don’t know how many tokens I’ve wasted trying to fix this bug but it has taken me several days in real life.