Without multiplexing, not only the Change Streams would be duplicated, but also all of the server-database communication. The former is already an issue on its own since MongoDB will struggle with hundreds of thousands of active Change Streams. The latter is even worse since if multiple people subscribe to the same collection with a slightly different query, every document would be sent multiple times (unnecessarily).
That’s what I wanted to do at first, but it doesn’t work Let’s say you start a Change Stream with a $match stage. When a new document is added, it works. When it’s removed, it’ll also work. But what about an update? It won’t be triggered when the document after the operation won’t match. It can be worked around using fullDocumentBeforeChange, but that, again, puts more load on the database.
This is a level of innovation I haven’t seen for a while in the Meteor, perhaps even the real-time ecosystem. I smell a great product brewing up in there. Congrats!
Hi @radekmie - this sounds so great. I also have a large prod meteor app on 2.8 (trying to migrate to 3) and would love to test this out. Also not on Galaxy.
Now that I am bit more educated on the question, it seems to me Rust is as capable in terms of async IO as Golang is. A bit more low-level perhaps, but gives you more control if you need it. The border between CPU-parallelism and IO-parallelism is a bit blurred in Golang.
I know I said “promise, last question” before, but here is another one. Does relying on change streams instead of the oplog mean no more polling? Does that mean we always get immediate feedback from subscription, even when using “raw” collection methods (transactions)?
As far as I know, change streams include transaction events just like oplog, so I wouldn’t expect any changes here. Have you had any issues with that in the past?
And if I recall correctly, there’s no pooling in Meteor’s MongoDB oplog driver, except for the oplog pooling, of course. But that’s about the same with change streams, as they operate on cursors under the hood, which are still pull-based.
IIRC, all Meteor Collection mutation methods have some wiring (Meteor.refresh, see Transactions are too slow, very slow - #10 by znewsham) to notify the pubsub system that the collections or ids they are subscribed to have changed, so that publications can be updated as fast as possible. Transactions on “raw” collections bypass this mechanism. As you noted, the mutations end up in the oplog anyway and are eventually taken into account by oplog polling. I was wondering if you managed to exploit change streams in a way that requires less or no polling.
There’s no “extra polling” involved. Sure, Meteor does notify the listeners before the mutations are actually committed, but that’s part of the optimistic UI flow, not a “true” optimization. For example, if the mutation fails due to validation, it has to retract the changes somehow.
Change streams are just an interface to oplog (as in, you have to keep enough oplog entries for them to work), and as such, there’s no way to know about transaction entries sooner (assuming you want only the committed ones).
cultofcoders:redis-oplog allows you to bypass oplog pooling with Redis pooling. The difference is that the latter are scoped better and, as such, lead to fewer messages being processed.
When combined with changestream-to-redis (or oplogtoredis), the Meteor server does not push any mutations to Redis directly, but we once again rely solely on oplog (or change streams). That’s the same as using rawCollection with some additional latency to communicate with Redis.
Finally, the DDP Router is somewhat closest to a dedicated Meteor server using cultofcoders:redis-oplog and running both Redis and changestream-to-redis inside. That means it doesn’t know about your mutations (because it doesn’t see them) and only publishes changes committed in oplog.
So the lag we witness with transactions can be attributed to a lack of stubs together with slow database commits? Nothing to do with polling of the oplog or something similar?
As far as I know how transactions work, yes, that’s it. They produce almost standard oplog entries (there’s additional information that they originate from a transaction; I’m not sure whether single operations are registered as “trivial transactions”), and as such, they require no additional handling.
It also depends on what you call “slow”. A few ms (rather 1-3, not 7-9) of delay between the nodes + a few on top to commit seems doable even for a sizable database. When it gets really busy, sure, let’s make it low tens (e.g., 20ms) total. But that’s about it – change is registered in oplog and it’s up to us to pick it up as soon as possible.
Can you see a difference between Collection.rawCollection().updateAsync() vs. Collection.updateAsync()?
If yes, potentially, Meteor is sending the changes without waiting for the oplog when using the native meteor function vs. waiting for the oplog updates when using rawCollection(), i.e., it might not be about transactions at all.
I saw this thread but did not investigate it myself. There has to be some problem in there and I doubt it’s related to transactions alon.e (I actually tested it just now with two mongosh opened – committed transaction was in oplog immediately.)
The newly added Collection.updateAsync is just like Collection.update, i.e., it’s wired with the Meteor optimistic update.
Let’s say you start a Change Stream with a 
