Now that I am bit more educated on the question, it seems to me Rust is as capable in terms of async IO as Golang is. A bit more low-level perhaps, but gives you more control if you need it. The border between CPU-parallelism and IO-parallelism is a bit blurred in Golang.
I know I said “promise, last question” before, but here is another one. Does relying on change streams instead of the oplog mean no more polling? Does that mean we always get immediate feedback from subscription, even when using “raw” collection methods (transactions)?
As far as I know, change streams include transaction events just like oplog, so I wouldn’t expect any changes here. Have you had any issues with that in the past?
And if I recall correctly, there’s no pooling in Meteor’s MongoDB oplog driver, except for the oplog pooling, of course. But that’s about the same with change streams, as they operate on cursors under the hood, which are still pull-based.
IIRC, all Meteor Collection mutation methods have some wiring (Meteor.refresh, see Transactions are too slow, very slow - #10 by znewsham) to notify the pubsub system that the collections or ids they are subscribed to have changed, so that publications can be updated as fast as possible. Transactions on “raw” collections bypass this mechanism. As you noted, the mutations end up in the oplog anyway and are eventually taken into account by oplog polling. I was wondering if you managed to exploit change streams in a way that requires less or no polling.
A lot to unpack here 
- There’s no “extra polling” involved. Sure, Meteor does notify the listeners before the mutations are actually committed, but that’s part of the optimistic UI flow, not a “true” optimization. For example, if the mutation fails due to validation, it has to retract the changes somehow.
- Change streams are just an interface to oplog (as in, you have to keep enough oplog entries for them to work), and as such, there’s no way to know about transaction entries sooner (assuming you want only the committed ones).
cultofcoders:redis-oplogallows you to bypass oplog pooling with Redis pooling. The difference is that the latter are scoped better and, as such, lead to fewer messages being processed.- When combined with
changestream-to-redis(oroplogtoredis), the Meteor server does not push any mutations to Redis directly, but we once again rely solely on oplog (or change streams). That’s the same as usingrawCollectionwith some additional latency to communicate with Redis. - Finally, the DDP Router is somewhat closest to a dedicated Meteor server using
cultofcoders:redis-oplogand running both Redis andchangestream-to-redisinside. That means it doesn’t know about your mutations (because it doesn’t see them) and only publishes changes committed in oplog.
1 Like
So the lag we witness with transactions can be attributed to a lack of stubs together with slow database commits? Nothing to do with polling of the oplog or something similar?
As far as I know how transactions work, yes, that’s it. They produce almost standard oplog entries (there’s additional information that they originate from a transaction; I’m not sure whether single operations are registered as “trivial transactions”), and as such, they require no additional handling.
It also depends on what you call “slow”. A few ms (rather 1-3, not 7-9) of delay between the nodes + a few on top to commit seems doable even for a sizable database. When it gets really busy, sure, let’s make it low tens (e.g., 20ms) total. But that’s about it – change is registered in oplog and it’s up to us to pick it up as soon as possible.
Can you see a difference between Collection.rawCollection().updateAsync() vs. Collection.updateAsync()?
If yes, potentially, Meteor is sending the changes without waiting for the oplog when using the native meteor function vs. waiting for the oplog updates when using rawCollection(), i.e., it might not be about transactions at all.
I saw this thread but did not investigate it myself. There has to be some problem in there and I doubt it’s related to transactions alon.e (I actually tested it just now with two mongosh opened – committed transaction was in oplog immediately.)
The newly added Collection.updateAsync is just like Collection.update, i.e., it’s wired with the Meteor optimistic update.
So is there any update on whether this will be available to all or a Galaxy-only feature ?
We’re running some tests now with both Galaxy and non-Galaxy projects.
I think in my case it might well be that this is a configuration issue with the deployment method I am using: I never saw the issue in development, only in production when deployed with Meteor Up.
@zodern Can you confirm that the oplog is not used in a Meteor Up deployment if METEOR_OPLOG_URL is not explicitly set? To me it looks like Meteor up expects the user to setup there own independent MongoDB instance but still allows, as a convenience, to spawn a MongoDB configured directly within MUP’s configuration. Meteor oplog was not initially supported through this convenience method (Add oplog support for MongoDB · Issue #21 · zodern/meteor-up · GitHub), and I guess once support was added, using it was not made the default behavior (Add oplog to docs · zodern/meteor-up@48cc313 · GitHub).
@radekmie Can you confirm that Meteor fallbacks to polling when MONGO_OPLOG_URL is not set, and that MONGO_OPLOG_URL is properly set when running in development (meteor run)?
Can we move this discussion elsewhere? I’d like this thread to stay only about the DDP Router.
Feel free to ping me somewhere else if needed.
I would like to raise this thread 
How is it going with DDP router?
Currently we have 1 real-life project fully working. It took a long time, but mostly because we cannot get the codebase to test, but rather work with logs (which not always are directly accessible, as they may include some sensitive information, etc.). We’re monitoring the performance difference in it.
Compatibility with another project is being ironed out now, but again, it takes time.
Overall it works, and only gets better. But due to the nature of the problem (i.e., fetching data), it’s really hard to find issues – “sometimes something’s wrong” is where most of the issues we have start
4 Likes
That’s good to hear 
And any further clarity on the project’s status? For everyone or just for Galaxy?
So far we were focused on testing it – nothing’s decided yet.
That’s why I wrote that mergebox is not a trivial problem and it will take a lot of time to iron things out. But once you finish then it’s very nice so we are waiting for good news
Actually, mergebox is the only thing that had no bugs so far 