Pub/Sub with Mongodb Change Stream

performance
21

Could we bring this back to the topic at hand. Where is this discussion now with regard to supporting change streams?

22

Looking at the docs, it seems there have recently been some more updates:

Starting in MongoDB 5.1, change streams are optimized, providing more efficient resource utilization and faster execution of some aggregation pipeline stages.

Maybe worth another look? The OPā€™s example looks like a potential drop-in internal replacement for Meteorā€™s publications and oplog tailing.

3 Likes
23

I plan to do an extensive test regarding that, but I doubt the feasibility of it. Maybe some mixed approach could work here (e.g., some publications would use change streams and the rest would still pool oplog), but thatā€™s even more complex.

2 Likes
24

Obviously if thereā€™s no benefit to gain then itā€™s not worth investing time in. But looking forward to your findings.

25

I believe Mongodb itself uses oplog tailing to handle change streams too. But if we use the change stream, there will be one place doing that.
If you can do some tests, please test the case where multiple Meteor servers work with one Mongodb database.
I believe when you have multiple Meteor servers, each one does itā€™s own oplog tailing and thatā€™s problem.
Imagin you have a big system which uses loadbalancing and your numbers of Meteor servers can be few dozens or more and they all doing oplog tailing at the same time.

26

I did some tests with change streams and wrote it down it meteor/meteor#11842.

9 Likes
27

So is the performance good enough to use? Then we may donā€™t need to access oplog to do Meteorā€™s magic (reactive data update).

28

On the 16th of this month Iā€™ll be in a workshop with the local Mongo team in Dubai. A new AWS Region has just been made available, hosted in UAE and MongoDB and AWS are promoting the infrastructure through some events such as workshops, dinners etc.

Please let me know if you have any questions you would want me to ask the team. I have a short list of my own focused on GDPR and sharding. Alternatively we can ask Karen in MongoDB to look into the Meteorā€™s context and hook up with someone in the team to discuss options/opportunities.

I think we should take a more scientific approach and get them involved. After all, Meteor is part of their sales force.

3 Likes
29

As I already wrote in there, I think itā€™s good enough. But the first step would be to expose this API for the users to play with it, and only then consider replacing oplog with that.

3 Likes