My app is build on React. The component which shows changes is waiting date from subscription. When I run the transaction it works fast and the result from
‘changeLanguage’: async function(poemId, newLanguageId) {
comes at once, no delay.
Delay happens in updating of the object through subscription. So it takes some seconds that subscription gets new changes.
Another observation: If I remove all .rawCollection() in ‘changeLanguage’ there is no delay, but transaction does not work. So there is something to do with .rawCollection.
So now my question: Is it normal that it takes so much time when subscription see changes? How to improve it?
I am also encountering the same problem. Operations run using simple Collection.update({…}) the Meteor way are almost instant. Similar operations (2 small updates to separate collections) within a transaction using rawCollection are taking seconds to complete. On local computer it is fine…
This is likely due to an optimization in meteor, if you use Collection.update Meteor will immediately push the notification to all clients connected, if you use Collection.rawCollection().update you bypass all Meteor code, so it has no choice but to wait for the oplog notification which requires a round trip to the DB - depending on your writeConcern this may require full replication (and write to disk) for members.
The use of transactions here probably makes things worse - by default Meteor observes the oplog of the primary, and as such the writeConcern doesn’t impact the timing (as soon as the primary tells the secondaries, it tells Meteor too). However with transactions enabled - I’m guessing that the oplog entries Meteor observes are delayed until the transaction is committed.
The reason you don’t see this in dev is you probably only have a single member replica set so no replication time, you will also have close to 0 network delay.
One option you may have is to use redis-oplog - you can push a change to redis independently of pushing the change to mongo, it also has optimizations built in to make this faster on the triggering server. I’ve not tested this myself, and you may have issues with it (I think when redis receives the notification it just repolls the DB to get the latest data, if your transaction isn’t complete it wont come back with a change).
Thanks very much for the reply. To me it would be an awesome new feature for Meteor to be able to work with MongoDB transactions in a “smart” way without needing to use rawCollection, since for certain types of apps where multiple collections need to be updated together in a relational-style way (e.g. a core document collection, with associated collection for more detailed data associated with the core), it is important to ensure both collections are updated in a controlled way.
Will look into your suggestion with the redis-oplog. Unfortunately I don’t know enough about Meteor inner workings to be able to contribute any improvements myself.
My understanding of this issue is when you use rawCollection() these operations happen on the server, no in the browser copy of mongodb, it will be synced but it runs each 5-7 seconds. On localhost it must run at once. Without using rawCollection changes happen on both miniMongodb and mongodb on the server. To increase speed I run both rawCollection and an usual update to get changes everywhere at once. Hope it helps.
I just want to add. To use rawCollection() is the same if you go to you database and change something manually. If you are subscribed to this collection you will see changes in some seconds (on production server, not localhost, of course).
Reporting back after a bit of experimentation @podeig :
I found a good hack to massively speed this up.
Meteor.refresh seems to be an undocumented function to force refresh on a document. Simply run it on your collection / document when the transaction successfully returns, and as we say in the UK “Bob’s your uncle”
A couple comments from my side, as we’re using TXN for a very long time.
I’ve recently posted about how slow it runs with TXN, increasing the poolSize made things a little bit better (for those local queries where we run Data Integration checks, so we fired a whole lot of queries in a short amount of time).
Be aware of writeConflict errors from MongoDb, coming out of the blue. We’re still fighting with them as we haven’t “converted” all of our MongoDb queries to the raw collection. But what MongoDb doesn’t like is if you mix normal collection with raw collection.
Did I mention that writeConflict errors are a PITA to fix? Because MongoDb doesn’t tell you anything. You’re blind. You can’t see what transactions have occured and why it blows exactly on number 387 (or whatever number).
I suggest you also have some additional checks especially when you run anything in a loop, like this:
if (mongoSession.transaction.state === 'NO_TRANSACTION' || /TRANSACTION_COMMITTED/gi.test(mongoSession.transaction.state)) {
mongoSession.startTransaction();
}
if (mongoSession.transaction.state === 'TRANSACTION_IN_PROGRESS') {
await mongoSession.commitTransaction();
}
and for aborting in your catch branch:
if (mongoSession.transaction.state !== 'TRANSACTION_COMMITTED') {
await mongoSession.abortTransaction();
}
mongoSession.endSession();
@radekmie Can you confirm that Meteor fallbacks to polling when MONGO_OPLOG_URL is not set, […]
Yes, I can. Setting this environmental variable sets the oplogUrl option here which is then used to create an OplogHandle instance here. The latter is responsible for polling the oplog, so if it’s not there, then there’s no oplog polling.
[…] and that MONGO_OPLOG_URL is properly set when running in development (meteor run )?
It is by default in here, as long as it wasn’t disabled with --disable-oplog as documented here. If you specify the MONGO_URL yourself, then you need to specify MONGO_OPLOG_URL as well.
Just to be super-clear, I only see two ways reactivity is achieved in Meteor: oplog TAILING (OplogObserveDriver) and database polling (PollingObserveDriver). What is oplog “polling”?
The oplog is not used if you use MUP and MONGO_OPLOG_URL is not set. On Galaxy, I suspect MONGO_OPLOG_URL should be set properly, so what the OP is experiencing might be related to the cases where observe fall backs to polling if it cannot implement oplog tailing for a particular cursor ($text queries, $natural sorting, etc.), see meteor/packages/mongo/mongo_driver.js at 7da5b32d7882b510df8aa2002f891fc4e1ae1126 · meteor/meteor · GitHub.
It’s the same. Tailing refers to cursor tailing which effectively means “get new entries from a cursor when they appear”. But the protocol is still pull-based, i.e., server has to ask the database for new entries (the database does not push new entries to the server), so I call it polling. Sorry for the misunderstanding!
