Transactions are too slow, very slow

minhna · August 22, 2019, 10:59am

Usually it takes only dozen of milliseconds to see changes.

podeig · August 22, 2019, 12:29pm

Yes, if changes was initialized without using .rawCollection. in my case. Can it be the reason?

minhna · August 22, 2019, 12:46pm

I don’t have experiences of working with mongodb transaction. I can’t tell.

globalise · August 21, 2020, 1:53pm

I am also encountering the same problem. Operations run using simple Collection.update({…}) the Meteor way are almost instant. Similar operations (2 small updates to separate collections) within a transaction using rawCollection are taking seconds to complete. On local computer it is fine…

znewsham · August 21, 2020, 1:59pm

This is likely due to an optimization in meteor, if you use Collection.update Meteor will immediately push the notification to all clients connected, if you use Collection.rawCollection().update you bypass all Meteor code, so it has no choice but to wait for the oplog notification which requires a round trip to the DB - depending on your writeConcern this may require full replication (and write to disk) for members.

The use of transactions here probably makes things worse - by default Meteor observes the oplog of the primary, and as such the writeConcern doesn’t impact the timing (as soon as the primary tells the secondaries, it tells Meteor too). However with transactions enabled - I’m guessing that the oplog entries Meteor observes are delayed until the transaction is committed.

The reason you don’t see this in dev is you probably only have a single member replica set so no replication time, you will also have close to 0 network delay.

One option you may have is to use redis-oplog - you can push a change to redis independently of pushing the change to mongo, it also has optimizations built in to make this faster on the triggering server. I’ve not tested this myself, and you may have issues with it (I think when redis receives the notification it just repolls the DB to get the latest data, if your transaction isn’t complete it wont come back with a change).

globalise · August 21, 2020, 2:09pm

Hi @znewsham

Thanks very much for the reply. To me it would be an awesome new feature for Meteor to be able to work with MongoDB transactions in a “smart” way without needing to use rawCollection, since for certain types of apps where multiple collections need to be updated together in a relational-style way (e.g. a core document collection, with associated collection for more detailed data associated with the core), it is important to ensure both collections are updated in a controlled way.

Will look into your suggestion with the redis-oplog. Unfortunately I don’t know enough about Meteor inner workings to be able to contribute any improvements myself.

podeig · August 21, 2020, 2:10pm

My understanding of this issue is when you use rawCollection() these operations happen on the server, no in the browser copy of mongodb, it will be synced but it runs each 5-7 seconds. On localhost it must run at once. Without using rawCollection changes happen on both miniMongodb and mongodb on the server. To increase speed I run both rawCollection and an usual update to get changes everywhere at once. Hope it helps.

globalise · August 21, 2020, 2:12pm

Hi @podeig

I am actually running all of my methods in if (Meteor.isServer) { } code, so I don’t think it is to do with the optimistic update in the browser…

podeig · August 21, 2020, 2:13pm

It is more correct explanation.

podeig · August 21, 2020, 2:15pm

I just want to add. To use rawCollection() is the same if you go to you database and change something manually. If you are subscribed to this collection you will see changes in some seconds (on production server, not localhost, of course).

globalise · August 23, 2020, 2:50pm

Reporting back after a bit of experimentation @podeig :

I found a good hack to massively speed this up.

github.com

mikowals/batch-insert/blob/3064f3504d8ff97a3fa8f1fe418d5dc556d4ee63/batch-insert-server.js#L134


_batchInsert = function (collection, docs, cb) {
  var connection = MongoInternals.defaultRemoteCollectionDriver().mongo;
  var write = connection._maybeBeginWrite();
  var _collection = collection.rawCollection();
  var future = new Future;
  _collection.insert( replaceTypes( docs, replaceMeteorAtomWithMongo), {safe:true}, future.resolver());
  try {
    var result = future.wait();
    result = getIdsFromMongoResult(result);
    docs.forEach( function( doc ){
      Meteor.refresh( { collection: collection._name, id: doc._id } );
    });
    write.committed();
    if (cb)
      return cb(null, result);
    return result;
  } catch (e){
    write.committed();
    if (cb)
      return cb(e);
    throw (e);

Meteor.refresh seems to be an undocumented function to force refresh on a document. Simply run it on your collection / document when the transaction successfully returns, and as we say in the UK “Bob’s your uncle”

podeig · August 24, 2020, 7:20am

It looks interesting. Thanks you shared it, I will check it out when I use transactions next time.

a4xrbj1 · August 26, 2020, 9:02pm

A couple comments from my side, as we’re using TXN for a very long time.

I’ve recently posted about how slow it runs with TXN, increasing the poolSize made things a little bit better (for those local queries where we run Data Integration checks, so we fired a whole lot of queries in a short amount of time).

Be aware of writeConflict errors from MongoDb, coming out of the blue. We’re still fighting with them as we haven’t “converted” all of our MongoDb queries to the raw collection. But what MongoDb doesn’t like is if you mix normal collection with raw collection.

Did I mention that writeConflict errors are a PITA to fix? Because MongoDb doesn’t tell you anything. You’re blind. You can’t see what transactions have occured and why it blows exactly on number 387 (or whatever number).

I suggest you also have some additional checks especially when you run anything in a loop, like this:

if (mongoSession.transaction.state === 'NO_TRANSACTION' || /TRANSACTION_COMMITTED/gi.test(mongoSession.transaction.state)) {
            mongoSession.startTransaction();
        }

if (mongoSession.transaction.state === 'TRANSACTION_IN_PROGRESS') {
            await mongoSession.commitTransaction();
        }

and for aborting in your catch branch:

if (mongoSession.transaction.state !== 'TRANSACTION_COMMITTED') {
                await mongoSession.abortTransaction();
            }
            mongoSession.endSession();

superfail · July 17, 2024, 9:14am

@radekmie ping Introduction of DDP Router - #52 by radekmie

radekmie · July 17, 2024, 9:43am

@radekmie Can you confirm that Meteor fallbacks to polling when MONGO_OPLOG_URL is not set, […]

Yes, I can. Setting this environmental variable sets the oplogUrl option here which is then used to create an OplogHandle instance here. The latter is responsible for polling the oplog, so if it’s not there, then there’s no oplog polling.

[…] and that MONGO_OPLOG_URL is properly set when running in development (meteor run )?

It is by default in here, as long as it wasn’t disabled with --disable-oplog as documented here. If you specify the MONGO_URL yourself, then you need to specify MONGO_OPLOG_URL as well.

superfail · July 19, 2024, 3:39pm

Just to be super-clear, I only see two ways reactivity is achieved in Meteor: oplog TAILING (OplogObserveDriver) and database polling (PollingObserveDriver). What is oplog “polling”?

superfail · July 19, 2024, 4:05pm

The default database polling interval is 10 seconds (meteor/packages/mongo/polling_observe_driver.js at 7da5b32d7882b510df8aa2002f891fc4e1ae1126 · meteor/meteor · GitHub) so the average sync time should be 5 seconds, which is close to what I have experienced.

superfail · July 19, 2024, 4:09pm

The oplog is not used if you use MUP and MONGO_OPLOG_URL is not set. On Galaxy, I suspect MONGO_OPLOG_URL should be set properly, so what the OP is experiencing might be related to the cases where observe fall backs to polling if it cannot implement oplog tailing for a particular cursor ($text queries, $natural sorting, etc.), see meteor/packages/mongo/mongo_driver.js at 7da5b32d7882b510df8aa2002f891fc4e1ae1126 · meteor/meteor · GitHub.

radekmie · July 19, 2024, 4:43pm

It’s the same. Tailing refers to cursor tailing which effectively means “get new entries from a cursor when they appear”. But the protocol is still pull-based, i.e., server has to ask the database for new entries (the database does not push new entries to the server), so I call it polling. Sorry for the misunderstanding!

superfail · July 21, 2024, 4:15pm

I think this explanation is still vague.

The Oplog observe driver is mainly driven by oplog tailing, and falls backs to “polling” (i.e. executing the query corresponding to the cursor) or “fetching” individual documents episodically. It only does so at initialization, or when it does not know how to compute the current state of the cursor from the previous state and the next oplog entry.

The Polling observe driver on the other hand, exclusively relies on polling the query, and the default polling period seems to be 10 seconds.

So back to my question: I am speculating that either the OP did not configure MONGO_OPLOG_URL, or since the OP did not share exactly what query they use in the subscription that updates slowly after transactions, that their query cannot be handled by the OplogObserveDriver. In both cases, they should be observing an average delay of 5+ seconds between triggering the action and seeing the updated subscription.

I originally mistakenly assumed that they were deploying on Galaxy, which should rule out a misconfiguration of MONGO_OPLOG_URL. But rereading the thread I realize they meant they are using official MongoDB hosting, but they said nothing about official Meteor hosting.