Transactions are too slow, very slow

I have implemented transaction in MongoDB as described in this article:

Utils.js

import { MongoInternals } from 'meteor/mongo';

// utility async function to wrap async raw mongo operations with a transaction
export const runTransactionAsync = async function (asyncRawMongoOperations, errorCode) {

// setup a transaction
const { client } = MongoInternals.defaultRemoteCollectionDriver().mongo;
const session = await client.startSession();
await session.startTransaction();
try {
    // running the async operations
    let result = await asyncRawMongoOperations(session);
    await session.commitTransaction();
    // transaction committed - return value to the client
    return result;
} catch (err) {
    await session.abortTransaction();
    console.error(err.message);
    // transaction aborted - report error to the client
    throw new Meteor.Error(errorCode, err.message);
} finally {
    session.endSession();
}
};

Example of using the transactions:

Meteor.methods({
   'changeLanguage': async function(poemId, newLanguageId) {

        // define the operations we want to run in transaction
        const asyncRawMongoOperations = async session => {

            const poem = Poems.findOne({ _id: poemId });
            const prevLanguage = poem.languageId;

            const profile = Profiles.findOne({ userId: this.userId });

            await Profiles.rawCollection().update(
                { userId: this.userId }, 
                    { 
                        $inc: { 
                            ['countLanguages.' + prevLanguage]: -1,
                            ['countLanguages.' + newLanguageId]: 1 
                        },
                        $addToSet: { languages: newLanguageId }
                    }, 
                    { session: session }
                );

            await Poems.rawCollection().update({
                    '_id': poemId,
                    'userId': this.userId
                },
                {
                    $set: {
                        languageId: newLanguageId
                    }
                }, 
                { session: session }
            );

            return true; // will be the result in the client
        };

        let result = await runTransactionAsync(asyncRawMongoOperations, 'Error-01');

        return result;
    }
});

At localhost it works well, fast, with no issue.
But at MongoDB hosting (MongoDB.Atlas. MongoDB v 4.0.12., cluster tier M10) it is very very slow. It can take up to 7 seconds.

The database is almost empty, max 50 records in each collection.

If I remove .rawCollection() it start to work fast, but transactions stops to work.

Where can be the bottleneck in this case?

Thanks in advance!

2 Likes

Maybe try using the Performance API and some timestamps to see where the execution slows down in your code, and then continue (if necessary) by profiling MongoDB: https://docs.mongodb.com/manual/tutorial/manage-the-database-profiler/

1 Like

Thanks Arggh,
I will have to do this, I think.
I see that transactions are quite new topic for Meteor+MongoDB. There is no so much information about this. Meteor team could write more about this, create some examples.

I’m sure whatever you learn in the process about MongoDB transactions with Meteor is very welcome to the Meteor guide as a PR!

4 Likes

I understod this problem.

My app is build on React. The component which shows changes is waiting date from subscription. When I run the transaction it works fast and the result from
‘changeLanguage’: async function(poemId, newLanguageId) {
comes at once, no delay.

Delay happens in updating of the object through subscription. So it takes some seconds that subscription gets new changes.

Another observation: If I remove all .rawCollection() in ‘changeLanguage’ there is no delay, but transaction does not work. So there is something to do with .rawCollection.

So now my question: Is it normal that it takes so much time when subscription see changes? How to improve it?

Usually it takes only dozen of milliseconds to see changes.

Yes, if changes was initialized without using .rawCollection. in my case. Can it be the reason?

I don’t have experiences of working with mongodb transaction. I can’t tell.

1 Like

I am also encountering the same problem. Operations run using simple Collection.update({…}) the Meteor way are almost instant. Similar operations (2 small updates to separate collections) within a transaction using rawCollection are taking seconds to complete. On local computer it is fine…

This is likely due to an optimization in meteor, if you use Collection.update Meteor will immediately push the notification to all clients connected, if you use Collection.rawCollection().update you bypass all Meteor code, so it has no choice but to wait for the oplog notification which requires a round trip to the DB - depending on your writeConcern this may require full replication (and write to disk) for members.

The use of transactions here probably makes things worse - by default Meteor observes the oplog of the primary, and as such the writeConcern doesn’t impact the timing (as soon as the primary tells the secondaries, it tells Meteor too). However with transactions enabled - I’m guessing that the oplog entries Meteor observes are delayed until the transaction is committed.

The reason you don’t see this in dev is you probably only have a single member replica set so no replication time, you will also have close to 0 network delay.

One option you may have is to use redis-oplog - you can push a change to redis independently of pushing the change to mongo, it also has optimizations built in to make this faster on the triggering server. I’ve not tested this myself, and you may have issues with it (I think when redis receives the notification it just repolls the DB to get the latest data, if your transaction isn’t complete it wont come back with a change).

2 Likes

Hi @znewsham

Thanks very much for the reply. To me it would be an awesome new feature for Meteor to be able to work with MongoDB transactions in a “smart” way without needing to use rawCollection, since for certain types of apps where multiple collections need to be updated together in a relational-style way (e.g. a core document collection, with associated collection for more detailed data associated with the core), it is important to ensure both collections are updated in a controlled way.

Will look into your suggestion with the redis-oplog. Unfortunately I don’t know enough about Meteor inner workings to be able to contribute any improvements myself.

1 Like

My understanding of this issue is when you use rawCollection() these operations happen on the server, no in the browser copy of mongodb, it will be synced but it runs each 5-7 seconds. On localhost it must run at once. Without using rawCollection changes happen on both miniMongodb and mongodb on the server. To increase speed I run both rawCollection and an usual update to get changes everywhere at once. Hope it helps. :slight_smile:

1 Like

Hi @podeig

I am actually running all of my methods in if (Meteor.isServer) { } code, so I don’t think it is to do with the optimistic update in the browser…

It is more correct explanation. :slight_smile:

2 Likes

I just want to add. To use rawCollection() is the same if you go to you database and change something manually. If you are subscribed to this collection you will see changes in some seconds (on production server, not localhost, of course). :slight_smile:

Reporting back after a bit of experimentation @podeig :

I found a good hack to massively speed this up.

Meteor.refresh seems to be an undocumented function to force refresh on a document. Simply run it on your collection / document when the transaction successfully returns, and as we say in the UK “Bob’s your uncle” :slight_smile:

2 Likes

It looks interesting. Thanks you shared it, I will check it out when I use transactions next time. :slight_smile:

A couple comments from my side, as we’re using TXN for a very long time.

I’ve recently posted about how slow it runs with TXN, increasing the poolSize made things a little bit better (for those local queries where we run Data Integration checks, so we fired a whole lot of queries in a short amount of time).

Be aware of writeConflict errors from MongoDb, coming out of the blue. We’re still fighting with them as we haven’t “converted” all of our MongoDb queries to the raw collection. But what MongoDb doesn’t like is if you mix normal collection with raw collection.

Did I mention that writeConflict errors are a PITA to fix? Because MongoDb doesn’t tell you anything. You’re blind. You can’t see what transactions have occured and why it blows exactly on number 387 (or whatever number).

I suggest you also have some additional checks especially when you run anything in a loop, like this:

if (mongoSession.transaction.state === 'NO_TRANSACTION' || /TRANSACTION_COMMITTED/gi.test(mongoSession.transaction.state)) {
            mongoSession.startTransaction();
        }

if (mongoSession.transaction.state === 'TRANSACTION_IN_PROGRESS') {
            await mongoSession.commitTransaction();
        }

and for aborting in your catch branch:

if (mongoSession.transaction.state !== 'TRANSACTION_COMMITTED') {
                await mongoSession.abortTransaction();
            }
            mongoSession.endSession();
1 Like