Transactions are too slow, very slow

help
1

I have implemented transaction in MongoDB as described in this article:

Utils.js

import { MongoInternals } from 'meteor/mongo';

// utility async function to wrap async raw mongo operations with a transaction
export const runTransactionAsync = async function (asyncRawMongoOperations, errorCode) {

// setup a transaction
const { client } = MongoInternals.defaultRemoteCollectionDriver().mongo;
const session = await client.startSession();
await session.startTransaction();
try {
    // running the async operations
    let result = await asyncRawMongoOperations(session);
    await session.commitTransaction();
    // transaction committed - return value to the client
    return result;
} catch (err) {
    await session.abortTransaction();
    console.error(err.message);
    // transaction aborted - report error to the client
    throw new Meteor.Error(errorCode, err.message);
} finally {
    session.endSession();
}
};

Example of using the transactions:

Meteor.methods({
   'changeLanguage': async function(poemId, newLanguageId) {

        // define the operations we want to run in transaction
        const asyncRawMongoOperations = async session => {

            const poem = Poems.findOne({ _id: poemId });
            const prevLanguage = poem.languageId;

            const profile = Profiles.findOne({ userId: this.userId });

            await Profiles.rawCollection().update(
                { userId: this.userId }, 
                    { 
                        $inc: { 
                            ['countLanguages.' + prevLanguage]: -1,
                            ['countLanguages.' + newLanguageId]: 1 
                        },
                        $addToSet: { languages: newLanguageId }
                    }, 
                    { session: session }
                );

            await Poems.rawCollection().update({
                    '_id': poemId,
                    'userId': this.userId
                },
                {
                    $set: {
                        languageId: newLanguageId
                    }
                }, 
                { session: session }
            );

            return true; // will be the result in the client
        };

        let result = await runTransactionAsync(asyncRawMongoOperations, 'Error-01');

        return result;
    }
});

At localhost it works well, fast, with no issue.
But at MongoDB hosting (MongoDB.Atlas. MongoDB v 4.0.12., cluster tier M10) it is very very slow. It can take up to 7 seconds.

The database is almost empty, max 50 records in each collection.

If I remove .rawCollection() it start to work fast, but transactions stops to work.

Where can be the bottleneck in this case?

Thanks in advance!

2 Likes
2

Maybe try using the Performance API and some timestamps to see where the execution slows down in your code, and then continue (if necessary) by profiling MongoDB: https://docs.mongodb.com/manual/tutorial/manage-the-database-profiler/

1 Like
3

Thanks Arggh,
I will have to do this, I think.
I see that transactions are quite new topic for Meteor+MongoDB. There is no so much information about this. Meteor team could write more about this, create some examples.

4

Iā€™m sure whatever you learn in the process about MongoDB transactions with Meteor is very welcome to the Meteor guide as a PR!

4 Likes
5

I understod this problem.

My app is build on React. The component which shows changes is waiting date from subscription. When I run the transaction it works fast and the result from
ā€˜changeLanguageā€™: async function(poemId, newLanguageId) {
comes at once, no delay.

Delay happens in updating of the object through subscription. So it takes some seconds that subscription gets new changes.

Another observation: If I remove all .rawCollection() in ā€˜changeLanguageā€™ there is no delay, but transaction does not work. So there is something to do with .rawCollection.

So now my question: Is it normal that it takes so much time when subscription see changes? How to improve it?

6

Usually it takes only dozen of milliseconds to see changes.

7

Yes, if changes was initialized without using .rawCollection. in my case. Can it be the reason?

8

I donā€™t have experiences of working with mongodb transaction. I canā€™t tell.

1 Like
9

I am also encountering the same problem. Operations run using simple Collection.update({ā€¦}) the Meteor way are almost instant. Similar operations (2 small updates to separate collections) within a transaction using rawCollection are taking seconds to complete. On local computer it is fineā€¦

10

This is likely due to an optimization in meteor, if you use Collection.update Meteor will immediately push the notification to all clients connected, if you use Collection.rawCollection().update you bypass all Meteor code, so it has no choice but to wait for the oplog notification which requires a round trip to the DB - depending on your writeConcern this may require full replication (and write to disk) for members.

The use of transactions here probably makes things worse - by default Meteor observes the oplog of the primary, and as such the writeConcern doesnā€™t impact the timing (as soon as the primary tells the secondaries, it tells Meteor too). However with transactions enabled - Iā€™m guessing that the oplog entries Meteor observes are delayed until the transaction is committed.

The reason you donā€™t see this in dev is you probably only have a single member replica set so no replication time, you will also have close to 0 network delay.

One option you may have is to use redis-oplog - you can push a change to redis independently of pushing the change to mongo, it also has optimizations built in to make this faster on the triggering server. Iā€™ve not tested this myself, and you may have issues with it (I think when redis receives the notification it just repolls the DB to get the latest data, if your transaction isnā€™t complete it wont come back with a change).

3 Likes
11

Hi @znewsham

Thanks very much for the reply. To me it would be an awesome new feature for Meteor to be able to work with MongoDB transactions in a ā€œsmartā€ way without needing to use rawCollection, since for certain types of apps where multiple collections need to be updated together in a relational-style way (e.g. a core document collection, with associated collection for more detailed data associated with the core), it is important to ensure both collections are updated in a controlled way.

Will look into your suggestion with the redis-oplog. Unfortunately I donā€™t know enough about Meteor inner workings to be able to contribute any improvements myself.

1 Like
12

My understanding of this issue is when you use rawCollection() these operations happen on the server, no in the browser copy of mongodb, it will be synced but it runs each 5-7 seconds. On localhost it must run at once. Without using rawCollection changes happen on both miniMongodb and mongodb on the server. To increase speed I run both rawCollection and an usual update to get changes everywhere at once. Hope it helps. :slight_smile:

1 Like
13

Hi @podeig

I am actually running all of my methods in if (Meteor.isServer) { } code, so I donā€™t think it is to do with the optimistic update in the browserā€¦

14

It is more correct explanation. :slight_smile:

2 Likes
15

I just want to add. To use rawCollection() is the same if you go to you database and change something manually. If you are subscribed to this collection you will see changes in some seconds (on production server, not localhost, of course). :slight_smile:

16

Reporting back after a bit of experimentation @podeig :

I found a good hack to massively speed this up.

Meteor.refresh seems to be an undocumented function to force refresh on a document. Simply run it on your collection / document when the transaction successfully returns, and as we say in the UK ā€œBobā€™s your uncleā€ :slight_smile:

2 Likes
17

It looks interesting. Thanks you shared it, I will check it out when I use transactions next time. :slight_smile:

18

A couple comments from my side, as weā€™re using TXN for a very long time.

Iā€™ve recently posted about how slow it runs with TXN, increasing the poolSize made things a little bit better (for those local queries where we run Data Integration checks, so we fired a whole lot of queries in a short amount of time).

Be aware of writeConflict errors from MongoDb, coming out of the blue. Weā€™re still fighting with them as we havenā€™t ā€œconvertedā€ all of our MongoDb queries to the raw collection. But what MongoDb doesnā€™t like is if you mix normal collection with raw collection.

Did I mention that writeConflict errors are a PITA to fix? Because MongoDb doesnā€™t tell you anything. Youā€™re blind. You canā€™t see what transactions have occured and why it blows exactly on number 387 (or whatever number).

I suggest you also have some additional checks especially when you run anything in a loop, like this:

if (mongoSession.transaction.state === 'NO_TRANSACTION' || /TRANSACTION_COMMITTED/gi.test(mongoSession.transaction.state)) {
            mongoSession.startTransaction();
        }

if (mongoSession.transaction.state === 'TRANSACTION_IN_PROGRESS') {
            await mongoSession.commitTransaction();
        }

and for aborting in your catch branch:

if (mongoSession.transaction.state !== 'TRANSACTION_COMMITTED') {
                await mongoSession.abortTransaction();
            }
            mongoSession.endSession();
1 Like