I have implemented transaction in MongoDB as described in this article:
Utils.js
import { MongoInternals } from 'meteor/mongo';
// utility async function to wrap async raw mongo operations with a transaction
export const runTransactionAsync = async function (asyncRawMongoOperations, errorCode) {
// setup a transaction
const { client } = MongoInternals.defaultRemoteCollectionDriver().mongo;
const session = await client.startSession();
await session.startTransaction();
try {
// running the async operations
let result = await asyncRawMongoOperations(session);
await session.commitTransaction();
// transaction committed - return value to the client
return result;
} catch (err) {
await session.abortTransaction();
console.error(err.message);
// transaction aborted - report error to the client
throw new Meteor.Error(errorCode, err.message);
} finally {
session.endSession();
}
};
Example of using the transactions:
Meteor.methods({
'changeLanguage': async function(poemId, newLanguageId) {
// define the operations we want to run in transaction
const asyncRawMongoOperations = async session => {
const poem = Poems.findOne({ _id: poemId });
const prevLanguage = poem.languageId;
const profile = Profiles.findOne({ userId: this.userId });
await Profiles.rawCollection().update(
{ userId: this.userId },
{
$inc: {
['countLanguages.' + prevLanguage]: -1,
['countLanguages.' + newLanguageId]: 1
},
$addToSet: { languages: newLanguageId }
},
{ session: session }
);
await Poems.rawCollection().update({
'_id': poemId,
'userId': this.userId
},
{
$set: {
languageId: newLanguageId
}
},
{ session: session }
);
return true; // will be the result in the client
};
let result = await runTransactionAsync(asyncRawMongoOperations, 'Error-01');
return result;
}
});
At localhost it works well, fast, with no issue.
But at MongoDB hosting (MongoDB.Atlas. MongoDB v 4.0.12., cluster tier M10) it is very very slow. It can take up to 7 seconds.
The database is almost empty, max 50 records in each collection.
If I remove .rawCollection() it start to work fast, but transactions stops to work.
Thanks Arggh,
I will have to do this, I think.
I see that transactions are quite new topic for Meteor+MongoDB. There is no so much information about this. Meteor team could write more about this, create some examples.
My app is build on React. The component which shows changes is waiting date from subscription. When I run the transaction it works fast and the result from
āchangeLanguageā: async function(poemId, newLanguageId) {
comes at once, no delay.
Delay happens in updating of the object through subscription. So it takes some seconds that subscription gets new changes.
Another observation: If I remove all .rawCollection() in āchangeLanguageā there is no delay, but transaction does not work. So there is something to do with .rawCollection.
So now my question: Is it normal that it takes so much time when subscription see changes? How to improve it?
I am also encountering the same problem. Operations run using simple Collection.update({ā¦}) the Meteor way are almost instant. Similar operations (2 small updates to separate collections) within a transaction using rawCollection are taking seconds to complete. On local computer it is fineā¦
This is likely due to an optimization in meteor, if you use Collection.update Meteor will immediately push the notification to all clients connected, if you use Collection.rawCollection().update you bypass all Meteor code, so it has no choice but to wait for the oplog notification which requires a round trip to the DB - depending on your writeConcern this may require full replication (and write to disk) for members.
The use of transactions here probably makes things worse - by default Meteor observes the oplog of the primary, and as such the writeConcern doesnāt impact the timing (as soon as the primary tells the secondaries, it tells Meteor too). However with transactions enabled - Iām guessing that the oplog entries Meteor observes are delayed until the transaction is committed.
The reason you donāt see this in dev is you probably only have a single member replica set so no replication time, you will also have close to 0 network delay.
One option you may have is to use redis-oplog - you can push a change to redis independently of pushing the change to mongo, it also has optimizations built in to make this faster on the triggering server. Iāve not tested this myself, and you may have issues with it (I think when redis receives the notification it just repolls the DB to get the latest data, if your transaction isnāt complete it wont come back with a change).
Thanks very much for the reply. To me it would be an awesome new feature for Meteor to be able to work with MongoDB transactions in a āsmartā way without needing to use rawCollection, since for certain types of apps where multiple collections need to be updated together in a relational-style way (e.g. a core document collection, with associated collection for more detailed data associated with the core), it is important to ensure both collections are updated in a controlled way.
Will look into your suggestion with the redis-oplog. Unfortunately I donāt know enough about Meteor inner workings to be able to contribute any improvements myself.
My understanding of this issue is when you use rawCollection() these operations happen on the server, no in the browser copy of mongodb, it will be synced but it runs each 5-7 seconds. On localhost it must run at once. Without using rawCollection changes happen on both miniMongodb and mongodb on the server. To increase speed I run both rawCollection and an usual update to get changes everywhere at once. Hope it helps.
I just want to add. To use rawCollection() is the same if you go to you database and change something manually. If you are subscribed to this collection you will see changes in some seconds (on production server, not localhost, of course).
Reporting back after a bit of experimentation @podeig :
I found a good hack to massively speed this up.
Meteor.refresh seems to be an undocumented function to force refresh on a document. Simply run it on your collection / document when the transaction successfully returns, and as we say in the UK āBobās your uncleā
A couple comments from my side, as weāre using TXN for a very long time.
Iāve recently posted about how slow it runs with TXN, increasing the poolSize made things a little bit better (for those local queries where we run Data Integration checks, so we fired a whole lot of queries in a short amount of time).
Be aware of writeConflict errors from MongoDb, coming out of the blue. Weāre still fighting with them as we havenāt āconvertedā all of our MongoDb queries to the raw collection. But what MongoDb doesnāt like is if you mix normal collection with raw collection.
Did I mention that writeConflict errors are a PITA to fix? Because MongoDb doesnāt tell you anything. Youāre blind. You canāt see what transactions have occured and why it blows exactly on number 387 (or whatever number).
I suggest you also have some additional checks especially when you run anything in a loop, like this:
if (mongoSession.transaction.state === 'NO_TRANSACTION' || /TRANSACTION_COMMITTED/gi.test(mongoSession.transaction.state)) {
mongoSession.startTransaction();
}
if (mongoSession.transaction.state === 'TRANSACTION_IN_PROGRESS') {
await mongoSession.commitTransaction();
}
and for aborting in your catch branch:
if (mongoSession.transaction.state !== 'TRANSACTION_COMMITTED') {
await mongoSession.abortTransaction();
}
mongoSession.endSession();
@radekmie Can you confirm that Meteor fallbacks to polling when MONGO_OPLOG_URL is not set, [ā¦]
Yes, I can. Setting this environmental variable sets the oplogUrl option here which is then used to create an OplogHandle instance here. The latter is responsible for polling the oplog, so if itās not there, then thereās no oplog polling.
[ā¦] and that MONGO_OPLOG_URL is properly set when running in development (meteor run )?
It is by default in here, as long as it wasnāt disabled with --disable-oplog as documented here. If you specify the MONGO_URL yourself, then you need to specify MONGO_OPLOG_URL as well.
