Mongo Transactions - insert is not being picked up by publication in production with Mongo Atlas

[jam]

I’m experimenting with Mongo Transactions. Locally, everything works great. When I push to production, my publication is not picking up the change – in this case a newly inserted doc. When I do a .find.fetchAsync after the insert, I see the document there but when I observeChanges on the publication, the added is never triggered so even though the document is in the database, the client never receives the document.

I’m using the free Mongo Atlas tier and I’ve configured the MONGO_OPLOG_URL. I see the using oplog in Monti APM when I call the Meteor Method to insert the document via a Transaction. I’ve tried various things with no luck.

Any ideas? Has anyone been able to get this to work in production?

I’m using Meteor 2.14

// settings.json
{
  "galaxy.meteor.com": {
    "env": {
      "MONGO_URL": "mongodb+srv://xxx:yyy@cluster0.fq1piue.mongodb.net/mainDB?retryWrites=true&w=majority&ssl=true",
      "MONGO_OPLOG_URL": "mongodb+srv://oplog:yyy@cluster0.fq1piue.mongodb.net/local?authSource=admin&ssl=true"
    }
  },
  "monti": {
    "appId": "xxx", 
    "appSecret": "xxx" 
  }
}

// /server/main.js
import { Meteor } from 'meteor/meteor';
import { Links } from '/imports/api/links';
import { MongoInternals } from 'meteor/mongo';
import { Random } from 'meteor/random'; 

Meteor.publish('links.all', function() {
  const cursor = Links.find();
  cursor.observeChanges({
    added(id, link) {
      // in production, this isn't triggered
      console.log('link added', id, link) 
    }
  })
  return cursor;
});

// Method
const insertLinkWithTransaction = async () => {
  const { client } = MongoInternals.defaultRemoteCollectionDriver().mongo;
  const session = client.startSession();

  try {
    await session.withTransaction(async() => {
      return await Links.rawCollection().insertOne({ _id: Random.id(), title: 'hello' }, { session });
    })
    // I see the newly inserted link when I console log here
    console.log('links after transaction', await Links.find({}).fetchAsync()) 
    return;
  } finally {
    await session.endSession();
  }
}

Meteor.methods({ insertLinkWithTransaction })

[jam]

When I upgraded to a paid tier of Mongo Atlas, it works as expected. I’m not sure why it wouldn’t work on the free tier.

[paulishca]

I think this information is wrong.

This is an instruction on how to run some queries on a M0 oplog.

Beside that, my APM shows my query ran with the use of oplog on a M0:

[jam]

Yeah I saw the same thing in APM on M0. So maybe the answer is a little more nuanced. I’ll update my previous response.

Do you have any ideas why the Transaction wouldn’t work on M0 but works on a paid tier?

[paulishca]

@jam

Do you have any ideas why the Transaction wouldn’t work on M0 but works on a paid tier?

I think transactions work on the free M0: Is there any way to use transactions with M0 Sandbox? - MongoDB Atlas - MongoDB Developer Community Forums

I think you just need to read the documentation.

[jam]

Thanks for the link. I should I have been more specific when I said “work”. On the M0 tier, operations within a Transaction are reflected in the db, but oplog tailing doesn’t appear to work for the Transaction so the publication in Meteor isn’t picking up the updates.

There must be some limitation here, since with the same set up, everything works as expected on the paid tier. Maybe there is a similar limitation around using $in as there is for change stream filtering on the free tier, though I haven’t found it explicitly mentioned. Mongo removed a lot of its previous content on oplog tailing because it wants people to use change streams. It looks like oplog_tailing.js in Meteor is using that operator.

[paulishca]

I see you commented on this thread. I am curious whether this could be a solution for your problem: Introduction of changestream-to-redis