MongoDB Change Streams support in Meteor

announce
1

:memo: [EDITED]
Hey everyone! :wave:

I’ve been working on a long-awaited feature: support for MongoDB Change Streams in Meteor!

This new capability allows you to listen to real-time events directly from MongoDB (like inserts, updates, and deletes), using either a new watchChangeStream api.

I just opened a PR in our github.

:white_check_mark: Work fronts

  1. I’m working in replace the current Oplog Observe Driver by a Change Stream observer driver, it will makes the meteor reactivity works via chnages streams instead oplog or pooling

  2. rodrigooler (Rodrigo Oler) · GitHub is working on the new change stream API, it should works similar to current observeChanges

Mongo.Collection.watch

You can now open a MongoDB Change Stream directly on a server-side collection:

// server.js 
const changeStream = Mongo.Collection.watch([
  { $match: { operationType: 'insert' } }
]);

changeStream.on('change', (change) => {
  console.log('Detected change event:', change);
});
changeStream.on('error', (err) => {
  console.error('Change stream error:', err);
});

:warning: Still a Work In Progress

This is the first working implementation, and while it’s already functional, I’d love to hear your feedback on the developer experience (DX), naming, ergonomics, and potential edge cases.

Let me know your thoughts, suggestions, or use cases you’d like this to support!

Thanks :pray:

8 Likes
2

Great work!

I assume you meant to write observeChangesAsync in the example?

2 Likes
3

Looks interesting! Does find() take in a query in the const handle example?

I imagine not too hard to whip up a similar feature on the client using Tracker

4

Amazing news! We’ve been using our own homebrew method for a while, and while it held well in production, I would be more comfortable using something already baked into Meteor.

One thing I made sure from the start was to prevent opening too many change streams (we have tens of collections) because that can become a performance bottleneck in specific situations. So we open one stream, something like this:

const WATCHABLE_COLLECTION_NAMES = ['CollectionA', 'CollectionB']
const db = this.client.db();
const pipeline = [{
    $match: {
        'ns.coll': { $in: WATCHABLE_COLLECTION_NAMES }
    }
}];
this.changeStream = db.watch(pipeline);
//... then separate logic for filtering and processing the events
// try { const change = await this.changeStream.next(); } catch (err) {}
// ...

The approach makes sense in our scenario, where we have to watch most documents and the number of events is not huge.

How does it work in your example above, would something like this open two streams?

const changeStream = MyCollection.watchChangeStream([
  { $match: { operationType: 'insert' } }
]);
//...
// then later on, maybe in a separate module or package
const changeStream2 = MyCollection.watchChangeStream([
  { $match: { operationType: 'delete' } }
]);
2 Likes
5

Nice this will be great to have in Meteor!

Some initial thoughts:

  1. I’m not sure how I feel about the changeStreams flag in observeChanges. Might be easy to miss. Also as radekmie mentioned in the PR comments, I think they represent two different things. Maybe something like .find().watchChanges() is more appropriate. What were the other options that you considered?
  2. Why not just use Collection.watch instead of Collection.watchChangeStream? This would align more closely with the nodejs mongo driver.

I hope that the Meteor team will steal the ideas and functionality from jam:pub-sub. :slight_smile: I don’t think it gets easier from a DX perspective than Meteor.publish.stream. See here for more info on how it uses change streams: GitHub - jamauro/pub-sub: Publish / subscribe using a Method and/or Change Streams, and cache subscriptions for Meteor apps

6 Likes
6

Thank you for your suggestions @jam , I’ll definitely take a look at your package!

To be honest, the .find().watchChanges() method is only included for backward compatibility purposes. I’m not sure if I’ll keep it; it will depend on the feedback from the community here.

As for the current MyCollection.watchChangeStream, I agree — I’ll rename it to MyCollection.watch. Thanks for the suggestion!

1 Like
7

That’s a great setup I really liked your scenario @illustreets!

It makes total sense to avoid opening multiple change streams when you’re dealing with many collections. Thanks for sharing that pattern; it’s definitely valuable.

And yes, in the current implementation, calling MyCollection.watchChangeStream() multiple times like in your example would open separate streams. But I agree with your approach — it’s much more efficient to centralize and filter events on a single stream when possible.

I’ll incorporate this idea into the next push for sure! :raised_hands:

8

@permb still didnt worked on observeChangesAsync, I was expecting to know the community opnion first

9

@msavin i dont think so, do you have any sceanario in mind?

10

Hurr durr

I hope I’m not missing the mark but I’m trying to look at this from the perspective of someone who’s entirely new to Meteor (Which is I’m since I’m constantly learning new things about Meteor everyday!!).

How does this impact the current state of Meteor? Do meteor applications scale better now? Do we get performance improvements? If so, by how much? What about backwards compatibility? What migration paths are we talking about here for old applications to reap the benefits of this new feature?

Also, why wasn’t @jam package integrated into the core? I feel its DX is way better and offers other cool benefits. Mainly subs caching which I feel is a huge missing part of pub-sub in Meteor.

The efforts to improve pub-sub are split into the following categories:

  • Observing Changes; How Meteor learns about changes (polling, redis-oplog and change streams),
  • Splitting Responsibility; Does the server handle calculating which documents got changed and which didn’t? Or maybe it’s the client responsibility. (mergebox and publication strategies)
  • Subs-caching; How often the client re-fetches those changes which packages like meteorhacks:subs-manager and ccorcos:subs-cache took a stab at.
  • Websockets; The method of communicating such changes (sockjs). I read somewhere that ws is more performant. This also has to do with DX which feathersjs comments on it compared to Meteor.

You @italojs implemented one of @zodern’s packages into the core so what prevented it this time?

After this PR gets merged when you do Meteor.publish it’d utilize change-streams instead of polling? Or I’d have to explicitly enable it by tapping into these weird APIs watchChangeStream or setting true on observeChanges?

In a nutshell, how’d this code be impacted in any shape/form or performance post this update?

import { Meteor } from 'meteor/meteor';
import { TasksCollection } from '/imports/db/TasksCollection';

Meteor.publish('tasks', function publishTasks() {
  return TasksCollection.find({ userId: this.userId });
});
12 Likes
11

Haha, dont horry my friend, your questions is AWESOME, to be hosnest I spent all this time to reply you because I didn’t have the answers yet.

Change streams are more efficient than the traditional oplog tailing approach for several reasons:

  • With oplog tailing, Meteor must continuously poll and process the entire MongoDB oplog, filtering out irrelevant changes for each collection and user. This results in unnecessary CPU and memory usage, especially as the number of collections, users, or writes increases.
  • Change streams, on the other hand, allow Meteor to subscribe directly to the specific changes relevant to the application, collections, or even queries. This means the server receives only the events it actually needs, reducing the amount of data processed and minimizing resource consumption.
  • Change streams are natively supported by MongoDB and are designed for scalability, providing a more direct and efficient notification mechanism for data changes.
  • In practice, this leads to lower latency for data updates, less server overhead, and better scalability for large applications.

About the performance improvements, my last benchmark using our benchmark repo said:

  • :rocket: 12–35% faster on critical web metrics
  • :floppy_disk: 27% lower memory usage
  • :zap: 14% faster user sessions

Additionally, change streams reduce the risk of missing events during high write loads and simplify the architecture for real-time data propagation, making Meteor more robust and future-proof for modern applications.

I will mantain the old Oplog and Pooling implementation, so if you arent able to use change stream or just dont want to do it, you will flag it and it SHOULD be transparent for you, any change else will be required(i hope).

He did an awesome work for sure but specifically the change stream part i’m visuallizing it in another way, instead add change stream as a new feature using a new API for it, I want to modify the current meteor’s reaticity system that uses oplog, to use change stream as the default option. If you take a look in my implementation, you will see what i’m talking about(feel free for more questions)

The sub cache feature is really impressive but here I’m trying to close only on change stream, I see we bringing it in another front/moment.

Until now is just to enable it in the package.json, nothing else.

7 Likes
12

Hello Italo, thank you for taking the time to thoroughly research this. :heart: Meteor is still evolving and this is an uncharted territory.

I guess this will get even clearer when you guys release a beta we can test out. Thank you for your efforts.

1 Like
13

:rocket: Change Streams for Meteor – Status Update, Performance Benchmarks & Call for Contributors

Hey everyone!

I wanted to give you an update on the progress of the Change Streams integration for Meteor, where we currently stand in terms of performance, challenges, and how you can help us push this forward.

:bar_chart: Performance Benchmark: Change Streams vs Oplog

We’ve run stress tests comparing the new Change Streams-based reactivity engine against the traditional Oplog-based one, using the performance tools available in the Meteor Performance repo. Here’s a summary of the results:

Attachment: you can find the log files here
Command used:

METEOR_NO_DEPRECATION=true METEOR_CHECKOUT_PATH=<my-meteor-path>/meteor  MONGO_URL="mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0" METEOR_REACTIVITY='CHANGE_STREAMS' ./scripts/monitor.sh tasks-3.x reactive-stress.yml

Test scenario:
• Artillery config: 30s duration, 2 users/sec, max 100 users
• Same codebase, one with METEOR_REACTIVITY=‘CHANGE_STREAMS’ and another with the default Oplog

:fire: Performance Summary

Metric Change Streams Oplog Best
FCP Mean 1546ms 2527.9ms :white_check_mark: CS
FCP Median 424.2ms 295.9ms :white_check_mark: Oplog
LCP Mean 1724.7ms 2706.9ms :white_check_mark: CS
LCP Median 441.5ms 713.5ms :white_check_mark: CS
TTFB Mean 194.4ms 196.5ms :white_check_mark: CS
TTFB Median 29.1ms 7.6ms :white_check_mark: Oplog

:brain: Resource Usage

Resource Change Streams Oplog Best
App CPU (avg) 21.43% 31.05% :white_check_mark: CS
App Memory (avg) 164 MB 179 MB :white_check_mark: CS
DB CPU (avg) 0.26% 0.25% ~ tie
DB Memory (avg) 39.1 MB 41.3 MB :white_check_mark: CS

:receipt: Glossary of Performance Metrics

Acronym Full Term Description
FCP First Contentful Paint Time when the first visible element is rendered on the screen.
LCP Largest Contentful Paint Time when the largest visible element is rendered (typically main content).
TTFB Time to First Byte Time from request initiation until the first byte is received from server.
FID First Input Delay Time between a user’s first interaction and the browser’s response.

:checkered_flag: Final Verdict

  • Oplog wins in early response time (TTFB + initial paint).
  • Change Streams is more stable, efficient, and consistent under load.
  • Change Streams ~45% less CPU usage and 8% less memory** on average for the app.
  • Change Streams is better for scaling and high-load scenarios* .

:jigsaw: Current Challenges

While Change Streams is working and reactive, making it fully compatible with Oplog expectations is tough. Many of the core tests fail not due to major bugs but minor semantic differences:

:warning: Examples of Issues:

  • _id formatting: Meteor sometimes expects a string, but Mongo returns ObjectId.
  • Remove events: Change Streams gives the full removed doc; Oplog relies on client cache.
  • Updates: Change Streams delivers pre/post update docs; Meteor expects a diff.

:name_badge: Known Broken Tests:

  • accounts-2fa - Meteor.loginWithPasswordAnd2faCode() fails with invalid code
  • accounts - verify setAdditionalFindUserOnExternalLogin hook can provide user
  • most of password_tests.js file tests
  • most of server_tests.js file
  • livedata - methods with nested stubs
  • livedata server - stopping a handle should preserve its context on callbacks (breaking for oplog and change streams)
  • ejson - stringify
  • mongo-livedata - fuzz test
  • mongo-livedata - observe sorted, limited, big initial set
  • email_tests.js
  • tinytest - observeChanges - single id - basics added
  • tinytest - observeChanges - callback isolation
  • tinytest - observeChanges - single id - initial adds
  • tinytest - observeChanges - unordered - initial adds
  • tinytest - observeChanges - unordered - basics
  • tinytest - observeChanges - unordered - specific fields
  • tinytest - observeChanges - unordered - specific fields + selector on excluded fields
  • tinytest - observeChanges - unordered - specific fields + modify on excluded fields
  • tinytest - observeChanges - unordered - unset parent of observed field
  • tinytest - observeChanges - unordered - enters and exits result set through change
  • observeChanges - tailable

Plus, there’s a ~100ms delay before data reaches the client, causing many tests with tight time expectations to fail.

:test_tube: How to Run the Tests

To help us debug and fix failing tests:

git clone git@github.com:meteor/meteor.git
cd meteor
git checkout feat/change-streams
MONGO_URL="mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0" METEOR_REACTIVITY='CHANGE_STREAMS' ./meteor test-packages

You might need to force METEOR_REACTIVITY=‘CHANGE_STREAMS’ in mongo_connection.js due to hot-reload not always preserving env vars:

const canUseChangeStreams = [
  function () {
    return true; // temporarily override reactivity detection
  }
];

:raised_hands: Call for Contributors

We’re getting really close — performance is solid, and the architecture is scalable. What we need now is help with:

  • Test fixing and standardization
  • Patch contributions to MiniMongo/LiveData to adapt to new event structure
  • Reviewing existing test failures and proposing changes

If you’re interested in real-time Meteor internals , this is a great opportunity to get involved with the core!

4 Likes
14

A few notes and questions:

  1. Did you perform any tests with more than 100 users? The main problem of change streams is that every change stream takes a single connection from the pool, and with hundreds/thousands of unique (i.e., non-multiplexed) subscriptions, it will run out of connections.
    • An ideal test would check what the average latency is between the database insert and UI update with 1x-100x more unique subscriptions than the pool size.
  2. Did you perform any long-running tests? 30 seconds is barely enough for JIT to warm up, and that’s where the memory difference may come from (a different code path selected, different amount of garbage generated along the way).
  3. Did you compare it against cult-of-coders:redis-oplog? (Ideally, both with and without an external publisher like changestream-to-redis.) I know that oplog is still the default Meteor method, but I believe nobody is using it at scale.
4 Likes
15

Did you perform any tests with more than 100 users? no, my machine cant handle so much, max of 200 for 60s, but i have similar results

Did you perform any long-running tests? my macine can hanlde maximum of 60s, but i have similar results

Did you compare it against cult-of-coders:redis-oplog? no, but it’s a good poc, i’ll do in the future, I only compared with native event streams, that is faster, but while i’m working on change stream, i’ll finish it first to take a look on node-event-streams

1 Like