MongoDB Change Streams support in Meteor

italojs · June 10, 2025, 7:24pm

[EDITED 2x]

Hey everyone!

I’ve been working on a long-awaited feature: support for MongoDB Change Streams in Meteor!

With this new capability, Meteor can listen to real-time events directly from MongoDB (inserts, updates, deletes, etc.) and use them to power reactivity.

I’ve just opened a PR in our GitHub repo.

Work status

POC
Oplog-compatible API
Stable implementation (all tests passing)
Reviewed by peers
Documentation

Why Change Streams?

TLDR;

Today, Meteor’s reactivity is based on two approaches:

1. Polling

Polling periodically checks for changes every X ms. It’s used for single Mongo instances or as a fallback when oplog is not available.

2. Oplog

With oplog, we listen to a special MongoDB collection that records all database operations. Meteor then filters those operations and forwards the relevant ones to clients via WebSocket.

For more details, @radekmie have an awesome blog post that mention the oplog issue

What’s the problem?

The oplog registers every operation in the database, even those your app doesn’t care about.

That means:

Unnecessary load on the server
More events to process and filter
Higher memory usage over time

So when you call find inside a publication, Meteor ends up watching all DB operations, then filtering and forwarding only what matches your query.

How Change Streams help

Change Streams create a stream of events based on your query, not the entire database.

So now, when you return a find inside a publication:

Meteor listens only to operations related to your query
Less data is transferred
Resource usage on the server is optimized

In short: more targeted reactivity, less noise.

Improvements

In my tests (I’ll share full benchmarks soon), I didn’t observe significant latency improvements

but I did see a huge reduction in memory usage .

How to use it

For now, this should be fully transparent from a DX perspective.

You just need to configure your reactivity options in your settings.json:

{
  "packages": {
    "mongo": {
      "reactivity": ["changeStreams", "pooling", "oplog"]
      // Meteor will try Change Streams first, then polling,
      // and if that doesn't work, fall back to oplog.
    }
  }
}

I’d love your feedback

This is the first stable implementation, and while it’s already functional, I’d really appreciate feedback on the developer experience (DX) and any edge cases you might think of.

Let me know:

What you think of this direction
Any real-world use cases you’d like to see tested
Any concerns or suggestions before this ships

Thanks a lot!

permb · June 10, 2025, 9:04pm

Great work!

I assume you meant to write observeChangesAsync in the example?

msavin · June 10, 2025, 10:39pm

Looks interesting! Does find() take in a query in the const handle example?

I imagine not too hard to whip up a similar feature on the client using Tracker

illustreets · June 11, 2025, 9:54am

Amazing news! We’ve been using our own homebrew method for a while, and while it held well in production, I would be more comfortable using something already baked into Meteor.

One thing I made sure from the start was to prevent opening too many change streams (we have tens of collections) because that can become a performance bottleneck in specific situations. So we open one stream, something like this:

const WATCHABLE_COLLECTION_NAMES = ['CollectionA', 'CollectionB']
const db = this.client.db();
const pipeline = [{
    $match: {
        'ns.coll': { $in: WATCHABLE_COLLECTION_NAMES }
    }
}];
this.changeStream = db.watch(pipeline);
//... then separate logic for filtering and processing the events
// try { const change = await this.changeStream.next(); } catch (err) {}
// ...

The approach makes sense in our scenario, where we have to watch most documents and the number of events is not huge.

How does it work in your example above, would something like this open two streams?

const changeStream = MyCollection.watchChangeStream([
  { $match: { operationType: 'insert' } }
]);
//...
// then later on, maybe in a separate module or package
const changeStream2 = MyCollection.watchChangeStream([
  { $match: { operationType: 'delete' } }
]);

jam · June 11, 2025, 4:01pm

Nice this will be great to have in Meteor!

Some initial thoughts:

I’m not sure how I feel about the changeStreams flag in observeChanges. Might be easy to miss. Also as radekmie mentioned in the PR comments, I think they represent two different things. Maybe something like .find().watchChanges() is more appropriate. What were the other options that you considered?
Why not just use Collection.watch instead of Collection.watchChangeStream? This would align more closely with the nodejs mongo driver.

I hope that the Meteor team will steal the ideas and functionality from jam:pub-sub. I don’t think it gets easier from a DX perspective than Meteor.publish.stream. See here for more info on how it uses change streams: GitHub - jamauro/pub-sub: Publish / subscribe using a Method and/or Change Streams, and cache subscriptions for Meteor apps

italojs · June 11, 2025, 6:02pm

Thank you for your suggestions @jam , I’ll definitely take a look at your package!

To be honest, the .find().watchChanges() method is only included for backward compatibility purposes. I’m not sure if I’ll keep it; it will depend on the feedback from the community here.

As for the current MyCollection.watchChangeStream, I agree — I’ll rename it to MyCollection.watch. Thanks for the suggestion!

italojs · June 11, 2025, 6:08pm

That’s a great setup I really liked your scenario @illustreets!

It makes total sense to avoid opening multiple change streams when you’re dealing with many collections. Thanks for sharing that pattern; it’s definitely valuable.

And yes, in the current implementation, calling MyCollection.watchChangeStream() multiple times like in your example would open separate streams. But I agree with your approach — it’s much more efficient to centralize and filter events on a single stream when possible.

I’ll incorporate this idea into the next push for sure!

italojs · June 11, 2025, 6:10pm

@permb still didnt worked on observeChangesAsync, I was expecting to know the community opnion first

italojs · June 11, 2025, 6:12pm

@msavin i dont think so, do you have any sceanario in mind?

harry97 · June 12, 2025, 1:54pm

Hurr durr

I hope I’m not missing the mark but I’m trying to look at this from the perspective of someone who’s entirely new to Meteor (Which is I’m since I’m constantly learning new things about Meteor everyday!!).

How does this impact the current state of Meteor? Do meteor applications scale better now? Do we get performance improvements? If so, by how much? What about backwards compatibility? What migration paths are we talking about here for old applications to reap the benefits of this new feature?

Also, why wasn’t @jam package integrated into the core? I feel its DX is way better and offers other cool benefits. Mainly subs caching which I feel is a huge missing part of pub-sub in Meteor.

The efforts to improve pub-sub are split into the following categories:

Observing Changes; How Meteor learns about changes (polling, redis-oplog and change streams),
Splitting Responsibility; Does the server handle calculating which documents got changed and which didn’t? Or maybe it’s the client responsibility. (mergebox and publication strategies)
Subs-caching; How often the client re-fetches those changes which packages like meteorhacks:subs-manager and ccorcos:subs-cache took a stab at.
Websockets; The method of communicating such changes (sockjs). I read somewhere that ws is more performant. This also has to do with DX which feathersjs comments on it compared to Meteor.

You @italojs implemented one of @zodern’s packages into the core so what prevented it this time?

After this PR gets merged when you do Meteor.publish it’d utilize change-streams instead of polling? Or I’d have to explicitly enable it by tapping into these weird APIs watchChangeStream or setting true on observeChanges?

In a nutshell, how’d this code be impacted in any shape/form or performance post this update?

import { Meteor } from 'meteor/meteor';
import { TasksCollection } from '/imports/db/TasksCollection';

Meteor.publish('tasks', function publishTasks() {
  return TasksCollection.find({ userId: this.userId });
});

italojs · July 1, 2025, 10:09pm

Haha, dont horry my friend, your questions is AWESOME, to be hosnest I spent all this time to reply you because I didn’t have the answers yet.

Change streams are more efficient than the traditional oplog tailing approach for several reasons:

With oplog tailing, Meteor must continuously poll and process the entire MongoDB oplog, filtering out irrelevant changes for each collection and user. This results in unnecessary CPU and memory usage, especially as the number of collections, users, or writes increases.
Change streams, on the other hand, allow Meteor to subscribe directly to the specific changes relevant to the application, collections, or even queries. This means the server receives only the events it actually needs, reducing the amount of data processed and minimizing resource consumption.
Change streams are natively supported by MongoDB and are designed for scalability, providing a more direct and efficient notification mechanism for data changes.
In practice, this leads to lower latency for data updates, less server overhead, and better scalability for large applications.

About the performance improvements, my last benchmark using our benchmark repo said:

12–35% faster on critical web metrics
27% lower memory usage
14% faster user sessions

Additionally, change streams reduce the risk of missing events during high write loads and simplify the architecture for real-time data propagation, making Meteor more robust and future-proof for modern applications.

I will mantain the old Oplog and Pooling implementation, so if you arent able to use change stream or just dont want to do it, you will flag it and it SHOULD be transparent for you, any change else will be required(i hope).

He did an awesome work for sure but specifically the change stream part i’m visuallizing it in another way, instead add change stream as a new feature using a new API for it, I want to modify the current meteor’s reaticity system that uses oplog, to use change stream as the default option. If you take a look in my implementation, you will see what i’m talking about(feel free for more questions)

The sub cache feature is really impressive but here I’m trying to close only on change stream, I see we bringing it in another front/moment.

Until now is just to enable it in the package.json, nothing else.

harry97 · July 4, 2025, 1:10pm

Hello Italo, thank you for taking the time to thoroughly research this. Meteor is still evolving and this is an uncharted territory.

I guess this will get even clearer when you guys release a beta we can test out. Thank you for your efforts.

italojs · July 22, 2025, 5:49pm

Change Streams for Meteor – Status Update, Performance Benchmarks & Call for Contributors

Hey everyone!

I wanted to give you an update on the progress of the Change Streams integration for Meteor, where we currently stand in terms of performance, challenges, and how you can help us push this forward.

Performance Benchmark: Change Streams vs Oplog

We’ve run stress tests comparing the new Change Streams-based reactivity engine against the traditional Oplog-based one, using the performance tools available in the Meteor Performance repo. Here’s a summary of the results:

Attachment: you can find the log files here
Command used:

METEOR_NO_DEPRECATION=true METEOR_CHECKOUT_PATH=<my-meteor-path>/meteor  MONGO_URL="mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0" METEOR_REACTIVITY='CHANGE_STREAMS' ./scripts/monitor.sh tasks-3.x reactive-stress.yml

Test scenario:
• Artillery config: 30s duration, 2 users/sec, max 100 users
• Same codebase, one with METEOR_REACTIVITY=‘CHANGE_STREAMS’ and another with the default Oplog

Performance Summary

Metric	Change Streams	Oplog	Best
FCP Mean	1546ms	2527.9ms	CS
FCP Median	424.2ms	295.9ms	Oplog
LCP Mean	1724.7ms	2706.9ms	CS
LCP Median	441.5ms	713.5ms	CS
TTFB Mean	194.4ms	196.5ms	CS
TTFB Median	29.1ms	7.6ms	Oplog

Resource Usage

Resource	Change Streams	Oplog	Best
App CPU (avg)	21.43%	31.05%	CS
App Memory (avg)	164 MB	179 MB	CS
DB CPU (avg)	0.26%	0.25%	~ tie
DB Memory (avg)	39.1 MB	41.3 MB	CS

Glossary of Performance Metrics

Acronym	Full Term	Description
FCP	First Contentful Paint	Time when the first visible element is rendered on the screen.
LCP	Largest Contentful Paint	Time when the largest visible element is rendered (typically main content).
TTFB	Time to First Byte	Time from request initiation until the first byte is received from server.
FID	First Input Delay	Time between a user’s first interaction and the browser’s response.

Final Verdict

Oplog wins in early response time (TTFB + initial paint).
Change Streams is more stable, efficient, and consistent under load.
Change Streams ~45% less CPU usage and 8% less memory** on average for the app.
Change Streams is better for scaling and high-load scenarios* .

Current Challenges

While Change Streams is working and reactive, making it fully compatible with Oplog expectations is tough. Many of the core tests fail not due to major bugs but minor semantic differences:

Examples of Issues:

_id formatting: Meteor sometimes expects a string, but Mongo returns ObjectId.
Remove events: Change Streams gives the full removed doc; Oplog relies on client cache.
Updates: Change Streams delivers pre/post update docs; Meteor expects a diff.

Known Broken Tests:

accounts-2fa - Meteor.loginWithPasswordAnd2faCode() fails with invalid code
accounts - verify setAdditionalFindUserOnExternalLogin hook can provide user
most of password_tests.js file tests
most of server_tests.js file
livedata - methods with nested stubs
livedata server - stopping a handle should preserve its context on callbacks (breaking for oplog and change streams)
ejson - stringify
mongo-livedata - fuzz test
mongo-livedata - observe sorted, limited, big initial set
email_tests.js
tinytest - observeChanges - single id - basics added
tinytest - observeChanges - callback isolation
tinytest - observeChanges - single id - initial adds
tinytest - observeChanges - unordered - initial adds
tinytest - observeChanges - unordered - basics
tinytest - observeChanges - unordered - specific fields
tinytest - observeChanges - unordered - specific fields + selector on excluded fields
tinytest - observeChanges - unordered - specific fields + modify on excluded fields
tinytest - observeChanges - unordered - unset parent of observed field
tinytest - observeChanges - unordered - enters and exits result set through change
observeChanges - tailable

Plus, there’s a ~100ms delay before data reaches the client, causing many tests with tight time expectations to fail.

How to Run the Tests

To help us debug and fix failing tests:

git clone git@github.com:meteor/meteor.git
cd meteor
git checkout feat/change-streams
MONGO_URL="mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0" METEOR_REACTIVITY='CHANGE_STREAMS' ./meteor test-packages

You might need to force METEOR_REACTIVITY=‘CHANGE_STREAMS’ in mongo_connection.js due to hot-reload not always preserving env vars:

const canUseChangeStreams = [
  function () {
    return true; // temporarily override reactivity detection
  }
];

Call for Contributors

We’re getting really close — performance is solid, and the architecture is scalable. What we need now is help with:

Test fixing and standardization
Patch contributions to MiniMongo/LiveData to adapt to new event structure
Reviewing existing test failures and proposing changes

If you’re interested in real-time Meteor internals , this is a great opportunity to get involved with the core!

radekmie · July 22, 2025, 6:52pm

A few notes and questions:

Did you perform any tests with more than 100 users? The main problem of change streams is that every change stream takes a single connection from the pool, and with hundreds/thousands of unique (i.e., non-multiplexed) subscriptions, it will run out of connections.
- An ideal test would check what the average latency is between the database insert and UI update with 1x-100x more unique subscriptions than the pool size.
Did you perform any long-running tests? 30 seconds is barely enough for JIT to warm up, and that’s where the memory difference may come from (a different code path selected, different amount of garbage generated along the way).
Did you compare it against cult-of-coders:redis-oplog? (Ideally, both with and without an external publisher like changestream-to-redis.) I know that oplog is still the default Meteor method, but I believe nobody is using it at scale.

italojs · July 22, 2025, 8:32pm

Did you perform any tests with more than 100 users? no, my machine cant handle so much, max of 200 for 60s, but i have similar results

Did you perform any long-running tests? my macine can hanlde maximum of 60s, but i have similar results

Did you compare it against cult-of-coders:redis-oplog? no, but it’s a good poc, i’ll do in the future, I only compared with native event streams, that is faster, but while i’m working on change stream, i’ll finish it first to take a look on node-event-streams

alimgafar · August 14, 2025, 8:28am

When the response emphasizes “my machine,” this is usually when we say let’s get a representative test environment and good test cases. The initial specs are very encouraging, but as @radekmie implies, it needs more robust verification.

That said, I’m really encouraged and excited to see how this will play out.

italojs · December 3, 2025, 12:52pm

Quick checkpoint on this thread: I’ve just pushed a major update to the Change Streams implementation and topic description as well, addressed all the questions raised so far.

So for new readers, you can start following the discussion from here.