🚀 Meteor 3.5-beta: Change Streams & Performance improvements

:rocket: Meteor 3.5-beta: Performance & Change Streams

Hello everyone! We are excited to announce the first beta of Meteor 3.5. This release is heavily focused on introducing MongoDB Change Streams to the ecosystem.

For a deep dive into the “why” behind Change Streams, check out our detailed discussion here.

:test_tube: Experimenting with 3.5-beta

To start testing the new features, use the following commands:

Create a New App

# Create a new Meteor app using Meteor 3.5-beta

meteor create my-app --release 3.5-beta.6

Note: Improvements in this beta are not enabled by default for new apps. See the Setup section below for instructions.

Update Your App

# Update your existing Meteor app to version 3.5-beta

meteor update --release 3.5-beta.6

:hammer_and_wrench: Change Streams Setup

To enable MongoDB optimizations, follow these steps:

  1. Database Requirement: Ensure you are using a Replica Set or a Sharded Cluster. See the MongoDB documentation for more details.
  2. Configuration: Add the following to your settings.json:
  {  
    "packages": {  
      "mongo": {  
        "reactivity": ["changeStreams", "oplog", "polling"]  
        *// Meteor will try Change Streams first, then oplog,*  
        *// and if that doesn't work, fall back to pooling.*  
      }  
    }  
  }  

Refer to the docs for further technical details.

ChangeStreams feature is transparent for you

After update the meteor version and setup the setting.json to use change streams, Meteor will care the rest for you automagically :sparkles:

Meteor.publish('links', function () {
   // This uses Change Streams ONLY for the LinksCollection 
  // and ONLY for documents matching { my: 'query' }
    return LinksCollection.find({ my: 'query' });
  });

different from oplog that observe ALL changes from a collection even you are using it or not, change streams just take care about the changes you are using/querying.

:sparkles: Highlights

:brain: 40% More Scalability with Change Streams (#13787)

The traditional Oplog driver works by tailing all database changes and performing a “diff” process against client-side documents. As connections scale, this architecture hits a critical bottleneck: processing capacity.

In our tests, we discovered that while Oplog manages memory well under normal loads, it struggles to deliver data to clients fast enough during high traffic. This creates a backlog of pending queries and connections that eventually leads to Out of Memory (OOM) crashes.

Change Streams solve this by handling data on-demand (streaming). Instead of accumulating data locally, it processes it as it flows, allowing the system to maintain stability. Even under extreme stress, the system experiences simple timeouts rather than a fatal process crash.

Preliminary Benchmarks

The following data was captured using an artillery script in a dedicated machine (2 vcpu with 8 GB of RAM) running a standard Meteor app (otel) hosted in galaxy (using premium instance - 1 container of 8 vcpus with 8GB ) with high-frequency add/delete operations:

  • :chart_with_downwards_trend: Oplog: Maxed out at 1200 VUs (10 VUs/s for 120s). Beyond this, the process suffered from persistent timeouts followed by an OOM crash.
  • :chart_with_upwards_trend: Change Streams: Handled up to 1680 VUs (14 VUs/s for 120s). Beyond this, we observed only timeouts, but the process remained stable with no OOM.

Each VU ran 10 connections, one connection each 0.5s doing a insert

VU = Artillery Virtual User

Conclusion: Change Streams offer a 40% increase in connection capacity and significantly better resilience, effectively eliminating OOM errors caused by high data transfer volume.

(Image from montiAPM)

:repeat: Reproducing the benchmark

> git clone git@github.com:meteor/performance.git
> cd otel  
> git checkout otel  
> docker-compose up -d mongo-replica  

# edit your settings.json enabling changeStreams or Oplog  
# then run the meteor app  

> MONGO_OPLOG_URL="mongodb://localhost:27017,localhost:27018,localhost:27019/local?replicaSet=rs0" MONGO_URL="mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0" METEOR_NO_DEPRECATION=true  meteor run --settings ./settings.json --port 8080 --inspect

# in another shell, run the artillery  

> npx artillery run tests/artillery/add-task.yml   

:zap: Faster CI/CD (#14177 and #13787)

We have removed Travis CI from our GitHub pipelines.

  • Before: ~40+ minutes waiting for PR checks.
  • Now: ~10 minutes via GitHub Actions.

:zap: DDP Session Resumption (#14051)

When a client loses its network connection and reconnects within the grace period (default: 15 seconds), Meteor now resumes the existing DDP session instead of creating a brand new one.
What this means in practice:

  • onConnection callbacks are not re-triggered on resume
  • The client keeps its original connection ID
  • No full session re-initialization and data re-fetch — significantly reducing CPU spikes on reconnect (e.g., after load balancer timeouts on platforms like Google Cloud Run)
  • Only ungraceful disconnects (network drops, browser close) are resumable. Intentional disconnects (explicit logout, server kick) are not.

Two new server-side options are available:

  • Meteor.server.options.disconnectGracePeriod (default: 15000ms)
  • Meteor.server.options.maxMessageQueueLength (default: 100)

Big thanks to @vlasky for the comprehensive implementation and test coverage.

:arrows_counterclockwise: DDPRateLimiter Now Supports Async Rule Matchers (#14182)

DDPRateLimiter rule matchers can now be asynchronous functions, enabling use cases like database lookups inside rate limiting rules, something that wasn’t possible before.
As a bonus, the internal logic was refactored to evaluate matching rules only once instead of twice, making rate limit checks slightly faster when your matchers do async work.

DDPRateLimiter.addRule({
  type: 'method',
  name: 'sendMessage',
  async userId(userId) {
    const user = await Meteor.users.findOneAsync(userId);
    return user && user.role !== 'admin';
  }
}, 10, 1000);

TypeScript type definitions and documentation examples have also been updated to reflect the new async-first approach.
Thanks to @9Morello for this contribution.

:shield: Email Warning When Accounts.emailTemplates.from Is Not Set (#14044)

Meteor has historically used Accounts Example no-reply@example.com as a default sender when Accounts.emailTemplates.from is not configured. Since example.com is a reserved domain, most SMTP providers silently reject these emails, making it very hard to debug.

Meteor now logs a clear warning at startup when this default is detected, helping developers catch misconfigured email setups early.

Thanks to @harry97 for tracking down this long-standing pain point.

:green_circle: Node.js 24.14.0 & NPM 11.10.1 (#14176)

Meteor 3.5 ships with Node.js 24.14.0 (LTS) and NPM 11.10.1, bringing all the stability, performance, and security improvements from the Node 24.x line. This has been a long-running effort led by @storyteller, huge thanks for keeping the runtime up to date.

Other PRs and improvments

Other PRs and improvments you can see here

:building_construction: Comming soon

:handshake: Big Thanks to Our Contributors

Community contributions are the backbone of Meteor 3.5.

Your Feedback

Your feedback is crucial in adjusting this beta before the official launch. Community testing is key to ensuring Meteor 3.5 and the Change streams integration is stable and flexible.

You can use this thread to ask questions.

You can also check the existing Change Streams integration forum post for more details and to share your feedback. The post will also serve as a place to add insights on the integration and explain further options and ways it can be used.

17 Likes

Amazing stuff, so glad to see the oplog issues being tackled.

Will it allow you to select which collections/fields should be observed?

Is Capacitor next in line for a major release?

6 Likes

Incredible work, @italojs! Thanks!

Now it’s time for everyone to try this version and give us feedback! :rocket:

@msavin, the next release is 3.4.1, but yes, speaking of major releases, Capacitor should be next in line.

6 Likes

I’m so happy for the progress, guys! Thank you very much for your hard work! :raised_hands:
The migration from Meteor2/Vue2 to Meteor3/Vue3 seem to be done and everything works great! :+1:
Change Streams will be the next thing I will try!

By the way, would Redis Oplog at least theoretically work with Change Streams and optimize this further? Or this efficiency is the practical maximum in real-time MongoDB polling/observing?

Ah, I’ve found this: GitHub - radekmie/changestream-to-redis: Automatically publish MongoDB changes to Redis for Meteor. - this would be the maximum :slight_smile:

4 Likes

[EDITED]
just for collection but not for fields yet

1 Like

awesome, I think once you get that in, the performance gains will be much much larger

can’t come soon enough!!

2 Likes

@zodern maybe this beta release is interesting for you and montiAPM

Great! This is a fantastic push for us to “rip the band-aid off” and upgrade several versions of Meteor!

7 Likes

So what will happen if you have to delete about 500k - 1 million temporary records per day? On Oplog, Meteor crashed. I guess the same will happen on change streams?

2 Likes

Possible, but if we can get per collection/field filters, that should solve it.

I know a lot of apps used to solve this by having a separate MongoDB cluster for those kinds of operations (i.e. storing bulk logs, mass deletes) to filter out the oplog operations, looks like that can be put in the past soon.

1 Like

sorry @msavin I missed a thing in my last comment, today we already listen per collection but not per field (yet not)

it depends, if you arent listening that collection, it wont affect your meteor app ¯\(ツ)/¯
the point is: oplog listen for oplog collection that register ALL operations in the db, change streams just get the changes that happens in the observed collection

1 Like

is there an API to control this

i.e.

// to turn off
const Chickens = new Mongo.Collection("chickens", { changeStreams: false })
// eventually
const Eggs = new Mongo.Collection("eggs", { changeStreams: ["weight", "mamaChickenId"] })

at least being able to turn a collection off could be a good start, looks like it would be easy to pass a flag into the changestream logic

2 Likes

I cannot wait to use aggregation pipelines with change streams :raised_hands:

3 Likes

Currently, reactivity is managed globally via your settings.json . By configuring the reactivity array, you can define which strategies the meteor prioritizes:

// settings.json
"reactivity": ["changeStreams", "oplog", "pooling"]

In most cases, Change Streams (CS) are the optimal choice. They are generally more performant and cost-effective than Oplog or polling (event oplog is in your maximum otimized version after X year of micro optimizations). It is rare to encounter a scenario where you would need to mix strategies (e.g., using pooling for one collection and CS for another).

So, is there an API to control this?

Meteor automatically handles filtering in the background. When you define a publication, the driver restricts the Change Stream to that specific collection and query.

  Meteor.publish('links', function () {
   // This uses Change Streams ONLY for the LinksCollection 
  // and ONLY for documents matching { my: 'query' }
    return LinksCollection.find({ my: 'query' });
  });

Note: Thanks for the question! I’ll update the root forum topic to make this clearer for everyone.

1 Like

Oh that’s slick, so there is collection/field level filtering

Does it start a new ChangeStreams for every query essentially? are there any unique constraints to consider with this change (i.e. maybe having 100s different ChangeStreams across many servers is problematic)

App-design wise, I would probably lean towards adapting to ChangeStreams and ditching oplog tailing for good. Especially if I have a logs collection that can flood the app. It would be nice to be able to control the pooling (polling?) speed, and to have a flag like Meteor.isChangeStreams / Meteor.isPooling to adjust.

Do ChangeStreams occasionally crash out or is this just like a precaution?

Not for each query but for each oberver, when we do a find in a publication, we do not return the result directly, we return an oberverDriver, the observeDriver is the responsable to watch(via polling, oplog or cs) the query and magically send the data to client
So it’s 1 CS per publication(observer) per connection

Following my benchmark, I opened approximately 800 parallel connections, each with its own change stream, on Galaxy. Are you able to share a benchmark from your application? I’m curious to see how it behaves under those conditions.

using change stream you dont need to care about hot collection affecting your meteor app performance.

Today you can do it but I guess it isnt documented
1. Per cursor

collection.find(selector, {
  pollingIntervalMs: 5000,  // default: 10,000ms (10s)
  pollingThrottleMs: 100    // default: 50ms
});
  • pollingIntervalMs — how often the query is re-executed to detect changes
  • pollingThrottleMs — minimum time between re-polls (increasing saves CPU/mongo load, decreasing is not recommended)

2. Environment variables

METEOR_POLLING_INTERVAL_MS=5000
METEOR_POLLING_THROTTLE_MS=100

Apply server-wide defaults when not specified on the cursor.

while I was diving into the code to bring the pollingInterval feaure, i just find a disableOplog flag, following it I can implement that for changeStream as well, covering your request to enalble it per cursor

Meteor.publish("currentRoom", function (roomName) {
  return Rooms.find(
    { name: roomName },
    {
      disableOplog: true,
      pollingIntervalMs: 5000,
      pollingThrottleMs: 5000,
    }
  );
});

disableOplog should be documented via jsdocs but for somereason it isnt in our published docs, i’ll investigate why

1 Like

I don’t have any apps that require such scale, but running into oplog tailing issues is something that has always haunted me and made me doubt Meteor. This is sick, I feel like it kills the last “hacky” part of Meteor, except maybe for the Capacitor/Electron issues but that looks like it will be solved soon too.

Now I’m back to, if you need SPA/real-time, why would you use Next/Supabase/etc… can’t think of a good reason anymore.

One last thing I’m finding missing in Meteor, just for feedback, is a way to stream data to the client. I think it’s especially important now that every AI service streams its results. I know you can do it with WebApp, but would be nice to have a neat solution with Meteor.user(), permissions, etc, easily supported.

5 Likes

This is exciting! I think Change Streams will make a big difference in some performance issues I run into especially when pushing new versions and having all users hit the new servers at the same time. (I hope so, anyway.)

As for Capacitor, I’ll update the thread I started on that soon but long story short I’ve been refining my Cordova backwards-compatibility package and plan to start public beta testing with my users early next month. I’ll probably have a few hundred folks kicking the tires over the beta period and I’ll keep the community posted on how it goes. Would definitely be nice to have a smoother DX here but I’m glad to have it working with cross-compatible hot code pushes.

5 Likes

Excited to see your updates on Capacitor, especially around backward compatibility with current Cordova HCP capability. Let us know how the beta period goes.

On the core side, we will start the CapacitorJS integration very soon, similar to how we recently included Rspack. After Meteor 3.4.1, we will have more time to focus on it and provide the smoother DX many people expect. If by then you have Cordova plugin compatibility ready, it will be great to see how you adapted it, in case we can include improvements in the related core plugin.

In a side project, I achieved HCP with Capgo updater, but it is definitely better to adapt the built-in Meteor capabilities, at least to allow opting in by default.

5 Likes