Amazing stuff, so glad to see the oplog issues being tackled.
Will it allow you to select which collections/fields should be observed?
Is Capacitor next in line for a major release?
Incredible work, @italojs! Thanks!
Now it’s time for everyone to try this version and give us feedback! 
@msavin, the next release is 3.4.1, but yes, speaking of major releases, Capacitor should be next in line.
I’m so happy for the progress, guys! Thank you very much for your hard work! 
The migration from Meteor2/Vue2 to Meteor3/Vue3 seem to be done and everything works great! 
Change Streams will be the next thing I will try!
By the way, would Redis Oplog at least theoretically work with Change Streams and optimize this further? Or this efficiency is the practical maximum in real-time MongoDB polling/observing?
Ah, I’ve found this: GitHub - radekmie/changestream-to-redis: Automatically publish MongoDB changes to Redis for Meteor. - this would be the maximum 
[EDITED]
just for collection but not for fields yet
awesome, I think once you get that in, the performance gains will be much much larger
can’t come soon enough!!
Great! This is a fantastic push for us to “rip the band-aid off” and upgrade several versions of Meteor!
So what will happen if you have to delete about 500k - 1 million temporary records per day? On Oplog, Meteor crashed. I guess the same will happen on change streams?
Possible, but if we can get per collection/field filters, that should solve it.
I know a lot of apps used to solve this by having a separate MongoDB cluster for those kinds of operations (i.e. storing bulk logs, mass deletes) to filter out the oplog operations, looks like that can be put in the past soon.
sorry @msavin I missed a thing in my last comment, today we already listen per collection but not per field (yet not)
it depends, if you arent listening that collection, it wont affect your meteor app ¯\(ツ)/¯
the point is: oplog listen for oplog collection that register ALL operations in the db, change streams just get the changes that happens in the observed collection
is there an API to control this
i.e.
// to turn off
const Chickens = new Mongo.Collection("chickens", { changeStreams: false })
// eventually
const Eggs = new Mongo.Collection("eggs", { changeStreams: ["weight", "mamaChickenId"] })
at least being able to turn a collection off could be a good start, looks like it would be easy to pass a flag into the changestream logic
I cannot wait to use aggregation pipelines with change streams
Currently, reactivity is managed globally via your settings.json . By configuring the reactivity array, you can define which strategies the meteor prioritizes:
// settings.json
"reactivity": ["changeStreams", "oplog", "pooling"]
In most cases, Change Streams (CS) are the optimal choice. They are generally more performant and cost-effective than Oplog or polling (event oplog is in your maximum otimized version after X year of micro optimizations). It is rare to encounter a scenario where you would need to mix strategies (e.g., using pooling for one collection and CS for another).
So, is there an API to control this?
Meteor automatically handles filtering in the background. When you define a publication, the driver restricts the Change Stream to that specific collection and query.
Meteor.publish('links', function () {
// This uses Change Streams ONLY for the LinksCollection
// and ONLY for documents matching { my: 'query' }
return LinksCollection.find({ my: 'query' });
});
Note: Thanks for the question! I’ll update the root forum topic to make this clearer for everyone.
Oh that’s slick, so there is collection/field level filtering
Does it start a new ChangeStreams for every query essentially? are there any unique constraints to consider with this change (i.e. maybe having 100s different ChangeStreams across many servers is problematic)
App-design wise, I would probably lean towards adapting to ChangeStreams and ditching oplog tailing for good. Especially if I have a logs collection that can flood the app. It would be nice to be able to control the pooling (polling?) speed, and to have a flag like Meteor.isChangeStreams / Meteor.isPooling to adjust.
Do ChangeStreams occasionally crash out or is this just like a precaution?
Not for each query but for each oberver, when we do a find in a publication, we do not return the result directly, we return an oberverDriver, the observeDriver is the responsable to watch(via polling, oplog or cs) the query and magically send the data to client
So it’s 1 CS per publication(observer) per connection
Following my benchmark, I opened approximately 800 parallel connections, each with its own change stream, on Galaxy. Are you able to share a benchmark from your application? I’m curious to see how it behaves under those conditions.
using change stream you dont need to care about hot collection affecting your meteor app performance.
Today you can do it but I guess it isnt documented
1. Per cursor
collection.find(selector, {
pollingIntervalMs: 5000, // default: 10,000ms (10s)
pollingThrottleMs: 100 // default: 50ms
});
pollingIntervalMs— how often the query is re-executed to detect changespollingThrottleMs— minimum time between re-polls (increasing saves CPU/mongo load, decreasing is not recommended)
2. Environment variables
METEOR_POLLING_INTERVAL_MS=5000
METEOR_POLLING_THROTTLE_MS=100
Apply server-wide defaults when not specified on the cursor.
while I was diving into the code to bring the pollingInterval feaure, i just find a disableOplog flag, following it I can implement that for changeStream as well, covering your request to enalble it per cursor
Meteor.publish("currentRoom", function (roomName) {
return Rooms.find(
{ name: roomName },
{
disableOplog: true,
pollingIntervalMs: 5000,
pollingThrottleMs: 5000,
}
);
});
disableOplog should be documented via jsdocs but for somereason it isnt in our published docs, i’ll investigate why
I don’t have any apps that require such scale, but running into oplog tailing issues is something that has always haunted me and made me doubt Meteor. This is sick, I feel like it kills the last “hacky” part of Meteor, except maybe for the Capacitor/Electron issues but that looks like it will be solved soon too.
Now I’m back to, if you need SPA/real-time, why would you use Next/Supabase/etc… can’t think of a good reason anymore.
One last thing I’m finding missing in Meteor, just for feedback, is a way to stream data to the client. I think it’s especially important now that every AI service streams its results. I know you can do it with WebApp, but would be nice to have a neat solution with Meteor.user(), permissions, etc, easily supported.
This is exciting! I think Change Streams will make a big difference in some performance issues I run into especially when pushing new versions and having all users hit the new servers at the same time. (I hope so, anyway.)
As for Capacitor, I’ll update the thread I started on that soon but long story short I’ve been refining my Cordova backwards-compatibility package and plan to start public beta testing with my users early next month. I’ll probably have a few hundred folks kicking the tires over the beta period and I’ll keep the community posted on how it goes. Would definitely be nice to have a smoother DX here but I’m glad to have it working with cross-compatible hot code pushes.
Excited to see your updates on Capacitor, especially around backward compatibility with current Cordova HCP capability. Let us know how the beta period goes.
On the core side, we will start the CapacitorJS integration very soon, similar to how we recently included Rspack. After Meteor 3.4.1, we will have more time to focus on it and provide the smoother DX many people expect. If by then you have Cordova plugin compatibility ready, it will be great to see how you adapted it, in case we can include improvements in the related core plugin.
In a side project, I achieved HCP with Capgo updater, but it is definitely better to adapt the built-in Meteor capabilities, at least to allow opting in by default.
it’s a thing I have in my mind, togheter of decoupling the reactivity system from the DB to use Node EventEmitters instead. It would make Meteor agnostic, but it’s a huge task and, unfortunately, not the priority at the moment.