I cannot wait to use aggregation pipelines with change streams 
3 Likes
Currently, reactivity is managed globally via your settings.json . By configuring the reactivity array, you can define which strategies the meteor prioritizes:
// settings.json
"reactivity": ["changeStreams", "oplog", "pooling"]
In most cases, Change Streams (CS) are the optimal choice. They are generally more performant and cost-effective than Oplog or polling (event oplog is in your maximum otimized version after X year of micro optimizations). It is rare to encounter a scenario where you would need to mix strategies (e.g., using pooling for one collection and CS for another).
So, is there an API to control this?
Meteor automatically handles filtering in the background. When you define a publication, the driver restricts the Change Stream to that specific collection and query.
Meteor.publish('links', function () {
// This uses Change Streams ONLY for the LinksCollection
// and ONLY for documents matching { my: 'query' }
return LinksCollection.find({ my: 'query' });
});
Note: Thanks for the question! I’ll update the root forum topic to make this clearer for everyone.
1 Like
Oh that’s slick, so there is collection/field level filtering
Does it start a new ChangeStreams for every query essentially? are there any unique constraints to consider with this change (i.e. maybe having 100s different ChangeStreams across many servers is problematic)
App-design wise, I would probably lean towards adapting to ChangeStreams and ditching oplog tailing for good. Especially if I have a logs collection that can flood the app. It would be nice to be able to control the pooling (polling?) speed, and to have a flag like Meteor.isChangeStreams / Meteor.isPooling to adjust.
Do ChangeStreams occasionally crash out or is this just like a precaution?
Not for each query but for each oberver, when we do a find in a publication, we do not return the result directly, we return an oberverDriver, the observeDriver is the responsable to watch(via polling, oplog or cs) the query and magically send the data to client
So it’s 1 CS per publication(observer) per connection
Following my benchmark, I opened approximately 800 parallel connections, each with its own change stream, on Galaxy. Are you able to share a benchmark from your application? I’m curious to see how it behaves under those conditions.
using change stream you dont need to care about hot collection affecting your meteor app performance.
Today you can do it but I guess it isnt documented
1. Per cursor
collection.find(selector, {
pollingIntervalMs: 5000, // default: 10,000ms (10s)
pollingThrottleMs: 100 // default: 50ms
});
pollingIntervalMs— how often the query is re-executed to detect changespollingThrottleMs— minimum time between re-polls (increasing saves CPU/mongo load, decreasing is not recommended)
2. Environment variables
METEOR_POLLING_INTERVAL_MS=5000
METEOR_POLLING_THROTTLE_MS=100
Apply server-wide defaults when not specified on the cursor.
while I was diving into the code to bring the pollingInterval feaure, i just find a disableOplog flag, following it I can implement that for changeStream as well, covering your request to enalble it per cursor
Meteor.publish("currentRoom", function (roomName) {
return Rooms.find(
{ name: roomName },
{
disableOplog: true,
pollingIntervalMs: 5000,
pollingThrottleMs: 5000,
}
);
});
disableOplog should be documented via jsdocs but for somereason it isnt in our published docs, i’ll investigate why
1 Like
I don’t have any apps that require such scale, but running into oplog tailing issues is something that has always haunted me and made me doubt Meteor. This is sick, I feel like it kills the last “hacky” part of Meteor, except maybe for the Capacitor/Electron issues but that looks like it will be solved soon too.
Now I’m back to, if you need SPA/real-time, why would you use Next/Supabase/etc… can’t think of a good reason anymore.
One last thing I’m finding missing in Meteor, just for feedback, is a way to stream data to the client. I think it’s especially important now that every AI service streams its results. I know you can do it with WebApp, but would be nice to have a neat solution with Meteor.user(), permissions, etc, easily supported.
5 Likes
This is exciting! I think Change Streams will make a big difference in some performance issues I run into especially when pushing new versions and having all users hit the new servers at the same time. (I hope so, anyway.)
As for Capacitor, I’ll update the thread I started on that soon but long story short I’ve been refining my Cordova backwards-compatibility package and plan to start public beta testing with my users early next month. I’ll probably have a few hundred folks kicking the tires over the beta period and I’ll keep the community posted on how it goes. Would definitely be nice to have a smoother DX here but I’m glad to have it working with cross-compatible hot code pushes.
5 Likes
Excited to see your updates on Capacitor, especially around backward compatibility with current Cordova HCP capability. Let us know how the beta period goes.
On the core side, we will start the CapacitorJS integration very soon, similar to how we recently included Rspack. After Meteor 3.4.1, we will have more time to focus on it and provide the smoother DX many people expect. If by then you have Cordova plugin compatibility ready, it will be great to see how you adapted it, in case we can include improvements in the related core plugin.
In a side project, I achieved HCP with Capgo updater, but it is definitely better to adapt the built-in Meteor capabilities, at least to allow opting in by default.
5 Likes
it’s a thing I have in my mind, togheter of decoupling the reactivity system from the DB to use Node EventEmitters instead. It would make Meteor agnostic, but it’s a huge task and, unfortunately, not the priority at the moment.
2 Likes
I just ended up vibe-coding a Streamer for Meteor and its working pretty neat. Even got Meteor.user() and Meteor.userId() working in it via DDPCommon.MethodInvocation
Also implemented “partial-JSON-parsing” for when using JSON outputs for the LLM, so you can start rendering the object data into your UI right away.
Would anybody want this as a package? Would MDG want this as a core feature?
Code samples:
On the server:
import { Streamer } from "meteor/msavin:streamer";
Streamer.define("generateJSON", async function (more = {}) {
const userId = Meteor.userId()
const userDoc = Meteor.user()
if (!userId) {
this.close()
}
const jsonToStream = JSON.stringify({
title: "Meteor Streaming",
features: ["Pub/Sub", "Methods", "Streaming"],
status: "Awesome",
userId,
userDoc,
...more,
});
// Simulate streaming token by token
for (let i = 0; i < jsonToStream.length; i++) {
await new Promise((r) => setTimeout(r, 50)); // sleep 50ms
console.log("streaming...", { value: jsonToStream[i], done: false });
this.send({ text: jsonToStream[i] });
}
this.close();
});
On the client:
const jsonStream = useMemo(() => Streamer.streamJSON("generateJSON"), []);
// Track the ReactiveVars
const { data, isStreaming } = useTracker(() => ({
data: jsonStream.data.get(),
isStreaming: jsonStream.isStreaming.get(),
}));
5 Likes
I think “pooling” should be “polling” - got an error when testing of Invalid Mongo reactivity method
1 Like
Just had this exact problem!
Error: Invalid Mongo reactivity method in settings: pooling
This worked:
{
"packages": {
"mongo": {
"reactivity": [
"changeStreams",
"oplog",
"polling"
]
}
}
}
I just shipped a major update to the meteor-react-tailwind-prettier-starter — bumping to Meteor 3.5-beta.3 and adding three production-quality real-time examples that showcase exactly what makes Meteor special.
https://todo-sample.meteorapp.com/
(sadly accounts-passwordless does not work on galaxy on --free plan. I’ll have to add a different way to login…)
3 Likes
I’d love to see this package in action. I’m particularly excited about these experiments on new features missing in Meteor. Could you publish it?
It would be great to have the streaming capabilities in Meteor core at some point. Feel free to open a PR there if you like. For the core integration, we may get some feedback from the community or us so we can align on an interface that matches Meteor conventions, test coverage, and so on.
Anyway, publish as community package for now is fine. The core integration can come later, once the approach is tested by the community and we have clear guidance for integrating it into the core. As you prefer.
3 Likes
This starter looks great, @wreiske!
What problem are you having with accounts-passwordless on the free plan? I don’t remember anything that would block it.
3 Likes
Thanks! Claud is good at making things pretty.
For some reason when trying to send the email, it causes an internal server error. I’m assuming emails are blocked to prevent spam.
It might be interesting to setup a fake mail server and when mail gets sent, users can go to inbox.meteorapp.com or something to read the mail. That would prevent abuse of outgoing mail but would still allow testing password less auth and other scenarios that require working email.
Yeah, I was saying on meteor deploy --free it might be nice to have a “mock” email server that you can go check your mail. Even in local development, instead of just console logging out the email, it could use a nice real-time meteor inbox.
I might work on this. This is something supabase and other projects have with Mailpit - email & SMTP testing tool Mailpit - email & SMTP testing tool
1 Like
1 Like
Our ‘Event Loop Delay’ went from 97ms avg ‘(v3.4)’ to 18ms avg ‘(v3.5-beta.4)’ when using Change Streams.
V3.4
V3.5-beta.4
7 Likes
Just pushed an update that will send email to an inbox! BTW… does the --free plan allow testing change streams? https://todo-sample.meteorapp.com
1 Like