🚀 Meteor 3.5-beta: Change Streams & Performance improvements

it’s a thing I have in my mind, togheter of decoupling the reactivity system from the DB to use Node EventEmitters instead. It would make Meteor agnostic, but it’s a huge task and, unfortunately, not the priority at the moment.

2 Likes

I just ended up vibe-coding a Streamer for Meteor and its working pretty neat. Even got Meteor.user() and Meteor.userId() working in it via DDPCommon.MethodInvocation

Also implemented “partial-JSON-parsing” for when using JSON outputs for the LLM, so you can start rendering the object data into your UI right away.

Streamer demo

Would anybody want this as a package? Would MDG want this as a core feature?


Code samples:

On the server:

import { Streamer } from "meteor/msavin:streamer";

Streamer.define("generateJSON", async function (more = {}) {
	const userId = Meteor.userId()
	const userDoc = Meteor.user()

	if (!userId) {
		this.close()
	}   

	const jsonToStream = JSON.stringify({
		title: "Meteor Streaming",
		features: ["Pub/Sub", "Methods", "Streaming"],
		status: "Awesome",
		userId,
		userDoc,
		...more,
	});

	// Simulate streaming token by token
	for (let i = 0; i < jsonToStream.length; i++) {
		await new Promise((r) => setTimeout(r, 50)); // sleep 50ms
		console.log("streaming...", { value: jsonToStream[i], done: false });
		this.send({ text: jsonToStream[i] });
	}

	this.close();
});

On the client:

const jsonStream = useMemo(() => Streamer.streamJSON("generateJSON"), []);

// Track the ReactiveVars
const { data, isStreaming } = useTracker(() => ({
	data: jsonStream.data.get(),
	isStreaming: jsonStream.isStreaming.get(),
}));
5 Likes

I think “pooling” should be “polling” - got an error when testing of Invalid Mongo reactivity method

1 Like

Just had this exact problem!

Error: Invalid Mongo reactivity method in settings: pooling

This worked:

{
  "packages": {
    "mongo": {
      "reactivity": [
        "changeStreams",
        "oplog",
        "polling"
      ]
    }
  }
}

I just shipped a major update to the meteor-react-tailwind-prettier-starter — bumping to Meteor 3.5-beta.3 and adding three production-quality real-time examples that showcase exactly what makes Meteor special.

https://todo-sample.meteorapp.com/

(sadly accounts-passwordless does not work on galaxy on --free plan. I’ll have to add a different way to login…)

3 Likes

I’d love to see this package in action. I’m particularly excited about these experiments on new features missing in Meteor. Could you publish it?

It would be great to have the streaming capabilities in Meteor core at some point. Feel free to open a PR there if you like. For the core integration, we may get some feedback from the community or us so we can align on an interface that matches Meteor conventions, test coverage, and so on.

Anyway, publish as community package for now is fine. The core integration can come later, once the approach is tested by the community and we have clear guidance for integrating it into the core. As you prefer.

3 Likes

This starter looks great, @wreiske!

What problem are you having with accounts-passwordless on the free plan? I don’t remember anything that would block it.

3 Likes

Thanks! Claud is good at making things pretty.

For some reason when trying to send the email, it causes an internal server error. I’m assuming emails are blocked to prevent spam.

It might be interesting to setup a fake mail server and when mail gets sent, users can go to inbox.meteorapp.com or something to read the mail. That would prevent abuse of outgoing mail but would still allow testing password less auth and other scenarios that require working email.

I think you just need a mail provider and configure a MAIL_URL in your application, @wreiske.

Yeah, I was saying on meteor deploy --free it might be nice to have a “mock” email server that you can go check your mail. Even in local development, instead of just console logging out the email, it could use a nice real-time meteor inbox.

I might work on this. This is something supabase and other projects have with Mailpit - email & SMTP testing tool Mailpit - email & SMTP testing tool

1 Like
1 Like

Our ‘Event Loop Delay’ went from 97ms avg ‘(v3.4)’ to 18ms avg ‘(v3.5-beta.4)’ when using Change Streams.

V3.4

V3.5-beta.4

6 Likes

Just pushed an update that will send email to an inbox! BTW… does the --free plan allow testing change streams? https://todo-sample.meteorapp.com

1 Like

it should support, i’ll double check it today

fixed in the root post

AMAZING! what observability tool are you using to mensure it?
[EDIT] : oh, just realized it’s SkySignal: Modern APM Built for Meteor 3.x

1 Like

agree 100% with nacho comment and add: please, open a forum topic only for it, let discuss better the usage, love it

x-posting this:

I would love to see this in core at some point!

Having it be able to yield would be :100:!

Is it open source? Could we see the package?

1 Like

Better visual indicator of the performance benefits achieved with 3.5-beta.4, the red line is where it was deployed. Change Streams are going to be a scalability gamechanger for Meteor.

3 Likes

Wish me luck… about to migrate a mission critical real-time monitoring app we use to monitor our entire infrastructure. 1.11.1… it hammers CPU and oplog… i’ll let you know how change streams changes it!

This project uses Meteor 1.11.1, which isn't available on this platform. To
work with this app on all supported platforms, use
meteor update --release METEOR@3.4 to pin this app to the newest compatible
release.

Upgrading to 3.5-beta.4!!!

How is the upgrade going? Excited to hear your experience.

1 Like

Hi,

I am currently testing 3.5-beta.4 and I have issues sur ObjectID.

I have collections with ObjectID fields. When I subscribe to a publication with these king of field, I can see the following in Meteor Dev Tools (DDP):

image

So, the application has errors.

With 3.4 or 3.5 / "mongo": { "reactivity": ["oplog", "polling"] }, I can see:

image

and everthing works well.

2 Likes