I am trying to adjust my app for scale, so I was wondering if there is any recent articles / blog posts that sum up the options you have in 2021 to do that. I know there are some hints in the Guide, and I am aware of things like Redis Oplog and other stuff.
But if you google for Meteor performance optimizations, you get many articles that are pretty outdated and refer to tech like meteor-cluster (which I believe has been abandoned, as it was a Kadira project).
I also found this thread:
which ends in a recommendation for a consultancy agency. Yet I would prefer to get my hands dirty myself
(If anyone has the knowledge to write such an article, I’d be more than happy to do a review.)
I don’t really have an answer for you, but I believe your assumption that meteor-cluster is outdated is not quite true. The package has been recently updated.
@waldgeist this is the thread that has the information you are after:
Also, you do not need Meteor-specific packages to take advantage of node’s inbuilt clustering support. Refer to this thread:
I have noticed that the presence of old unmaintained Meteor-specific packages - often nothing more than glorified wrappers around node or NPM packages - causes some prospective users to draw the wrong conclusions about the Meteor ecosystem’s state of health.
A major cause of this issue is that Meteor originally didn’t have official NPM support - it was added in Meteor 1.3 which was released in 2016. That’s when many of these ‘wrapper’ packages were abandoned.
The difficulty is that many of these old packages rank highly in search engine results. We should work together to remedy this problem.
For instance, we could make contact with the original authors of the abandoned packages on GitHub/atmosphere and get them to link to contemporary packages, Meteor forum posts or guides.
However, maybe I wasn’t clear enough what my key point was: I know there’s a lot of forum threads out there, all hinting to certain measures. But I could not find any (recent) tutorial / guide that covers this topic in a concise and up-to-date way. Starting with simple things like unblocking methods, and then going further up the complexity ladder towards clustering, load-balancing etc.
I read through a lot of threads already, but this felt like skimming through a lot of “noise”, especially since many articles you find online are pretty outdated, pointing to abandoned packages or referring to hacks that have most likely been superseded by proper solutions by now.
A big part of my job is a performance consultant in Node.js applications (not all of them are Meteor apps) and I can tell, that most often Meteor is not the source of the problems. Start with treating your app as a standard Node.js app - use profiler, measure all the things you can, analyze your monitoring data. From my experience, the vast majority of performance problems is clearly visible, even with basic monitoring.
I also second what @vlasky posted - this forum thread has the best wisdom / reading time ratio you can find for Meteor. We’ve made a short article about Meteor scaling recently, maybe it’ll be a good start. Or at least a checklist.
To be honest I never used any of the clustering solutions. For scaling, using Docker helps a lot - adding and removing containers is simple (both manually and automatically). For more direct task offloading the worker_threads module feels better.
I am the dev behind https://github.com/nathanschwarz/meteor-cluster .
It was first published a few month ago, I released @2.1.0 yesterday, with an IPC feature.
So I can assure you it’s quite up to date.
I know you’d prefer a more “condensed” format, kind of like an up to date guide . Yet given the difference in the natures of the beasts involved in a high performance environment (Meteor.js, Node, one or more reverse proxies, MongoDB, and so on), each with their own scalability challenges, it would be very difficult to keep everything in one place, and also always up to date. Besides, different Meteor apps require different scalability strategies.
A blog may need SSR, which becomes a Node worker clustering & HTTP cache challenge among other things, while a live IoT dashboard could pose a pub/sub scalability issue, in which case you need to scale MongoDB and Redis. Sometimes one needs all of these.
The threads @vlasky mentions are gold and have excellent references and examples. The advice regarding Cloudflare under the first thread is very on point. It’s astonishing how much it can increase the perceived speed of your app, even if it does nothing to accelerate DDP.
The fact that performance is both application-dependent and often meteor-independent does not make the need for an authoritative, up-to-date starting point for understanding performance issues in Meteor any less critical.
This is where Tiny can step up and allocate “documentation resources” to create a section called “Performance Issues” under the “Production” section of the Meteor Guide.
As this thread indicates, there is a lot of home-grown wisdom available in the community, as well as a lot of out-of-date and/or incorrect information. The community needs leadership and resources dedicated to separating the wheat from the chaff and putting it into an authoritative document so we aren’t relying on google.
BTW, the Meteor Guide has a section called “Migrating to Meteor 1.7”. Huh?
I agree, although I definitely appreciate the help from all the people who already answered to this thread! To me, it’s not a matter of “one guide can rule them all”, but rather a curated “entry-point” to the most important things you might want to consider for your app. I know that performance tuning is a tricky thing and takes the actual app and its specifics into account. Yet at least you should get an overview about the options you have, albeit this overview might not be (or even able to be) complete.
If you want my advice there’s 2 core principles to optimizing performance in a Meteor app: monitoring and understanding what’s going in Meteor APM (formerly Kadira) and use redis-oplog as much as you can. From my experience the most critical bottlenecks often comes from the publication observers and the built-in oplog tailing mechanism.
Just wanted to add my voice to this, I was about to write this exact forum post until I came upon this one.
For me, what’s missing is a best practices for Meteor, and more specifically best practices for the Pub/Sub system and Meteor Methods. I understand the point that your app is just a node app; however, the data layer is still the unique component that I find leaves me asking the most questions.
I lack clarity on things like:
What’s the best way to organize Publications, should I have many, or consolidate them in to one?
When should I use Meteor Methods for loading data instead, what is the best way to do this in React?
When should I implement Redis Oplog?
What are the performance implications of each decision?
My task for today is to dig through all of these forum posts and find answers to my questions. But i think it would be nice to have a performance best practices included in the Meteor Guide.
I also would be up to collaborating on something like this, but I just don’t feel like I have the knowledge or experience to be confident in writing the best practices yet.
When reading through this, I read a section that explained that some Mongo DB queries cannot use Oplog, just because they are using operators that are not supported by MiniMongo, which is being used by Oplog behind the scenes. Among these operators are $elemMatch and geo-spatial operators like $nearSphere. I was totally unaware of this limitation, and for our geo-spatial app, this is actually a bummer. Checked back with MontiAPM, and in fact my geo-spatial pubs report back as “no oplog”. Wish I read this before. Most of the performance-related articles and blog posts I am aware of focus on scaling (only), but not on these basics.
Now I am wondering:
How does the (inefficient) poll and diff mechanism actually work? Can this even be used in a production app that uses geo-spatial queries heavily, or does “poll” mean that Meteor fetches all documents of a particular collection?
Does Redis-Oplog have the same query limitations as well, so I might workaround my issue using this instead? Does it have other limitations?
I wouldn’t expect the answer to all these questions in one guide, but at least some hints that let you dig further. In a guide that it carefully being maintained and updated to reflect newer Meteor versions.
@arunoda gave me permission to use a copy of Kadira’s Academy for Monti APM. Next month I will be digging into the details of DDP and updating the academy. If anyone wants to help in any way, I would appreciate it. It will be in a public GitHub repository so anyone can view or contribute to it.
. Yet given the difference in the natures of the beasts involved in a high performance environment (Meteor.js, Node, one or more reverse proxies, MongoDB, and so on), each with their own scalability challenges, it would be very difficult to keep everything in one place, and also always up to date. Besides, different Meteor apps require different scalability strategies.