I am trying to adjust my app for scale, so I was wondering if there is any recent articles / blog posts that sum up the options you have in 2021 to do that. I know there are some hints in the Guide, and I am aware of things like Redis Oplog and other stuff.
But if you google for Meteor performance optimizations, you get many articles that are pretty outdated and refer to tech like meteor-cluster (which I believe has been abandoned, as it was a Kadira project).
I also found this thread:
which ends in a recommendation for a consultancy agency. Yet I would prefer to get my hands dirty myself
(If anyone has the knowledge to write such an article, Iâd be more than happy to do a review.)
I donât really have an answer for you, but I believe your assumption that meteor-cluster is outdated is not quite true. The package has been recently updated.
@waldgeist this is the thread that has the information you are after:
Also, you do not need Meteor-specific packages to take advantage of nodeâs inbuilt clustering support. Refer to this thread:
I have noticed that the presence of old unmaintained Meteor-specific packages - often nothing more than glorified wrappers around node or NPM packages - causes some prospective users to draw the wrong conclusions about the Meteor ecosystemâs state of health.
A major cause of this issue is that Meteor originally didnât have official NPM support - it was added in Meteor 1.3 which was released in 2016. Thatâs when many of these âwrapperâ packages were abandoned.
The difficulty is that many of these old packages rank highly in search engine results. We should work together to remedy this problem.
For instance, we could make contact with the original authors of the abandoned packages on GitHub/atmosphere and get them to link to contemporary packages, Meteor forum posts or guides.
However, maybe I wasnât clear enough what my key point was: I know thereâs a lot of forum threads out there, all hinting to certain measures. But I could not find any (recent) tutorial / guide that covers this topic in a concise and up-to-date way. Starting with simple things like unblocking methods, and then going further up the complexity ladder towards clustering, load-balancing etc.
I read through a lot of threads already, but this felt like skimming through a lot of ânoiseâ, especially since many articles you find online are pretty outdated, pointing to abandoned packages or referring to hacks that have most likely been superseded by proper solutions by now.
A big part of my job is a performance consultant in Node.js applications (not all of them are Meteor apps) and I can tell, that most often Meteor is not the source of the problems. Start with treating your app as a standard Node.js app - use profiler, measure all the things you can, analyze your monitoring data. From my experience, the vast majority of performance problems is clearly visible, even with basic monitoring.
I also second what @vlasky posted - this forum thread has the best wisdom / reading time ratio you can find for Meteor. Weâve made a short article about Meteor scaling recently, maybe itâll be a good start. Or at least a checklist.
To be honest I never used any of the clustering solutions. For scaling, using Docker helps a lot - adding and removing containers is simple (both manually and automatically). For more direct task offloading the worker_threads module feels better.
I am the dev behind https://github.com/nathanschwarz/meteor-cluster .
It was first published a few month ago, I released @2.1.0 yesterday, with an IPC feature.
So I can assure you itâs quite up to date.
I know youâd prefer a more âcondensedâ format, kind of like an up to date guide . Yet given the difference in the natures of the beasts involved in a high performance environment (Meteor.js, Node, one or more reverse proxies, MongoDB, and so on), each with their own scalability challenges, it would be very difficult to keep everything in one place, and also always up to date. Besides, different Meteor apps require different scalability strategies.
A blog may need SSR, which becomes a Node worker clustering & HTTP cache challenge among other things, while a live IoT dashboard could pose a pub/sub scalability issue, in which case you need to scale MongoDB and Redis. Sometimes one needs all of these.
The threads @vlasky mentions are gold and have excellent references and examples. The advice regarding Cloudflare under the first thread is very on point. Itâs astonishing how much it can increase the perceived speed of your app, even if it does nothing to accelerate DDP.
The fact that performance is both application-dependent and often meteor-independent does not make the need for an authoritative, up-to-date starting point for understanding performance issues in Meteor any less critical.
This is where Tiny can step up and allocate âdocumentation resourcesâ to create a section called âPerformance Issuesâ under the âProductionâ section of the Meteor Guide.
As this thread indicates, there is a lot of home-grown wisdom available in the community, as well as a lot of out-of-date and/or incorrect information. The community needs leadership and resources dedicated to separating the wheat from the chaff and putting it into an authoritative document so we arenât relying on google.
BTW, the Meteor Guide has a section called âMigrating to Meteor 1.7â. Huh?
I agree, although I definitely appreciate the help from all the people who already answered to this thread! To me, itâs not a matter of âone guide can rule them allâ, but rather a curated âentry-pointâ to the most important things you might want to consider for your app. I know that performance tuning is a tricky thing and takes the actual app and its specifics into account. Yet at least you should get an overview about the options you have, albeit this overview might not be (or even able to be) complete.
If you want my advice thereâs 2 core principles to optimizing performance in a Meteor app: monitoring and understanding whatâs going in Meteor APM (formerly Kadira) and use redis-oplog as much as you can. From my experience the most critical bottlenecks often comes from the publication observers and the built-in oplog tailing mechanism.
Just wanted to add my voice to this, I was about to write this exact forum post until I came upon this one.
For me, whatâs missing is a best practices for Meteor, and more specifically best practices for the Pub/Sub system and Meteor Methods. I understand the point that your app is just a node app; however, the data layer is still the unique component that I find leaves me asking the most questions.
I lack clarity on things like:
Whatâs the best way to organize Publications, should I have many, or consolidate them in to one?
When should I use Meteor Methods for loading data instead, what is the best way to do this in React?
When should I implement Redis Oplog?
What are the performance implications of each decision?
My task for today is to dig through all of these forum posts and find answers to my questions. But i think it would be nice to have a performance best practices included in the Meteor Guide.
I also would be up to collaborating on something like this, but I just donât feel like I have the knowledge or experience to be confident in writing the best practices yet.
When reading through this, I read a section that explained that some Mongo DB queries cannot use Oplog, just because they are using operators that are not supported by MiniMongo, which is being used by Oplog behind the scenes. Among these operators are $elemMatch and geo-spatial operators like $nearSphere. I was totally unaware of this limitation, and for our geo-spatial app, this is actually a bummer. Checked back with MontiAPM, and in fact my geo-spatial pubs report back as âno oplogâ. Wish I read this before. Most of the performance-related articles and blog posts I am aware of focus on scaling (only), but not on these basics.
Now I am wondering:
How does the (inefficient) poll and diff mechanism actually work? Can this even be used in a production app that uses geo-spatial queries heavily, or does âpollâ mean that Meteor fetches all documents of a particular collection?
Does Redis-Oplog have the same query limitations as well, so I might workaround my issue using this instead? Does it have other limitations?
I wouldnât expect the answer to all these questions in one guide, but at least some hints that let you dig further. In a guide that it carefully being maintained and updated to reflect newer Meteor versions.
@arunoda gave me permission to use a copy of Kadiraâs Academy for Monti APM. Next month I will be digging into the details of DDP and updating the academy. If anyone wants to help in any way, I would appreciate it. It will be in a public GitHub repository so anyone can view or contribute to it.
. Yet given the difference in the natures of the beasts involved in a high performance environment (Meteor.js, Node, one or more reverse proxies, MongoDB, and so on), each with their own scalability challenges, it would be very difficult to keep everything in one place, and also always up to date. Besides, different Meteor apps require different scalability strategies.