I don’t really have an answer for you, but I believe your assumption that meteor-cluster is outdated is not quite true. The package has been recently updated.
That’s good to know! I only found the meteorhacks and arunoda versions. They still show up first when you google Meteor cluster.
This is a jobs queue
@waldgeist this is the thread that has the information you are after:
Also, you do not need Meteor-specific packages to take advantage of node’s inbuilt clustering support. Refer to this thread:
I have noticed that the presence of old unmaintained Meteor-specific packages - often nothing more than glorified wrappers around node or NPM packages - causes some prospective users to draw the wrong conclusions about the Meteor ecosystem’s state of health.
A major cause of this issue is that Meteor originally didn’t have official NPM support - it was added in Meteor 1.3 which was released in 2016. That’s when many of these ‘wrapper’ packages were abandoned.
The difficulty is that many of these old packages rank highly in search engine results. We should work together to remedy this problem.
For instance, we could make contact with the original authors of the abandoned packages on GitHub/atmosphere and get them to link to contemporary packages, Meteor forum posts or guides.
What do you think?
6 Likes
Thanks @all for your thread suggestions.
However, maybe I wasn’t clear enough what my key point was: I know there’s a lot of forum threads out there, all hinting to certain measures. But I could not find any (recent) tutorial / guide that covers this topic in a concise and up-to-date way. Starting with simple things like unblocking methods, and then going further up the complexity ladder towards clustering, load-balancing etc.
I read through a lot of threads already, but this felt like skimming through a lot of “noise”, especially since many articles you find online are pretty outdated, pointing to abandoned packages or referring to hacks that have most likely been superseded by proper solutions by now.
1 Like
That’s a very good idea!
1 Like
A big part of my job is a performance consultant in Node.js applications (not all of them are Meteor apps) and I can tell, that most often Meteor is not the source of the problems. Start with treating your app as a standard Node.js app - use profiler, measure all the things you can, analyze your monitoring data. From my experience, the vast majority of performance problems is clearly visible, even with basic monitoring.
I also second what @vlasky posted - this forum thread has the best wisdom / reading time ratio you can find for Meteor. We’ve made a short article about Meteor scaling recently, maybe it’ll be a good start. Or at least a checklist.
To be honest I never used any of the clustering solutions. For scaling, using Docker helps a lot - adding and removing containers is simple (both manually and automatically). For more direct task offloading the worker_threads module feels better.
6 Likes
I am the dev behind https://github.com/nathanschwarz/meteor-cluster .
It was first published a few month ago, I released @2.1.0 yesterday, with an IPC feature.
So I can assure you it’s quite up to date.
3 Likes
I know you’d prefer a more “condensed” format, kind of like an up to date guide 
. Yet given the difference in the natures of the beasts involved in a high performance environment (Meteor.js, Node, one or more reverse proxies, MongoDB, and so on), each with their own scalability challenges, it would be very difficult to keep everything in one place, and also always up to date. Besides, different Meteor apps require different scalability strategies.
A blog may need SSR, which becomes a Node worker clustering & HTTP cache challenge among other things, while a live IoT dashboard could pose a pub/sub scalability issue, in which case you need to scale MongoDB and Redis. Sometimes one needs all of these.
The threads @vlasky mentions are gold and have excellent references and examples. The advice regarding Cloudflare under the first thread is very on point. It’s astonishing how much it can increase the perceived speed of your app, even if it does nothing to accelerate DDP.
1 Like
This package does not offer the same functionality as the original meteor-cluster.
My bad, you are correct, I definitely conflated those two packages based on the name
The fact that performance is both application-dependent and often meteor-independent does not make the need for an authoritative, up-to-date starting point for understanding performance issues in Meteor any less critical.
This is where Tiny can step up and allocate “documentation resources” to create a section called “Performance Issues” under the “Production” section of the Meteor Guide.
As this thread indicates, there is a lot of home-grown wisdom available in the community, as well as a lot of out-of-date and/or incorrect information. The community needs leadership and resources dedicated to separating the wheat from the chaff and putting it into an authoritative document so we aren’t relying on google.
BTW, the Meteor Guide has a section called “Migrating to Meteor 1.7”. Huh?
6 Likes
I agree, although I definitely appreciate the help from all the people who already answered to this thread! To me, it’s not a matter of “one guide can rule them all”, but rather a curated “entry-point” to the most important things you might want to consider for your app. I know that performance tuning is a tricky thing and takes the actual app and its specifics into account. Yet at least you should get an overview about the options you have, albeit this overview might not be (or even able to be) complete.
There was a bunch of very useful articles from Arunoda Susiripala but I’m not sure if they’re still available… The only one I could get my hand on is this one http://www.discovermeteor.com/blog/improving-the-performance-of-your-meteor-app/. It’s dating from 2014 but imo it’s still very relevant.
If you want my advice there’s 2 core principles to optimizing performance in a Meteor app: monitoring and understanding what’s going in Meteor APM (formerly Kadira) and use redis-oplog as much as you can. From my experience the most critical bottlenecks often comes from the publication observers and the built-in oplog tailing mechanism.
1 Like
Just wanted to add my voice to this, I was about to write this exact forum post until I came upon this one.
For me, what’s missing is a best practices for Meteor, and more specifically best practices for the Pub/Sub system and Meteor Methods. I understand the point that your app is just a node app; however, the data layer is still the unique component that I find leaves me asking the most questions.
I lack clarity on things like:
What’s the best way to organize Publications, should I have many, or consolidate them in to one?
When should I use Meteor Methods for loading data instead, what is the best way to do this in React?
When should I implement Redis Oplog?
What are the performance implications of each decision?
My task for today is to dig through all of these forum posts and find answers to my questions. But i think it would be nice to have a performance best practices included in the Meteor Guide.
I also would be up to collaborating on something like this, but I just don’t feel like I have the knowledge or experience to be confident in writing the best practices yet.
2 Likes
To add on my original post and why I still think it would be very beneficial to have such a general Meteor performance guide:
I just stumbled upon the old Kadira performance guide created by Arunoda, which you can still find via the Wayback Machine:
https://web.archive.org/web/20170518115618mp_/https://old.kadira.io/academy/meteor-performance-101
When reading through this, I read a section that explained that some Mongo DB queries cannot use Oplog, just because they are using operators that are not supported by MiniMongo, which is being used by Oplog behind the scenes. Among these operators are $elemMatch and geo-spatial operators like $nearSphere. I was totally unaware of this limitation, and for our geo-spatial app, this is actually a bummer. Checked back with MontiAPM, and in fact my geo-spatial pubs report back as “no oplog”. Wish I read this before. Most of the performance-related articles and blog posts I am aware of focus on scaling (only), but not on these basics.
Now I am wondering:
- How does the (inefficient) poll and diff mechanism actually work? Can this even be used in a production app that uses geo-spatial queries heavily, or does “poll” mean that Meteor fetches all documents of a particular collection?
- Does Redis-Oplog have the same query limitations as well, so I might workaround my issue using this instead? Does it have other limitations?
I wouldn’t expect the answer to all these questions in one guide, but at least some hints that let you dig further. In a guide that it carefully being maintained and updated to reflect newer Meteor versions.
@arunoda gave me permission to use a copy of Kadira’s Academy for Monti APM. Next month I will be digging into the details of DDP and updating the academy. If anyone wants to help in any way, I would appreciate it. It will be in a public GitHub repository so anyone can view or contribute to it.
9 Likes
Yeay, that’s amazing! More than happy to help, if I can contribute something.
1 Like
Tom,
we’re having a very database heavy app as well. We plan to use the $nearSphere in the future as well but for now we just paint locations on a world map using leaflet package.
I mentioned to you in another thread before, separate frontend and backend in separate apps first of all. That allows independent scaling which isn’t only very cost effective but let’s you react to heavy CPU or database processes that are unpredictable (meaning, user triggered).
For regular jobs use Lambda and monitor its performance so that you can increase CPU/memory when you need more performance. This can scale up to handle a lot of things.
Use MongoDb on Atlas, not only do they offer the tools but they also have the best knowledge to scale your MongoDb. It’s worth it.
Lastly, limit what data you send to the Frontend. By separating your apps, your backend or Lambda’s will do all the heavy database lifting together with MongoDb. Run aggregations whenever possible, so that most database work happens in database and not in your app (never a good idea). Then send only the needed results of your heavy database work to the end user, nothing less but nothing more.
We don’t use Redis Oplog, we have only a few pub/sub left and almost all our database queries are optimized. That results in performance improvements by several hundred percent and the app feels completely different to the end user, way more snappy.
Lastly, @radekmie as a performance consultant for Node.js apps already gave the best advices. Especially:
To be honest I never used any of the clustering solutions. For scaling, using Docker helps a lot - adding and removing containers is simple (both manually and automatically).
Check out AWS Fargate. Both our frontend and backend apps run as Fargate on AWS and we auto-scale them there. Cheap and works like a charm when performance is needed. Start with the smallest available configuration but ensure you have enough memory as you otherwise will see that you app just crashes with no clear indication what is wrong (it ran out of memory).
Hope this helps.
2 Likes