I’m also excited about the new features coming to meteor
My team and I have a pretty big service built with meteor (with a 2+ years codebase!), which tracks phones, has a task completion system, and provides real-time stats and maps.
As we started, we used Meteor for everything. At the start, our needs were mostly basic. A panel to see devices on a map in real time and an app that sends regularly it’s position to the server.
The app was made with older versions of react-native, using DDP directly, and the user base at that time was pretty small (like 10 users online at the same time!) so there wasn’t any kind of problem there.
But as we grew, we found problems scaling out what we had. Since we were adding real-time statistics and graphs, it quickly became a memory-intensive task to recalculate everything each time a device moves (which is pretty often).
So we started caching everything with redis. And it was great. We just needed to calculate everything one time and then store it on redis. The only tricky part was to know when to expire the cache.
And then another big problem came to us: We needed to provide integrations to external tools.
An API was ideal, but building one with meteor would just add a lot more complexity to the codebase than we expected, and, also, we weren’t sure how to properly handle and scale an HTTP API inside meteor, so we started playing with Amazon Lambda.
We built the API with Python, since it was (in our opinion) easier to use for APIs than node, and we used the great framework Zappa to enable us to write a Flask (another framework) app on top of lambda.
And things were great. We had a cache for every expensive calculation, and the API was running outside of Meteor, so we could invest all of our time to improve the frontend on Meteor.
But then we faced another big trouble: with the amount of data that we were receiving, calculating graphs inside Meteor methods wasn’t enough.
But we solved it quickly moving everything to Mongo Aggregate queries, which improved the speed of calculations to 100x.
This wasn’t a problem with the tools we were using, but our lack of knowledge about the database engine we were using. (As a side note, after moving to WiredTiger some queries improved a lot, too)
And we continued to make calculations with Meteor Methods without any trouble.
And, as you might expect up to this point: we faced another problem
Oplog updates were too fast for Meteor to process, so the publications took a lot longer to be ready and huges amounts of CPU each time MergeBox detected it needed to update a publication.
For this one, we did not find a holy tool to solve the problem. We just started looking at every publication, making things we did not need to be reactive as non-reactive, and minimizing the amount of data published to the bare minimum.
We also modified the API to write to mongo in batches (
insertMany instead of
insert), and things improved a lot.
And, in the future, we’re planning to use redis-oplog to offload the meteor process of having to calculate every update; but right now we’re not suffering. These optimizations were enought. We just want to use redis-oplog to get rid totally of this problem without it coming back in the future. The only real problem we have is re implementing the redis-oplog’s logic on our API.
We, of course, stumbled across a lot of problems in other areas, but these are not Meteor-related. For example, we rewrote the phone app again with react-native but using the API instead of DDP because the app needed to be offline-first, and making an offline-first app is easier in a non-reactive non-pubsub environment.
If I had to start all over again, I would totally go with Apollo. Since it’s more of a frontend for APIs than a full-blown solution from database to client synchronization, improving the performance and scaling it out will be a lot easier. Also, cache all the things! Caching becomes trivial with GraphQL
But I would totally go with Meteor again. We’ve built the first mock up in just a few weeks, and today making a big change is trivial. Not only because we learnt a lot of Meteor’s internal structure, but because Meteor enables you to run instead of walking.
Understanding how Tracker works, understanding when to use
nonreactive and other methods was crucial for me. All the magic Meteor provides made sense to me after spending some time playing with Tracker’s source code.
These days it’s not as useful as it was, since Meteor is moving to Apollo, which comes with it’s own “reactivity”, but understanding the tools you use is crucial to know how to scale and understand where bugs really are hiding.
I hope you find this answer useful. IMO scaling isn’t really about the tools being “scalable-ready”, but about understanding when and when not to use the right tools for the job, and fully understanding the pros and cons.