To stay meteorjs or move to nodejs

1

Hola Meteor experts,

I have a full fledged app built on meteor hosted on Galaxy and Atlas. We are to change our data architecture to scale up to process say 120Mn transactions in a month to begin with…which is a happy problem to solve. Which also means quite some rewriting/re-referencing to mongocollection in the Meteor code.

At this junction I hear these as reasons from a few pro meteor and many anti-meteor experts to get away from Meteorjs:

  1. Meteor cannot listen selectively…At 120mn transactions a month, you might face performance with Meteor if you don’t need it to be reactive…

  2. Meteor still hasn’t mastered client side caching…So dynamic UI/UX changes based on customer preferences will again affect app loading time.

  3. Galaxy could turn very expensive @120Mn transactions per month.

  4. Upgrading from Meteor 1.8 to 1.11 is going to be a challenge and you anyways need to rewrite some of your existing code.

Do you folks see the above as a challenge? Am I better off changing to Nodejs, Angular, Ionic and Graphql on say Gcloud?

Kindly advise!

Jay

2 Likes
2

This is how I like to look at it. With Meteor you have access to all the stuff you would when building your app without Meteor, plus (optionally) any and all the stuff that makes Meteor amazing, so why limit your ability to use all the fantastic things that Meteor offers? When all is said an done, Meteor is just build tooling with a bunch of handy packages that help you to create NodeJS apps in a highly productive manor. The packages and other pieces you fit together for your app are your choice. Switching from Meteor to vanilla node would just be trading all of the awesome, for lots tooling setup and boiler plate code, just so you could arrive back at another, albeit slightly differently structured and packaged, NodeJS app.

16 Likes
3

@copleykj You hit the nail correctly! Today, my stack is mostly based on Meteor, Apollo, Prisma2, Postgres, RabbitMQ…etc…I could easily integrate anything NodeJS from NPM just like every other node app. I still use the awesome Meteor account package(MongoDB only) and keep every other data in Postgres using Prisma2. In my resolvers, I just do ctx.prisma.products…

4 Likes
4

As @copleykj and @martineboh said Meteor is Node.js, and ultimately you decide what parts of it you want to use. Meteor comes with tons of useful stuff, therefore moving away from it doesn’t seem like a good idea – at least not for the reasons you mentioned.

If your app is arriving at hundreds of millions of transactions per month, then you will have to consider very carefully whether you can still leverage Meteor’s reactivity, and to what degree and how exactly. If you decide that you do need reactivity, your best bet is still Meteor, because realistically there is no way that a small team could magically come up with a purely Node.js-based pub/sub reactivity solution that outperforms what was developed into Meteor in several years of work by dozens of experts.

Plus, if you’re expecting hundreds of millions of transactions per month, then it means that your business is skyrocketing, so you should be prepared to see that volume on a per day basis quite soon. Meaning: your architecture needs to be a) zero downtime and b) infinitely scalable.

Meteor is a client-server framework to build modern webapps. It’s not necessarily true that all of your transactions will have to be routed through your very webapp. If I were you I would consider to introduce at least one auxiliary system that takes to heat, so that all the heavy load is diverted away from your Meteor app.

Realistically your clients will not want to constantly drill down into the data of the hundreds of millions of transactions: they will want to see aggregated data most of the time. So you will have to cope with the challenge of the large amount of transactions when they occur and provide a view to the aggregated data (per customer) in a reactive way: those are two different stories.

I was involved in the development of a digital advertising solution, an SSP (Supply Side Platform) for RTB (Real Time Bidding) in the capacity as Head of Development and Lead Architect. We’ve had 4 billion (! not million) incoming requests per day (! not per month) and one hundred billion (! not million) outgoing requests per day, and with the proper architecture and stack this was perfectly doable, 24/7, zero downtime, infinitely scalable. Of course it wasn’t Meteor / Node.js stack, but the principles are the same.

Re: Galaxy – I don’t think you can (or should) do all of it via Galaxy. As they say, if all you have is a hammer, everything looks like a nail, and you should avoid that fallacy. For the volume you anticipate you will need an architecture far beyond a single Meteor app. I think you do need a Meteor app, that’s quite good, but you will probably need more items in your puzzle.

5 Likes
5

As stated above, Meteor is built on Node. And the other components like Angular, Ionic, and Graphql don’t necessarily help with the scalability issues (GraphQL maybe, but only in certain circumstances).

What you really want to do (imho), is add the JsonRoutes package, so you can build out REST endpoints to handle high-traffic operations.

Once minimongo is decoupled from DDP and the default settings (which apparently aren’t suitable for your use case), it works perfectly fine for client side caching - but you have to optimize it and write cache invalidation rules specific to your app.

1 Like
6

Its strange how people try to re-invent the wheel and end up using a dozen technologies trying to replace meteor, the end result is spaghetti code, lots of bugs, which leads to exhaustion and most likely abandonment.

4 Likes
7
  1. Meteor cannot listen selectively…At 120mn transactions a month, you might face performance with Meteor if you don’t need it to be reactive…

Meteor does listen selectively, if you don’t have any active publications querying some data, Meteor won’t respond to it. Though you’re correct in thinking that reactivity is going to be a major challenge at that scale.
There are some fantastic tools to help (redis-oplog, grapher, etc) reduce the pain, but ultimately you will need to switch to fetching some data non-reactively through methods or plain HTTP calls.

  1. Meteor still hasn’t mastered client side caching…So dynamic UI/UX changes based on customer preferences will again affect app loading time.

I’m a bit confused by this one, Minimongo caches all subscribed data, so it doesn’t need a network request to re-fetch.
If you’re talking about caching between page visits using localstorage, you’re right that this doesn’t come built-in. You will need to add that yourself, jsut as you would have to do with a non-Meteor app.
There are a few threads in the forums here where people have managed to build PWAs with localstorage.

  1. Galaxy could turn very expensive @120Mn transactions per month.

Yep.

  1. Upgrading from Meteor 1.8 to 1.11 is going to be a challenge and you anyways need to rewrite some of your existing code.

Curious what issues you’ve had upgrading? I didn’t have to rewrite anything for an upgrade of a 50k line app (though I did each update about a month after release of new major versions, so it was probably easier). I’m guessing Mongo version issues (requires separate upgrade for each major version) and package dependencies?

Do you folks see the above as a challenge? Am I better off changing to Nodejs, Angular, Ionic and Graphql on say Gcloud?

Your situation does sound like a challenge, a great challenge to have!
It sounds like you will need to do a lot of work regardless of staying with Meteor or rolling your own solution, so if I were you my decision would come down to team happiness.

My gut feeling is that it would take longer to move off Meteor and replace all the parts of it you use with other libs or home grown solutions.

Personally, I started a new job where they don’t use Meteor a month ago and I miss it already. That’s partly missing features of the build tool (Now I need to worry about which polyfills are loaded, and don’t have differential bundling), part the data model (we frequently end up with inconsistent client state), and partly that I already know how to do most things in Meteor and I spend a lot of time looking up all the options and deciding how to proceed

10 Likes
8

@coagmano You have summarised it all for me. I am fully convinced and I will NOT change Meteor…Thanks a ton each one of you.

This has been an epic response on the thread from all you experts and I can’t express how relieved I am with your reassurance…Thanks a ton…

5 Likes
9

Note, that scaling is not only an application level issue. You can also scale your whole infrastructure (horizontally) using clusters of apps and dbs instead of single-app performance (vertically).

If you have 120Mn transaction then how big is the chance that these come from the same AWS region if your customer base is global? A global customer base is a good use case to scale horizontally so why not analyze where your customers are located at and split your service into these regions. Then break down the requests per region and there you will get a realistic performance demand of requests you can use as a foundation for further optimizations.

4 Likes
10

Personally, I started a new job where they don’t use Meteor a month ago and I miss it already. That’s partly missing features of the build tool (Now I need to worry about which polyfills are loaded, and don’t have differential bundling), part the data model (we frequently end up with inconsistent client state), and partly that I already know how to do most things in Meteor and I spend a lot of time looking up all the options and deciding how to proceed

Hopefully you will get a chance for a proof-of-concept to show how to solve these issues in less time with “an existing tool you know from the past” :wink:

3 Likes
11

Hopefully! There’s not much appetite for a major refactor, but some new work is being added as microservices, which might give me an ‘in’ so show how good Meteor can be.
More likely, they’ll just be annoyed that I’m not matching the rest of the app :laughing:

12

We already have geographically distributed the traffic. The 120 MN is potentially coming from 1 country as we are working with the government…Happy problem to have indeed…

3 Likes
13

If you’re dealing with a consistent 120M transactions, it may be worth looking at your Meteor app as a front-end and offload that work to a separate job server with a queue in front of it. That would help to keep your Galaxy bill low and allow you to adjust infrastructure to accommodate varying workloads on the job server. Would also deliver a better customer experience because the web app isn’t bogged down processing transactions, just serving up pages and simple reads/writes from the DB.

1 Like
14

If you want another option, some apps with a large number of users use mup-aws-beanstalk since it is cheaper and, at least in the past, was faster while keeping features like load balancing, sticky sessions, and auto scalling.

2 Likes
15

We’ve switched long time ago from Galaxy to AWS and this is like Premier League vs 3rd league. What you want is more flexibility in designing your infrastructure that it not only fits your needs but is also cheaper than what you run right now.

None of the other cloud provider has the number of different products that AWS offers.

Eg., we’re running our Meteor apps (we have dedicated apps for Frontend and Backend to scale independently) in Fargate. Scales automatically and if you go with AWS recommendation engine to buy what you’re going to need in the next year on resources anyway you can safe a lot of money.

Galaxy doesn’t offer you such a service. I’m also not expecting them to offer this as like I said it’s Premier league vs 3rd league (from number of employees, capital to invest etc.).

So I don’t want to make this a Galaxy hate post but like I wrote, if you run app(s) at this level then go and use what the big companies in the world are using.

1 Like
16

Oh this is very useful information, thank you. I wasn’t even aware of Fargate’s existence.

1 Like
17

I can’t agree with the Meteor has specific performance issue than the other platform.
If your transaction grows, scale up/out is the mandatory option for all platforms,
and Meteor is good at this point of view.

For example, specific functions that are required a lot of traffic, you could consist it as a methods/call than a pub/sub reactivity to make the performance expected (single request/response),
then you could scale up the server as many as you want.
In this case, you have to consider database performance could afford your scaled up servers, and I believe that this is all same with the other platform as well.

If you need a reactivity functionality, then Meteor is best bet still as @peterfkruger said. I think no platform catchs up Meteor for reactivity yet.

Like this, there are a lot of hurdles for scale up and raise up the performance,
but I believe that is same for all platform,
and Meteor has good architecture to make the each server as a independent model to be scaled up.

So I think the point is.
Not the which platform.
Understanding the platform we are using exacly.

2 Likes