Looking for solutions to make Meteor cheaper

Hi all,

At present Meteor is a bit expensive to run in production due to mainly two reasons. This is based on what I know, and I am not sure I am being 100% scientific here.

Reason 1
Meteor cannot produce an arm64 build for Graviton (or equivalent) servers in the clouds.
At this link you can see that “g” servers such as m8g.medium are cheaper than m7a.medium (which are not even based on intel processors but on AMD). “G” processors are considered more performant and efficient and 20-40% cheaper.

Reason 2
Meteor no longer has a package or any development to run in cluster mode and make use of all processors in a server and the entire memory.
Presently, if you run Meteor on a 2vCPU server at 1GB of RAM, you only use 1vCPU and half the RAM, because RAM is divided by processors.

The biggest saving can be achieved when running single “G” (arm64) processors. They come in large processing power options, avail of the entire memory paid for and are cheaper.

Questions:
What would it take to produce an arm64 production bundle?
What would it take to embed PM2 in the build script/tools and have it as an option for deployment?

6 Likes

Interesting observations and questions. Any thoughts @fredmaiaarantes @gabsferreira @nachocodoner ?

Yes, you definitely can use pm2 to run multiple instances of Meteor on a single VPS. That way you can utilize all the cores and memory. It work’s really well, but takes a bit of time and research to set up.

Obviously self-hosting and deploying Meteor app is not as smooth as hosting in Galaxy, but it’s not that hard.

I think there is no limit at how sophisticated your build & deployment process is, but I stick to the very crude and basic way:

  1. Run o local bash script to build Meteor app for linux and scp to the server
  2. Run a server bash script to delete prev build, unarchive new build, npm install, restart pm2 process

build-and-deploy.sh example (run on my machine)

cd ../appname
meteor build --architecture=os.linux.x86_64 ../builds/appname
scp ../builds/appname/appname.tar.gz user@serverIP:/server/path/appname
ssh user@serverIP "cd /server/path/apps-server && bash ./replace-appname-prod.sh" &

replace-appname-prod.sh (runs on server)

#! /bin/bash
cd ../appname
rm -rf bundle
tar xvf appname.tar.gz
cd bundle
(cd programs/server && npm install --verbose)
cd ../../apps-server
pm2 restart ecosystem.prod.config.js --only appname

The last script mentioning folder “apps-server”.
In that folder i keep pm2 configs for all the parts. I have main Meteor app, separate Meteor app for admins, and a bunch of express microservices, such as PDF generator, notifications service, etc. On the topic of optimizations: I had very heavy methods for aggregating statistical data from the DB. I’ve moved it out of Meteor apps to the separate services with express endpoints.

Here is an example of pm2 config to run first Meteor app in cluster mode, and second app with one instance only:

ecosystem.prod.config.js

require('dotenv').config({ path: './.env.prod' })

module.exports = {
  apps : [
    {
      name: "appname",
      cwd: "../appname/bundle",
      script: "main.js",
      exec_mode: "cluster",
      instances: 6,
      watch: false,
      log_date_format: "MM.DD HH:mm:ss",
      out_file: '../logs/appname-out.log',
      error_file: '../logs/appname-error.log',
      env: {
        NODE_ENV: "prod",
        PORT: 8080,
        ROOT_URL: "https://myappdomain.com",
        MONGO_URL: ``,
        MONTI_APP_ID_: process.env.MONTI_APP_ID_APPNAME,
        MONTI_APP_SECRET_: process.env.MONTI_APP_SECRET_APPNAME,
        METEOR_SETTINGS: {
          "redisOplog": {
            "redis": {
              "port": process.env.REDIS_PORT,
              "host": process.env.REDIS_HOST
            },
            "retryIntervalMs": 10000, // Retries in 10 seconds to reconnect to redis if the connection failed
            "mutationDefaults": {
                "optimistic": true, // Does not do a sync processing on the diffs. But it works by default with client-side mutations.
                "pushToRedis": true // Pushes to redis the changes by default
            },
            "debug": false,
          }
        }
      },
    },
    {
      name: "admin",
      cwd: "../admin/bundle",
      script: "main.js",
      watch: false,
      log_date_format: "MM.DD HH:mm:ss",
      out_file: '../logs/admin-out.log',
      error_file: '../logs/admin-error.log',
      env: {
        // pretty much the same except for root url and monti creds
      }
    }
  ]
}

For initial start of pm2 to you need to just run:

pm2 start ecosystem.prod.config.js

pm2 can be configured to handle server restarts properly, see: PM2 - Startup Script

Also, you need to make sure to config nginx to cluster mode to work and support load balancing: PM2 - Production Setup with Nginx

And don’t forget to configure pm2 logs, for example:

pm2 install pm2-logrotate
pm2 conf
pm2 set pm2-logrotate:max_size 50M

Redis OPLOG

One of the best things you can do to optimize CPU usage is to use OPLOG tailing. Redis OPLOG works great. GitHub - cult-of-coders/redis-oplog: Redis Oplog implementation to fully replace MongoDB Oplog in Meteor

When setting up Redis OPLOG it’s important to put packages redis-oplog and disable-oplog at the top of “packages” file. Otherwise they will not work properly.

Monti APM

When self-host you loose Galaxy APM, but, luckily there is Monti APM. It’s a paid service, and it’s charging you per instance of your app, so for my 6 instances I pay $5*6=$30 per month. Still worth it.

More Reading

Here is a few links I’ve based my setup upon:

4 Likes

Ok this assumes that you have one server (or more) such as a Linux server in the cloud. This covers one way of Meteor cloud environment setup.

In autoscaling environments, servers are ephemeral, so there are no local folders or you cannot delete previous builds. A machine is being terminated and replaced by a new machine. Everything has to be done in the build script of the new machine. The more I read your solution, it seems that I understand more of what needs to be done for ephemeral server environments.

Maybe this is where the meteor bundle could build for arm64 servers.

Sorry, you’ve mentioned pm2, so I wasn’t thinking about autoscaling servers. You can use pm2 to run Meteor on a virtual server, and you can have multiple processes utilized no problem, but you will need to handle virtual servers management and load-balancing some other way. But the question is: is it possible to have two layers of load balancing?

I’ve been using autoscaling feature in Galaxy, so when I decided to go for self-hosting Meteor, I wanted to make it autoscale. But I couldn’t figure out reasonably easy way to achieve it. So I went for a single server with a room enough to scale. We are in a way lucky with B2B system with pretty linear growth, so I don’t have to account for x100 scaling overnight.

Now we have about 20% CPU peaks ~600MB memory usage per host, so there is room for unexpected load within reason. And our costs went from ~$500/month in Galaxy to $80/month 8CPU/8GB VPS (+$30 for APM).

1 Like

I host my own Kadira APM, there nothing to save.

In my initial post, Reason 2, I explain that most x64 machines in the clouds have at least 2 cores. In the AWS Elastic Beanstalk (the autoscaling environment) a new machine may be added, and I would end up paying 4 processors and 2 x 2 GB RAM when in fact I’d be using 2 processors and 2 x 1GB RAM.
Autoscaling and balancing are one thing. Node multithreading is another thing. I want to use most of the resources of available Linux machines in the cloud, not only half or 1/4 or 1/8 (depending on how many cores a machine has).
Building for os.linux.arm64 yielded no success so far. I am trying various options now.

How do you deploy and serve your builds?

I’m curious what would happen if each of your machines had pm2 running with multiple instances of Meteor. pm2 will work (PM2 - PM2 in ElasticBeanstalk), but I’m not sure about cluster mode.

We run our Meteor app on AWS ECS with the following Linux/X86_64 tasks, depending on the micro-service:

  • 2 vCPU | 8GB
  • 4 vCPU | 8GB
  • 4 vCPU | 16GB

You’re saying on the 2 vCPU we only get 1 vCPU and 4GB of RAM. And on the 4 vCPUs were getting 1 vCPU and 2GB/4GB of RAM? I didn’t realize this. Calling my DevOps guy!

Also, I thought there was an ARM build that was in the works… did that never get released?

@evolross Answer via Perplexity AI

How Node Runs on AWS ECS with 4 vCPU

Node.js Application Behavior on ECS

  • Node.js is single-threaded by default, meaning that a single Node process will only utilize one CPU core, regardless of how many vCPUs are available on the ECS instance[5].
  • On a server with 4 vCPUs (which equates to 4096 CPU units in ECS terminology), a single Node.js process will not automatically use all available CPU resources[4][5].
  • To utilize all 4 vCPUs, you should run your Node.js application in cluster mode, using either the built-in cluster module or a process manager like PM2. This approach forks multiple worker processes (ideally one per vCPU), allowing your application to handle more concurrent requests and make full use of the server’s CPU capacity[4][8].

ECS Task and Container Setup

  • In your ECS task definition, you can specify the CPU and memory resources for your container. For a 4 vCPU server, you would set the CPU parameter to 4096 units (1024 units per vCPU)[2][4].
  • If you run a single Node.js process in a container with 4 vCPUs, most of the CPU capacity will remain unused due to Node’s single-threaded nature[5].
  • The optimal setup is to configure your container to spawn four Node.js worker processes (using cluster mode or PM2) so that each process can be scheduled on a separate vCPU[4][8].

Performance Considerations

  • Over-allocating CPU (e.g., assigning 4 vCPUs to a single Node.js process) does not improve performance and can actually introduce overhead and resource waste[5].
  • A common best practice is to match the number of Node.js worker processes to the number of vCPUs allocated to the ECS task/container[4][8].
  • For high-performance scenarios, running multiple containers (tasks) and leveraging ECS/Fargate’s scaling capabilities is recommended[8].

Example Setup

  • ECS EC2 instance or Fargate task with 4 vCPUs (4096 CPU units).
  • Container launches Node.js in cluster mode with 4 worker processes.
  • Each worker process handles requests independently, maximizing CPU utilization.

Summary Table

ECS Resource Node.js Setup Utilization
4 vCPU (4096 units) 1 Node process Low (1 core used)
4 vCPU (4096 units) 4 Node cluster/PM2 workers High (all cores used)

Key Takeaway:
To fully utilize a 4 vCPU server on AWS ECS, run your Node.js app in cluster mode or with a process manager like PM2 to spawn one worker per vCPU. This ensures optimal CPU usage and better application performance[4][5][8].

Sources
[1] ECS Instance Size for Node.js Applications Instance Size for Node.js Applications - Flightcontrol
[2] Amazon ECS task definition parameters for the Fargate … Amazon ECS task definition parameters for the Fargate launch type - Amazon Elastic Container Service
[3] Exploring Deployment of Node.js Application to AWS ECS … Exploring Deployment of Node.js Application to AWS ECS Fargate with GitHub Actions
[4] What is the optimal way to run a Node API in Docker … https://stackoverflow.com/questions/27585168/what-is-the-optimal-way-to-run-a-node-api-in-docker-on-amazon-ecs
[5] From Lambda to Fargate: How We Optimized Node.js … From Lambda to Fargate: How We Optimized Node.js Performance with the Right Task Specs - DEV Community
[6] How does a Node.js process behave in AWS Fargate? https://stackoverflow.com/questions/63107690/how-does-a-node-js-process-behave-in-aws-fargate
[7] Tutorial: Deploy an application to Amazon ECS Tutorial: Deploy an application to Amazon ECS - Amazon CodeCatalyst
[8] Scaling Node.js microservices on AWS to handle 5M … https://sirinsoftware.com/blog/scaling-node-js-microservices-on-aws-to-handle-5m-requests-per-minute
[9] Comparison between ARM64 and X86_X64 on ECS … https://www.reddit.com/r/aws/comments/131n3aq/comparison_between_arm64_and_x86_x64_on_ecs/

3 Likes

Here you go :+1: This is how I deploy my meteor apps to a VPS. One click deploy, handles the server, and handled multiple apps on one server.

https://jamesloper.com/meteor-deployments-without-mup

3 Likes

This solution seems very cost-effective but not a production grade solution. It is … the opposite of high availability (low availability ?!), and all projects share the network interface and SDD. I think this could work (except guaranteed uptime) for projects such as multi-tenant SaaS where each client runs a core and the number of users is predictable. But here you are facing another cost issue. A (4-core + 8GB) server is usually at the same price of more expensive than 4 x (1 core + 2GB). And horizontal scalability/elasticity is probably not possible.

You can put it behind a load balancer like any other node.js app, you just need to enable “sticky session” support. I believe digitalocean has this exact service available without needing to built it yourself too.

Edit: Almost forgot you can run 2 instances on the same server if it’s dual core! (Different ports)

Response to 1: There is a good chance you can run a Meteor app production bundle on that arm64 CPU running Linux as long as certain conditions are met:

  • You must have the correct arm64 Linux-compiled Node.js version installed
  • Any NPM packages your app uses must be free of native pre-compiled binary code, otherwise they will need to be rebuilt.

Response to 2: You have been misled. The Node.js cluster module is alive and well.

I have previously posted about using cluster in Meteor apps:

3 Likes

@vlasky

Very good insights, thank you. I used the --architecture: 'os.linux.aarch64' but this probably does nothing or not much. I will need to dig deeper into the build tool and see if it does anything.
I compiled NPMs with Docker with the exact env I am deploying to, added them back to the project and pushed with MUP. I just don’t seem to nail down the right way with all the proper configurations. My next step is to build the whole project in Docker.
To generate a bundle and send it to a bare machine, everything seems straight forward for me. But when I have to push to the elastic environment of AWS, I miss visibility over so many things. For instance, if the machine doesn’t start, I cannot pull logs.

On Response 2, the same situation. I learned a lot from your shares, but again … when I have to do it in my environment, nothing works.

First, I had to discover myself that I cannot put the cluster script in Meteor startup. Right now I am not yet convinced that it actually works with Meteor. I first need to cluster.fork() and then start everything else so that nothing runs on the Master (now cluster.isPrimary). Most examples I’ve seen show this setup:

if (cluster.isPrimary) {
  console.log(`Master ${process.pid} is running`);

  // Fork workers.
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork();
  }

  cluster.on('exit', (worker, code, signal) => {
    console.log(`worker ${worker.process.pid} died`);
  });
} else {
  // Worker processes have a http server.
  http.createServer((req, res) => {
    res.writeHead(200);
    res.end(`Hello from worker ${process.pid}\n`);
  }).listen(3000);

  console.log(`Worker ${process.pid} started`);
}

My understanding from the example above: I first fork and once the cluster is forked I can start the webserver per fork. But the webserver in Meteor starts before the startup files, or the entry point main.js (server).
Add to my lack of understanding (or perhaps trust) in your example (in startup) vs these examples on the web, I was not able to find the right configuration to send the NGINX configurations to Elastic Beanstalk.
Part of the problem is that servers are ephemeral and all configurations need to be sent and done in the deployment process as well as when autoscaling and starting new machines.

The thread and code commits you shared are from 7 years ago, and the cluster part seems theoretical. I am curious if you actually deployed Meteor to any kind of servers in cluster mode.
As for the UNIX socks, that seems to be the recommended way to connect behind NGINX (which I never knew). My focus is now on finding the right configuration for Elastic Beanstalk. When you deploy with MUP to a Linux machine, you can add the full proxy configuration. For Elastic Beanstalk … it is complicated :(.
I am learning a lot from your git thread on UNIX ports.

1 Like

Your struggle justifies my love of dedicated Linux servers where you can control and monitor everything and no management layer interferes with you.

This is actual code taken from a Meteor app that uses node cluster . You will see the important detail that the PORT environment variable has to be passed to the fork() call. That’s how each worker Meteor instance knows to listen on a different port to the master/primary Meteor process.

Meteor.startup(() => {
    if (cluster.isMaster) {
        const startWorker = (name, port) => {
            const w = cluster.fork({PORT: port});
            w.process.name = name;
            w.process.port = port;
        }

        cluster.on('exit', (worker, code, signal) => {
            const {name, port, pid} = worker.process;
            console.log(`Worker ${name} - ${pid} died`);
            console.log(`Starting over...`);
            startWorker(name, port);
        });

        if(isProduction)
        {
            const portStart = 3002;
            const totalWorkers = 10;
            _.each(_.range(0, totalWorkers + 1), i => {
                startWorker((i+1).toString(), portStart+i);
            })
        }       
    } else {
        console.log(`============================================`);
        console.log(`Worker ${worker.process.env.name} - ${process.pid} started`)
        console.log(`============================================`);
    }
});
5 Likes

Thanks for sharing an example and all the info about your take on multiple workers for Meteor, out of curiosity, what kind of strategy do you think is the best for zero downtime deployments when you use dedicated linux servers?

3 Likes

We run our meteor apps on caprover (docker swarm) on top of Oracle Cloud arm64 servers quite successfully. The always-free tier allows for quite a lot of compute and enough memory on Ampere (arm64).

The essence of it is to build the node version elsewhere, like github actions, (meteor build --server-only --server https://yourdomain.com) and then use a Dockerfile like this:

# https://github.com/productiveme/meteor-docker
FROM productiveme/meteor
USER app
WORKDIR /built_app
COPY --chown=app:app . .
# Uncomment additional npm steps below if needed
RUN cd /built_app/programs/server \
	&& npm install \
	# && npm rebuild --build-from-source \
	&& true
# RUN cd /built_app/programs/server/npm \
#   && npm install \
#   # && npm rebuild --build-from-source \
#   && true

HEALTHCHECK CMD curl --fail http://localhost:3000/healthz || exit 1
  • The healthz endpoint helps docker swarm to deploy with zero downtime.
  • You will need a NODE_VERSION env variable for the productiveme/meteor image to start correctly

I plan to write a more complete blog post on this

3 Likes

After 2 days of struggles I finally made it to find a solution for Reason 2 for my 2-core server. I will need to test the solution with multiple cores and see if results are consistent or not.

My environment is AWS Elastic Beanstalk, and I deploy with MUP with the beanstalk plugin for MUP. The challenge was to send the configurations to the elastic environment. At least 1.5 days I spent to try to make Meteor work with UNIX sockets. It just won’t do it.
I managed to do every configuration I wanted, but Meteor would just not listen on that port no matter what I did. I don’t know if this has something to do with Express which was recently added. I am curious if anyone can confirm they are running UNIX sockets between Node and NGINX on Meteor 3.
Another challenge was with AI chats. Different models provide different answers, and in general, it took me a while to differentiate between pre Linux 2 configurations and post as EC2 (linux machines) handle configuration updates differently.

The result:

At the time stamp on Memory Usage is when I am hammering the server with multiple methods per second and multiple users.

The exact same test after implementing the cluster, shows half of everything, because APM only sees one process. The real load is 2x, like in the previous test screenshot.

I now know that I use what I am paying for.

@vlasky I didn’t manage to use UNIX sockets. I am pretty sure I do everything right in Meteor (not much to do here anyway) and in NGINX but my Linux skills are almost 0.You mentioned it above in the thread and back in 2020 too … go simple, one dedicated server. Unless you want to have a look together at your convenient time, I will have to abandon this venue. I managed to make the cluster work with http ports, and the NGINX is using a round robin algorithm to send to the ports.

2 Likes

Looks like Meteor support cluster but does note implement it self as you can see here

if (cluster.isWorker) {
        const workerName = cluster.worker.process.env.name || cluster.worker.id;
        unixSocketPath += '.' + workerName + '.sock';
      }

did you try to use node cluster for your main.js server app?
like this

const cluster = require('cluster');
const os = require('os');

// Número de workers (instâncias)
const numWorkers = process.env.NUM_WORKERS || os.cpus().length;

if (cluster.isMaster) {
  // Fork workers
  for (let i = 0; i < numWorkers; i++) {
    const worker = cluster.fork();
  }

  // Listen for worker exit and restart
  cluster.on('exit', (worker, code, signal) => {
    cluster.fork();
  });

  // Graceful shutdown
  process.on('SIGINT', () => {
    for (const id in cluster.workers) {
      cluster.workers[id].kill();
    }
    
    setTimeout(() => {
      process.exit(0);
    }, 5000);
  });

} else {
  // Worker process - start the Meteor app
  // Set unique port for each worker if needed
  const basePort = parseInt(process.env.PORT || 3000);
  const workerPort = basePort + cluster.worker.id - 1;
  
  process.env.PORT = workerPort;
  
  // Start the Meteor application
  require('./main.js');
} 
3 Likes

I think these kinds of code should be a part of meteor core, and we can configure max number of workers Meteor App may use in Meteor settings.

4 Likes