Hello, I’m opening the new discussion thread to collect all information about Meteor building duration problem on one place. Building the Meteor app inside the docker container takes multiple longer time like on host system. In my case, the building time on host takes about 3-6minutes and building time on docker container started by gitlab is 45minutes+, sometimes more than hour
Building I’m running on my own dedicated server, which is dedicated only for this building. Server has 256GB RAM and 8core Xeon (not latest, but previous generation). Host system is Ubuntu 22.04, docker system is my own container based also on Ubuntu 22.04. During the building, the docker stats is displaying about 250% cpu usage and memory only few percent of 256GB.
What I investigate:
checked in documentation, If I have a correct information, Docker and Gitlab runner is running without cpu/memory limitations in default configuration. For a try, I set cpus=“8” in GitLab runner and I’m able to see 8 building processes in htop on host computer after start of building.
by the recommendations, I tried to build on another containers, I tried meteor/meteor-base, also I tried to prepare another containers based on another linux distributions, but the problem is still the same.
I tried to build another of my projects, situation was the same, also I tired to build only “simple-todos” with React and also on this tiny project was building inside docker significant slower.
I tried also build on another host with docker (still Ubuntu) and the problem was still the same.
And I tried also build the app manually in Ubuntu docker container (without GitLab runner), the building duration was also similar long.
Only what I didn’t tried yet is change host OS to another linux distribution or change another technology instead of docker and I not yet tested building of projects on another JS frameworks.
Probably it’s something wrong with Docker configuration or something (not with Meteor), but I don’t have and idea what else I can do or try to solve it. I’m very appreciate any answer and help.
I don’t have an answer but it might be able to help if you will post the last few lines of the console logs to know where the building was taking its time
There may be something in the OS, these times are above normal.
Some questions:
Does this server have an SSD or a normal HD?
What version of Docker are you using?
Can you share which step of the Dockerfile is taking the longest?
it might be able to help if you will post the last few lines of the console logs to know where the building was taking its time
I will send complete log
There may be something in the OS, these times are above normal.
The building time is long, I’m able to accept it, if it is normal. But what I’m not understand is, that If I will build the app outside the docker directly on the host machine, the building is multiple times faster. I tried the multiple docker containers, I’m sending most simple container, on which I’m testing the building now:
FROM amazonlinux:latest
# The (rough) equivalent of the build-essential meta-package for yum is: yum install make glibc-devel gcc patch
# https://serverfault.com/questions/204893/aws-ec2-and-build-essential
RUN yum -y install curl tar gzip python3 make glibc-devel gcc patch g++ && yum -y clean all && rm -rf /var/cache
ENV METEOR_ALLOW_SUPERUSER 1
RUN curl https://install.meteor.com/ | sh
If I don’t find the reason, I’m thinking to run some benchmarks to compare performace of CPU, memory and filesystem on host and inside container, to know, what is making this problem. My assumption is, that problem can be with slow filesystem operations. Do you have some tips for some simple tools? I need some tool, which gets some numeric (points) results to be able to compare performance.
I’m not saying it is the problem. I’m saying how you can get the timestamp when building so you zero-in on what step is taking a long time. You are trying to solve a problem without knowing what the real problem is. So step 1 is to identify the problem first.
@rjdavid@philippeoliveira@truedon sorry for my later answer, I was little bit testing, and I found, that If I will use the METEOR_DISABLE_OPTIMISTIC_CACHING: 1, the build is still much slower, like outside Docker, but 3x faster. Without this option, the building takes about 21minutes, with using this option, the building takes about 7minutes.
I also compared it during the building outside docker. If the option was enabled or disabled, in the building process outside docker (in host machine) was not any significant time difference. The building time was very similar (about 3minutes).
@truedon I don’t know, where I can find the docker config I’m using default Docker config.
FROM amazonlinux:latest
# The (rough) equivalent of the build-essential meta-package for yum is: yum install make glibc-devel gcc patch
# https://serverfault.com/questions/204893/aws-ec2-and-build-essential
RUN yum -y install curl tar gzip python3 make glibc-devel gcc patch g++ && yum -y clean all && rm -rf /var/cache
ENV METEOR_ALLOW_SUPERUSER 1
RUN curl https://install.meteor.com/ | sh
CI pipeline:
build:
stage: build
variables:
GIT_STRATEGY: clone
GIT_DEPTH: 1000
# https://docs.meteor.com/environment-variables.html
#METEOR_PROFILE: 100 # Logs Meteor building operations longer than 1000ms
METEOR_DISABLE_OPTIMISTIC_CACHING: 1 #can help improve memory usage during meteor build
image: klaucode/meteor-amazonlinux
artifacts:
paths:
- $CI_PROJECT_NAME-$CI_COMMIT_SHORT_SHA.tar.gz
only:
- master
- tags
except:
- schedules
script:
# Remove local
- if [ -d .meteor/local ]; then rm -rf .meteor/local; fi
# Modify instance file
- sed -i "s/__BUILD_NUMBER__/$BUILD_NUMBER-$(date +%Y%m%d_%H%M%S)/g" both/instanceName.ts
- cat both/instanceName.ts
# Install meteor
- meteor npm install --production
- meteor npm install -D tailwindcss #must be installed manually, because an error occured
#- meteor npx kendo-ui-license activate
#- meteor npx browserslist@latest --update-db
- meteor --allow-superuser build --architecture os.linux.x86_64 --server-only ../new_package && mv ../new_package/*.tar.gz $TAR_FILENAME
# Remove local
- if [ -d .meteor/local ]; then rm -rf .meteor/local; fi
- ls
Docker stats during the building:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
26292c5***** runner-********** 209.32% 1.898GiB / 251.4GiB 0.75% 270MB / 6.52MB 3.74MB / 424MB 14
The build duration is all down to cpu power bro.
You can try building on a local machine on your lan a gaming rig or a new mac will probably smoke any virtual instance you are using.
@truedon bro, thank you and I’m really appreciate your help, I agree with you.
Before I buyed the server, I tried to multiple servers and multiple configurations in Meteor build. I tried AMD Epic, multiple Intel CPUs with combinations of motherboards (and on each of them, I was building the same app and measure the time). It was confirmed to me that building a nodejs app does not optimally use the processor’s performance of all cores and it is more advantageous to have a higher frequency than more CPU cores.
…based on real tests results I decided to buy this concrete HW configuration with (Xeon W-2245) (before 1,5 years).
But what I didn’t expected, that the building inside Docker container will be so slow. MULTIPLE times slower like directly on host computer. …also I tried build on the mac with M1, situation is same.
Welcome bro just trying to help! It’s true about the frequency that’s why a overclocked gaming rig can do so well for compilations. I’m surprised to hear that about the new Mac also, because build speed for other stuff has been hell of a fast on it.
Docker is really slow and resource heavy, have you tried vagrant?
Another point here is you do not need to build inside of docker.
You just can have a mounted directory, build on the host and read from that path. I think doing any compilation inside of docker, or vagrant or any cli pipeline in general is doomed and kind of asinine to be totally honest - unless the build is key to your release which if it’s internal software that you host, is definitely not. Because when you deploy on your servers you won’t need to worry about build issues and if they do occur you can fix them easily enough as you control the servers anyway.
we also build meteor apps in gitlab-ci (self-hosted, with runners on kubernetes), the biggest one is 25min (but its not newest meteor).
We do first a meteor build in one job and afterward a docker build that dockerizes the bundle, that takes another 4 min.
meteor build is slow and cpu heavy and memory hungry (because of mininfying and bundling). Of course you can give up continuous delivery and integration and just build locally on your fancy gaming pc, but do you want to go back to that?
There are now faster bundlers and minifizers, but i doubt that meteor will change the bundling anytime soon.
You can try to increase memory (and cpu) available on the runner. is it self-hosted gitlab or gitlab.com ?
@bladerunner2020 finally someone confirmed my experience. Thanks a lot. Try to build once outside docker directly on host system and it will takes, Im guessing 8 minutes.
Of course you can give up continuous delivery and integration and just build locally on your fancy gaming pc, but do you want to go back to that?
You don’t need to give up CI you just build locally and mount the drive on your LAN. The alternative is significantly slower and you’re paying Jeff Bezos for another space flight… why would you want to do that?
If you pay employees salaries you need this to execute as fast and as cheaply as possible, that’s why all my clients have CI in house on their LANs and use gaming machines no one pays AWS for this kind of stuff unless they working for a bank.
MULTIPLE times 