The BIGASS self-hosted CONTINUOUS INTEGRATION comparison (Jenkins vs. GitLab CI vs. Buildbot vs. Drone vs. Concourse vs. …)

Hi guys,

the goal of this topic is to find a CHEAP solution for a good (CircleCi-like) CONTINUOUS INTEGRATION environment. Maxium budget $ 10 per month…

Background: With my bootstrapped project I hit the limits on CircleCi’s free package (1500 minutes per month) and I’d love to find a cheap alternative…

Right now I am doing about 1-5 commits per day ( 80 - 150 per month).

Let’s do a little research:

  1. Which commercial options are available? What are the limits of the free offer. How much is the price for the paid package?
  2. Which open-source / free / self-hosted options are available?
  3. Summary: What is a really simple way to set up a CHEAP CI for your meteor project?

Goals

  • max 10 $ per month
  • run meteor unit- integrational- and acceptance-tests (chimp) per commit
  • bitbucket support (auto-build per commit)
  • Email-Notifications on result

1) Commercial Options

CircleCi:

TravisCi

  • https://travis-ci.com/plans
  • Free: NOT really a free plan. ONLY 100 free builds, then you have to pay
  • Paid plans start at 69 $ per month
  • does NOT support Bitbucket

Question: Any others that work with meteor and have good tutorials on how to set them up?

2) Self-Hosted Options

DigitalOcean has a good resource on this: https://www.digitalocean.com/community/tutorials/ci-cd-tools-comparison-jenkins-gitlab-ci-buildbot-drone-and-concourse

I played around with Jenkins for 2 hours and have not come to a conclusion. For sure this is much more time-consuming than choosing an hosted plan.

Jenkins

Tutorials:


GitLab CI

forum topics: https://forums.meteor.com/search?q=GitLab%20CI

Before making this REALLY BIGASS: @warehouseman @awatson1978 @serkandurusoy: Is there a quick solution you can recon? Right now it seems like TravisCi free plan might work for me.

3 Likes

Can you reduce

  • number of commits that require a CI build (5 commits a day is a lot of change for the staging/production targets)
  • build time per commit (10 mins sounds like quite a long time and if the app is really not very large, you might be able to make gains here)

Maintaining a self hosted CI is not a very smooth experience if you are a one-man team and it will steal away from your development time.

3 Likes

@serkandurusoy you’re a legend!

JENKINS

I spent another 2 hours trying to get jenkins to work and it is a mess!!! I feel like you can easily spend a full week setting it up (there are full video-tutorials on this https://www.youtube.com/watch?v=68cDNUz7uro). Summary: Jenkins will NOT work for me.

BUILD TIME

My test-runs are actually REALLY long… I have a lot of chimp-based acceptance tests and a run can easily 4 hours (240 minutes). On CircleCi that would mean 6 full builds per month. Hmmmm… not a lot.

Next action

I could move my repository to a paid git account. 7 $ per month on github seams reasonable.
And then use travis 100 builds per month…

I’ll have a read (Integrating with Travis CI?) and let you know about the progress.

OR: does anyone know a bitbucket compatible plattform which has cheaper pricing?

1 Like

Well actually it looks like TravisCi just has a Trial with 100 builds… NOT per month… IN TOTAL… :persevere:

VSTS may be worth a look. Their free tier doesn’t seem to limit how many builds you do, as long as it’s only one-at-a-time.

1 Like

Self hosted gitlab https://about.gitlab.com/pricing/#self-managed might also be an option because it provides you with an integrated environment, including CI.

Self hosting it is also quite easy because there are one-click install options with digital ocean, too.

1 Like

Updates:

VSTS

unfortunatly does NOT support bitbucket account. Only via a hack 1) switching repositories 2) Running builds every x minutes, which then are NOT related to the commit (https://blogs.msdn.microsoft.com/dmx/2016/10/13/integrating-vsts-with-bitbucket/)

GITLAB

Just installed the 1-click on DO. I need to grab the 15 $ DO-Droplet… let’s see how this one goes. I’ll keep you updated

Have you tried Codeship? Maybe it suits your needs on the free plan (100 deployments per month)

2 Likes

@tcastelli Thanks a lot for the tip. The free plan looks great.

Are you using it yourself? If yes: do you remember which tutorial you followed? Do you mind sharing your setup file?

GITLAB updates

Yes, I installed DO 1-click (need 15 $ droplet).

At first glance it looks much easier to use than jenkins, BUT it looks like sync to bitbucket is NOT supported in the community edition. Yes, you can link to bitbucket, BUT then you’ll have to import sources in order to use the CI-feature…

summary: for now I’ll skip gitlab and try out Codeship as suggested by @tcastelli. Gitlab might be an option to go back if Codeship doesn’t work out… I’ll keep you updated

1 Like

For the setup we used “I want to create my own custom commands” and have something like this

# Meteor Installation
cd ..
\curl -sSL https://install.meteor.com | sed -e 's/PREFIX=.*/PREFIX="${HOME}"/g' | sh
#test meteor
meteor --version
cd clone
meteor npm install

Something like this is enough for the setup, then you execute the tests with whatever command you are running (meteor, chimp, etc) and finally, as for deployment you could use something like this(for galaxy)

meteor npm prune --production
METEOR_SESSION_FILE=XXTOKENFILEXX meteor deploy APPNAME --settings XXSETTINGSFILEXX

We create a token and store it on the repo encrypted so it can be referenced on the deployment script and used for automatic deployments.

Hope this helps

1 Like

Just ran into some weird behaviour in Codeship: big installs like npm install -g chimp randomely timeout from build to build.

Somethings npm install -g chimp would simply pass… the other time it will timeout with

--------------------------------------------------------------------------------
This command didn't output anything for 10 minutes, thus we stopped it.
Please make sure your steps regularly print to standard out or standard error.
If the error is on our end please inform us so we can help you to fix this.
--------------------------------------------------------------------------------

@tcastelli any ideas? is this happening to you too?

Not really, not using chimp nowadays, so not sure what has changed to make it faild. Try asking their support, they tend to be very responsive.

1 Like

Have you looked at Bitbucket pipelines?

We use them for unit tests on every commit. The key thing is to keep the total time down using agressive caching of dependencies

Pricing: $10/month for 1000 mins

1 Like

Just came here to say that my experience so far with self-hosted is that it’s not worth the time. We’ve been using circle ci for a while and I think it’s worth every penny.

Even “simple” things like UI headers that alert you to potential build issues with npm, etc. are taken for granted but would be a potential nightmare to debug when your builds are failing to production.

I think it would make much more sense to refine your workflow a bit to fit into circleci’s free plan than to introduce a huge layer of complexity into your pipeline. As some others have suggested, you can get pretty aggressive with the caching to trim down your build times. Or limit builds to a pre-prod branch that you are confident will pass. Just some ideas =)

2 Likes

I just wrote a blog post recently on my experience after I hit CircleCI’s free limit.

2 Likes

Hi guys,

first of all: THANK YOU SO MUCH for your support! I really love this forum - it has been such a great inspiration over the last years.

Concerning BUILD-TIME: The main reason for my 4 hour build-time are my chimp-based acceptance tests which are just running ultra-long. If I would start from scratch I’d do it a bit differently, use react to test components via SSR, etc… but at the current state there is NOT really an easy way to reduce build. But also my integrational-tests run for 80 minutes…

CURRENT STATE:

Codeship

I spent some time on this one. The support is really good. My main problem is that I am hitting timeouts here…

Codeship Basic will stop after 4 hours and single calls after 1 hour. It’s a pitty, but it will NOT work for me… this is way I went back to Gitlab…

Gitlab

Yesterday I continued with the self-hosted GitLab option as suggested by @serkandurusoy. I started from scratch with DigitalOceans 1-click install, am using https (via my own domain) and so far things look pretty interesting. It feels pretty awesome to have a tool in my hand, where I have unlimited build-time and ALL options in my own hand. Email-notifications work out of the box, which I find awesome.

I’ll try to get this one to work and if I do I’ll post a mini-tutorial here.

Right now I am hitting a wall - does someone have an idea how I could solve it?

The problem: Calling meteor list --allow-superuser gives a warning about --allow-superuser, then hangs and hence the process is being killed after 1 hour…

The error message is

/bin/bash: line 89:  2363 Killed                  meteor list --allow-superuser
ERROR: Job failed: execution took longer than 1h0m0s seconds

I am using this docker image https://hub.docker.com/r/blurri/meteor-node/~/dockerfile/

This is my current Gitlab script

# see https://hub.docker.com/r/blurri/meteor-node/~/dockerfile/
image: blurri/meteor-node

variables:
  METEOR_ALLOW_SUPERUSER: "true"
  METEOR_NO_RELEASE_CHECK: "true"

before_script:
  - echo "======================================"
  - echo "==== START auto full script v0.1 ====="
  - echo "======================================"

cache:
  paths:
    - "node_modules"
    - "~/.npm"
    - "~/.meteor"
    - "~/meteor"
    - ".yarn"

stages:
  - test

run_all_tests:
  stage: test
  script:
    - node --version
    - meteor --version
    - pwd
    - ls -al
    - sudo sh ./continuous-integration/upgrade_chrome_version.sh
    - google-chrome --version
    - cd src
    - pwd
    - curl https://install.meteor.com?release=1.4.4.1 | /bin/sh 2>&1 | cat  # NOTE: ALWAYS sync to the version we have on
    - meteor --version --allow-superuser
    - meteor list --allow-superuser  # HANGS RIGHT HERE… TIMEOUT AFTER 1 hour
# ...

THE LOGS LOOK LIKE:

$ node --version
v6.11.3
$ meteor --version

Even with METEOR_ALLOW_SUPERUSER or --allow-superuser, permissions in your app
directory will be incorrect if you ever attempt to perform any Meteor tasks as
a normal user. If you need to fix your permissions, run the following command
from the root of your project:

  sudo chown -Rh <username> .meteor/local

Meteor 1.5.2
$ pwd

...

$ google-chrome --version
Google Chrome 69.0.3497.92 
$ cd src
$ pwd
/builds/thebarty/theproject/src
$ curl https://install.meteor.com?release=1.4.4.1 | /bin/sh 2>&1 | cat
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  7786    0  7786    0     0  14155      0 --:--:-- --:--:-- --:--:-- 14156
Removing your existing Meteor installation.
Downloading Meteor distribution

Meteor 1.4.4.1 has been installed in your home directory (~/.meteor).
Writing a launcher script to /usr/local/bin/meteor for your convenience.

To get started fast:

  $ meteor create ~/my_cool_app
  $ cd ~/my_cool_app
  $ meteor

Or see the docs at:

  docs.meteor.com

$ meteor --version --allow-superuser

Even with METEOR_ALLOW_SUPERUSER or --allow-superuser, permissions in your app
directory will be incorrect if you ever attempt to perform any Meteor tasks as
a normal user. If you need to fix your permissions, run the following command
from the root of your project:

  sudo chown -Rh <username> .meteor/local

Meteor 1.4.4.1
$ meteor list --allow-superuser

Even with METEOR_ALLOW_SUPERUSER or --allow-superuser, permissions in your app
directory will be incorrect if you ever attempt to perform any Meteor tasks as
a normal user. If you need to fix your permissions, run the following command
from the root of your project:

  sudo chown -Rh <username> .meteor/local

/bin/bash: line 89:  2363 Killed                  meteor list --allow-superuser
Pulling docker image gitlab/gitlab-runner-helper:x86_64-35e8515d ...
ERROR: Job failed: execution took longer than 1h0m0s seconds

@arturs @rohanray @idmontie @macrozone you’ve been there before. Can you help out and post your current working Gitlab configuration?

@idmontie: is this your current Gitlab setup working? https://gitlab.com/memosa/meteor-gitlab-ci? Any tips?

Gitlab

have not found a solution, but it might simply have to do with the speed of the docker-image and RAM allocated. Seems like the meteor list command really does take longer than 1 hour to execute in the runner…

… right now I am hosting this on 3 GB RAM DO-Droplet for 15 $ per month.

I’ll try to add swapping on both the droplet and the docker image

I feel your pain what’s why I wrote my own … it builds then redeploys our meteor server side + android apk triggered by a git push since its using a webhook ( https://github.com/adnanh/webhook ) to listen to github publish actions … nothing on its own too mind bending just a lot of shell and a sprinkling of golang … using docker in docker (which is just a few config files) so every server we have (in addition to meteor) is inside its own container … been running in prod for 6 months as well as a dozen dev domains … at some point I’d like to carve it out of our repo and either open source it or spin it into a paid service … was tons of work yet since we are HIPAA some existing tooling would not fit our reqs

essentially I have a container dedicated to devops actions where all of above happens … on rebuild it brings down then launches fresh docker images

as an aside I’m using https://github.com/tomav/docker-mailserver as a postfix email server which sends out as well as receives SMTP traffic - then wrote a custom postfix content filter in golang to parse incoming email then persist body as needed into alt messaging drains

nice side benefit of rolling your own is I also use this to cut a virgin VPS box into a fully functioning dev toy domain using a one-liner which sets up all of above

2 Likes

Hi guys,

I need your help!

I’m stuck on Gitlab CE and can NOT get Chimp (chimpy) to run…for some reason chimp has problems getting chrome started. Please have a look here: https://github.com/TheBrainFamily/chimpy/issues/131.

Does anyone have an idea what else to try?