The BIGASS self-hosted CONTINUOUS INTEGRATION comparison (Jenkins vs. GitLab CI vs. Buildbot vs. Drone vs. Concourse vs. …)

Self hosted gitlab https://about.gitlab.com/pricing/#self-managed might also be an option because it provides you with an integrated environment, including CI.

Self hosting it is also quite easy because there are one-click install options with digital ocean, too.

1 Like

Updates:

VSTS

unfortunatly does NOT support bitbucket account. Only via a hack 1) switching repositories 2) Running builds every x minutes, which then are NOT related to the commit (https://blogs.msdn.microsoft.com/dmx/2016/10/13/integrating-vsts-with-bitbucket/)

GITLAB

Just installed the 1-click on DO. I need to grab the 15 $ DO-Droplet… let’s see how this one goes. I’ll keep you updated

Have you tried Codeship? Maybe it suits your needs on the free plan (100 deployments per month)

2 Likes

@tcastelli Thanks a lot for the tip. The free plan looks great.

Are you using it yourself? If yes: do you remember which tutorial you followed? Do you mind sharing your setup file?

GITLAB updates

Yes, I installed DO 1-click (need 15 $ droplet).

At first glance it looks much easier to use than jenkins, BUT it looks like sync to bitbucket is NOT supported in the community edition. Yes, you can link to bitbucket, BUT then you’ll have to import sources in order to use the CI-feature…

summary: for now I’ll skip gitlab and try out Codeship as suggested by @tcastelli. Gitlab might be an option to go back if Codeship doesn’t work out… I’ll keep you updated

1 Like

For the setup we used “I want to create my own custom commands” and have something like this

# Meteor Installation
cd ..
\curl -sSL https://install.meteor.com | sed -e 's/PREFIX=.*/PREFIX="${HOME}"/g' | sh
#test meteor
meteor --version
cd clone
meteor npm install

Something like this is enough for the setup, then you execute the tests with whatever command you are running (meteor, chimp, etc) and finally, as for deployment you could use something like this(for galaxy)

meteor npm prune --production
METEOR_SESSION_FILE=XXTOKENFILEXX meteor deploy APPNAME --settings XXSETTINGSFILEXX

We create a token and store it on the repo encrypted so it can be referenced on the deployment script and used for automatic deployments.

Hope this helps

1 Like

Just ran into some weird behaviour in Codeship: big installs like npm install -g chimp randomely timeout from build to build.

Somethings npm install -g chimp would simply pass… the other time it will timeout with

--------------------------------------------------------------------------------
This command didn't output anything for 10 minutes, thus we stopped it.
Please make sure your steps regularly print to standard out or standard error.
If the error is on our end please inform us so we can help you to fix this.
--------------------------------------------------------------------------------

@tcastelli any ideas? is this happening to you too?

Not really, not using chimp nowadays, so not sure what has changed to make it faild. Try asking their support, they tend to be very responsive.

1 Like

Have you looked at Bitbucket pipelines?

We use them for unit tests on every commit. The key thing is to keep the total time down using agressive caching of dependencies

Pricing: $10/month for 1000 mins

1 Like

Just came here to say that my experience so far with self-hosted is that it’s not worth the time. We’ve been using circle ci for a while and I think it’s worth every penny.

Even “simple” things like UI headers that alert you to potential build issues with npm, etc. are taken for granted but would be a potential nightmare to debug when your builds are failing to production.

I think it would make much more sense to refine your workflow a bit to fit into circleci’s free plan than to introduce a huge layer of complexity into your pipeline. As some others have suggested, you can get pretty aggressive with the caching to trim down your build times. Or limit builds to a pre-prod branch that you are confident will pass. Just some ideas =)

2 Likes

I just wrote a blog post recently on my experience after I hit CircleCI’s free limit.

2 Likes

Hi guys,

first of all: THANK YOU SO MUCH for your support! I really love this forum - it has been such a great inspiration over the last years.

Concerning BUILD-TIME: The main reason for my 4 hour build-time are my chimp-based acceptance tests which are just running ultra-long. If I would start from scratch I’d do it a bit differently, use react to test components via SSR, etc… but at the current state there is NOT really an easy way to reduce build. But also my integrational-tests run for 80 minutes…

CURRENT STATE:

Codeship

I spent some time on this one. The support is really good. My main problem is that I am hitting timeouts here…

Codeship Basic will stop after 4 hours and single calls after 1 hour. It’s a pitty, but it will NOT work for me… this is way I went back to Gitlab…

Gitlab

Yesterday I continued with the self-hosted GitLab option as suggested by @serkandurusoy. I started from scratch with DigitalOceans 1-click install, am using https (via my own domain) and so far things look pretty interesting. It feels pretty awesome to have a tool in my hand, where I have unlimited build-time and ALL options in my own hand. Email-notifications work out of the box, which I find awesome.

I’ll try to get this one to work and if I do I’ll post a mini-tutorial here.

Right now I am hitting a wall - does someone have an idea how I could solve it?

The problem: Calling meteor list --allow-superuser gives a warning about --allow-superuser, then hangs and hence the process is being killed after 1 hour…

The error message is

/bin/bash: line 89:  2363 Killed                  meteor list --allow-superuser
ERROR: Job failed: execution took longer than 1h0m0s seconds

I am using this docker image https://hub.docker.com/r/blurri/meteor-node/~/dockerfile/

This is my current Gitlab script

# see https://hub.docker.com/r/blurri/meteor-node/~/dockerfile/
image: blurri/meteor-node

variables:
  METEOR_ALLOW_SUPERUSER: "true"
  METEOR_NO_RELEASE_CHECK: "true"

before_script:
  - echo "======================================"
  - echo "==== START auto full script v0.1 ====="
  - echo "======================================"

cache:
  paths:
    - "node_modules"
    - "~/.npm"
    - "~/.meteor"
    - "~/meteor"
    - ".yarn"

stages:
  - test

run_all_tests:
  stage: test
  script:
    - node --version
    - meteor --version
    - pwd
    - ls -al
    - sudo sh ./continuous-integration/upgrade_chrome_version.sh
    - google-chrome --version
    - cd src
    - pwd
    - curl https://install.meteor.com?release=1.4.4.1 | /bin/sh 2>&1 | cat  # NOTE: ALWAYS sync to the version we have on
    - meteor --version --allow-superuser
    - meteor list --allow-superuser  # HANGS RIGHT HERE… TIMEOUT AFTER 1 hour
# ...

THE LOGS LOOK LIKE:

$ node --version
v6.11.3
$ meteor --version

Even with METEOR_ALLOW_SUPERUSER or --allow-superuser, permissions in your app
directory will be incorrect if you ever attempt to perform any Meteor tasks as
a normal user. If you need to fix your permissions, run the following command
from the root of your project:

  sudo chown -Rh <username> .meteor/local

Meteor 1.5.2
$ pwd

...

$ google-chrome --version
Google Chrome 69.0.3497.92 
$ cd src
$ pwd
/builds/thebarty/theproject/src
$ curl https://install.meteor.com?release=1.4.4.1 | /bin/sh 2>&1 | cat
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  7786    0  7786    0     0  14155      0 --:--:-- --:--:-- --:--:-- 14156
Removing your existing Meteor installation.
Downloading Meteor distribution

Meteor 1.4.4.1 has been installed in your home directory (~/.meteor).
Writing a launcher script to /usr/local/bin/meteor for your convenience.

To get started fast:

  $ meteor create ~/my_cool_app
  $ cd ~/my_cool_app
  $ meteor

Or see the docs at:

  docs.meteor.com

$ meteor --version --allow-superuser

Even with METEOR_ALLOW_SUPERUSER or --allow-superuser, permissions in your app
directory will be incorrect if you ever attempt to perform any Meteor tasks as
a normal user. If you need to fix your permissions, run the following command
from the root of your project:

  sudo chown -Rh <username> .meteor/local

Meteor 1.4.4.1
$ meteor list --allow-superuser

Even with METEOR_ALLOW_SUPERUSER or --allow-superuser, permissions in your app
directory will be incorrect if you ever attempt to perform any Meteor tasks as
a normal user. If you need to fix your permissions, run the following command
from the root of your project:

  sudo chown -Rh <username> .meteor/local

/bin/bash: line 89:  2363 Killed                  meteor list --allow-superuser
Pulling docker image gitlab/gitlab-runner-helper:x86_64-35e8515d ...
ERROR: Job failed: execution took longer than 1h0m0s seconds

@arturs @rohanray @idmontie @macrozone you’ve been there before. Can you help out and post your current working Gitlab configuration?

@idmontie: is this your current Gitlab setup working? https://gitlab.com/memosa/meteor-gitlab-ci? Any tips?

Gitlab

have not found a solution, but it might simply have to do with the speed of the docker-image and RAM allocated. Seems like the meteor list command really does take longer than 1 hour to execute in the runner…

… right now I am hosting this on 3 GB RAM DO-Droplet for 15 $ per month.

I’ll try to add swapping on both the droplet and the docker image

I feel your pain what’s why I wrote my own … it builds then redeploys our meteor server side + android apk triggered by a git push since its using a webhook ( https://github.com/adnanh/webhook ) to listen to github publish actions … nothing on its own too mind bending just a lot of shell and a sprinkling of golang … using docker in docker (which is just a few config files) so every server we have (in addition to meteor) is inside its own container … been running in prod for 6 months as well as a dozen dev domains … at some point I’d like to carve it out of our repo and either open source it or spin it into a paid service … was tons of work yet since we are HIPAA some existing tooling would not fit our reqs

essentially I have a container dedicated to devops actions where all of above happens … on rebuild it brings down then launches fresh docker images

as an aside I’m using https://github.com/tomav/docker-mailserver as a postfix email server which sends out as well as receives SMTP traffic - then wrote a custom postfix content filter in golang to parse incoming email then persist body as needed into alt messaging drains

nice side benefit of rolling your own is I also use this to cut a virgin VPS box into a fully functioning dev toy domain using a one-liner which sets up all of above

2 Likes

Hi guys,

I need your help!

I’m stuck on Gitlab CE and can NOT get Chimp (chimpy) to run…for some reason chimp has problems getting chrome started. Please have a look here: https://github.com/TheBrainFamily/chimpy/issues/131.

Does anyone have an idea what else to try?

Did you try BuddyWorks? Tried a few options and this one was really pleasant to use especially for a beginner like me :wink:

1 Like

@nicooprat Thanks for the hint. But I see that it only includes 120 free runs. Unfortunatley his is not enough for my project and small commit-habits. :grinning:

Still hanging with getting chimp to run on Gitlab (see above). Anyone got an idea what else to try?

I’ve written about how to setup AWS codestar for Meteor. I’m not sure if and how it works with Bitbucket, but its for sure cheap…

At time of writing this guide it was still in an early stage though. Not sure how it is now

1 Like

@cloudspider Thanks a lot for the link. It looks really interesting… I might give it a try if my Gitlab efforts fail.

Current Status

With Gitlab I ran into 2 problems, which I am currently trying to fix:

  1. Chimp does NOT work https://github.com/TheBrainFamily/chimpy/issues/131
  2. XML reports are NOT reported via email and in the jobs-list of the user-interface. This is something I loved at CircleCi: Getting an email stating “10 tests fail. 190 tests passed" and then being able to dig deeper with a few clicks. Right now I am having problems configuring this on Gitlab. see https://forum.gitlab.com/t/10-tests-passed-190-tests-failed-how-to-setup-circleci-style-test-reports-possible-at-all/19093

Does anyone have a few hints? :smiley:

@thebarty , you can refer to the dockerfile that I use for carrying out chimpy tests here https://gitlab.com/ziinc/barebones/blob/master/.config/dockerfiles/ci

Basically, it’s a docker file that extends an image with meteor installed, installing headless chrome and the necessary fibers build tools. you’ll need to register it as a repository image before using it on the CI runner.

The chimp config that you will need to use is here gitlab.com/ziinc/barebones/blob/master/.config/chimp-ci.js

You will also need to run both the meteor app and chimpy as child processes (as gitlab ci does not support background processes), so you will have to use the node script .config/ci-subprocess.js and run it with babel-node.

You can view the .gitlab-ci.yml used for running your user acceptance stage.

do note that chimpy (in it’s current version) still requires node v.8.x as fibers has not been upgraded yet.

Also, check out the npm script test-ci in the package.json to see how the node subprocess script is used.

1 Like