Enhanced, but simply-stupid CircleCi setup that first runs UNIT-INTEGRATION-tests and then CHIMP-ACCEPTANCE tests?

Hi guys,

my unit- and integration-test’s run-time has increased to over 10 minutes (WUUUUHHHHHH). Right now I am running them locally before each commit, but waiting for those tests to pass sucks a lot.

Now I’d love to extend my CircleCli-Setup to also run my unit- and integrational-tests. It would be cool to find a simple solution for this.

@warehouseman: do you have a quick hint where to look on how to do this?

I am following meteor-guides and run my unit-tests via practicalmeteor:mocha and meteor test --driver-package practicalmeteor:mocha and I saw that it is possible to run those tests on commandline via spacejam.

This is my current circle.yml:

# References
#  * https://forums.meteor.com/t/what-is-the-simplest-setup-for-meteor-chimp-on-circle-ci/31627
machine:
  node:
    version: 4.6.2  # note: 6.5.0 did NOT work (chimp installs, but meteor is NOT available)
  java:
    # This is needed by Chimp to run Selenium so we can start Chrome and Firefox
    version: oraclejdk8
  environment:
    _JAVA_OPTIONS: "-Xms512m -Xmx1024m"
    CIRCLE_ENV: circle  # for setting stuff in our .js files depending on environment

dependencies:
  pre:
    - pwd
    - cd ~; pwd;  # find HOME
    # START of chrome-driver-update
    #  NOTE: without this, we will get a "Chrome version must be >= 54.0.2840.0"-error
    #  copied from https://github.com/meteor/todos/blob/master/.testing/upgrade_chrome_version.sh
    - google-chrome --version
    - wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
    - sudo sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >>    /etc/apt/sources.list.d/google.list'
    - sudo apt-get update
    - sudo apt-get install google-chrome-stable
    - google-chrome --version
    # END of chrome-driver-update
    # START of meteor pre-heat
    - cp src/.meteor/versions ${CIRCLE_ARTIFACTS}/versions_pre;
    - cd src; mkdir -p .meteor
    - if [ ! -e src/.meteor/meteor ]; then cd src; curl https://install.meteor.com | /bin/sh 2>&1 | cat; fi
    - pwd; cd src; pwd; meteor npm install;  # install npm-packages from ./src/packages.json
    - cd src; meteor --version;  # trying to cache and pre-heat meteor
    # - cd src; meteor --get-ready;  # currently FAILS, probably NOT enough memory
    # END of meteor pre-heat
    - mkdir -p ${CIRCLE_ARTIFACTS}/screenshots
    - cd src; npm install -g chimp@0.45.0  # @0.46.0 might NOT work. in general this is NOT working with LOW node-version.
    - cd src; chimp --path=noexist # Cache chimp deps by running it without any tests
    - npm install -g junit-merge  # https://github.com/drazisil/junit-merge

  # Run METEOR IN PARALLEL
  post:
    - pwd
    # debug used versions
    - cp src/.meteor/versions ${CIRCLE_ARTIFACTS}/versions_post;
    - less src/.meteor/versions > ${CIRCLE_ARTIFACTS}/meteor.log;
    - date > ${CIRCLE_ARTIFACTS}/meteor.log
    - date  # for easy access on circleCi's overview
    - cd src; meteor test --full-app --driver-package tmeasday:acceptance-test-driver --port 3000 >> ${CIRCLE_ARTIFACTS}/meteor.log 2>&1 & :
        background: true

test:
  post:
    - date  # log time so we can check, if chimp was started BEFORE meteor has finished loading
    - pwd
    - ls -al
    - cd src; chimp -version; node --version; npm --version; java -version; meteor --version; google-chrome --version;  # log versions for debug
    - sleep 5m  # give meteor-background some time to wake up. NOTE: 10m will NOT work either with newest chimp-problem. Keep it at 5m.
    - date  # log time so we can check, if chimp was started BEFORE meteor has finished loading
    # SPLIT CHIMP into 2 runs, so that we do NOT timeout on CircleCi (120 minutes-limit).
    #  We are using "mocha-circleci-reporter", which writes into "test-results.xml" by default,
    #  so after EACH run we need to copy&rename the file to $CIRCLE_TEST_REPORTS
    #  and then combine both results into ONE final "$CIRCLE_TEST_REPORTS/test-results.xml"
    # REFERENCES
    #   => Timeout-problem: https://discuss.circleci.com/t/timeout-more-than-120-minutes/640/5
    #   => CircleCi-reporter https://github.com/sandcastle/mocha-circleci-reporter
    #   => Merge-tool: https://www.npmjs.com/package/junit-merge
    - less $CIRCLE_TEST_REPORTS
    - less ${CIRCLE_TEST_REPORTS}
    - cd src; chimp ./tests/_circle-ci-chimp-config-run-1.js --ddp=http://localhost:3000 --path=tests --mocha
    - cd src; cp test-results.xml ${CIRCLE_TEST_REPORTS}/test-results-1.xml
    - cd src; cp test-results.xml ${CIRCLE_ARTIFACTS}/test-results-1.xml
    - cd src; chimp ./tests/_circle-ci-chimp-config-run-2.js --ddp=http://localhost:3000 --path=tests --mocha
    - cd src; cp test-results.xml ${CIRCLE_TEST_REPORTS}/test-results-2.xml
    - cd src; cp test-results.xml ${CIRCLE_ARTIFACTS}/test-results-2.xml
    - junit-merge -d ${CIRCLE_TEST_REPORTS}  # see https://www.npmjs.com/package/junit-merge
    - cd ${CIRCLE_TEST_REPORTS}; ls -al;  # lets see why the next line fails
    - cp ${CIRCLE_TEST_REPORTS}/test-results.xml ${CIRCLE_ARTIFACTS}/test-results-merged.xml

Hi @thebarty

Sorry for the delay in replying.

You may not realize that you can run your own scripts to do just about anything; it’s really not necessary to use their clunky “itemized list” structure for each single shell command.

At line 33 in this circle.yml you can see I call install_all.sh, which sources other shell scripts. Calling their subroutines, it performs all the same installations and preparations I do on my development machine.

1 Like

@warehouseman: thanks a lot for your HYPERFAST reply!!! :slight_smile:

Yeah I have seen this stuff before. :slight_smile:

So I guess right now in your setup you are NOT running unit-tests on CircleCi?

I’d like to have the following logic:

  1. Have my CircleCi-script run UNIT-tests / INTEGRATIONAL-tests first
    => IF they PASS: run CHIMP-tests see 2)
    => IF they FAIL: have CircleCi send me a nice error-report by email (containing a nice unit-integrationa-test-results.xml). This means that I need SpaceJam to output a CircleCi-compatible xml

  2. AFTER 1) has passed: Run my existing CHIMP-tests and send me the results.

Yeah, You caught me! I am in the middle of a bit of a renovation of my circle.yml, and left off my rather useless unit tests, temporarily.

As I understand it, your scripts are supposed to halt the build if they terminate with a non-zero status. So when a script fails you must do, for example …

 exit 1;

… and CircleCI should cease processing from that point.

I noticed you don’t specify any cache directories like I do on line 24. That’ll cost ya!

Jo man! Thanks a lot for the hints and the discussion. You have helped me clarify what I want to accomplish. :slight_smile:

I’ll give it a try and post my results here.

By the way: You should try a tool like camtasia and get a cheap headset to record your voice. I am really missing it in your videos. Without a voice (to me) watching a video feels even more exhausting than reading. And did you know that even Jimi Hendrix hated his voice, but his audience still loves it? Same with Bob Dylan and a lot others. So give it a try and after the 5th video you’ll love your voice. Promised!

Glad to help.

It’ll be a long time before I make another video series, but I’ve had enough feedback about the silence, that I will voiceover next time.

Thanks!

Hi guys,

for your reference: FINALLY - after hours of time wasted - I got it to work:

THE SOLUTION

LOCAL DEVELOPMENT MACHINE

LOCALLY (OSX) we use practicalmeteor:mocha to have a nice html-reporter.
By the way: We found that this reporter is MUCH faster than using dispatch:mocha or practicalmeteor:mocha-xunit-reporter. Even if it runs ONLY server-tests. Why? No idea!

CIRCLECI

On CircleCi we need to meteor remove practicalmeteor:mocha and meteor add dispatch:mocha.
Furthermore we need to “HACK” dispatch:mocha to generate xml-files by installing a FORK of “practicalmeteor:mocha-core”.

Install dispatch:mocha on CircleCi and make it generate an XML-output

=> see https://github.com/DispatchMe/meteor-mocha/issues/2 for more info

meteor remove practicalmeteor:mocha;  # cleanup for `dispatch:mocha` (otherwise we'll get a version-constraint)
meteor add dispatch:mocha  # we'll use this package to generate CircleCi compatible xml-files
meteor add practicalmeteor:chai  # needed for our tests
cd src/packages; git clone https://github.com/ayhid/meteor-mocha-core.git
# finally create the xml file
targetXmlFilePath="$(pwd)/unit.xml"; MOCHA_REPORTER=XUnit SERVER_TEST_REPORTER=xunit-file XUNIT_FILE=$targetXmlFilePath meteor test --once --driver-package dispatch:mocha

Have fun and thanks for your help!

1 Like