Coverage on meteor


#1

Hello everyone !
After my first attempt to add coverage to meteor by editing Meteor (Package test coverage using Istanbul), I come back with coverage, but this time with a debugOnly package.
You can find additional informations here:
https://atmospherejs.com/lmieulet/meteor-coverage

Hope it will help people :wink:


Code coverage for meteor app?
#2

The package is still working on meteor 1.3 :wink:

Here is the status of this package (a fresh one may exists on the repository)

Limitation(s) / open issues

  • Improve performences
  • Test with coffee script, typescript
  • Create some repo exemple to show how to do
    • test-packages a package
    • test-packages an app that have some packages
    • test
    • test --full-app

#3

Changes:

  • Update to Istanbul-api (support source map)
  • Add global environment variables which disable by default the coverage processing
  • Add config file to let the user declares which files will be ignored

#5

Changes 0.2.0

  • Source map support on client and server side
  • Tested on windows, linux, osx

Some screens:



#6

Changes 0.4.0

  • When there are several types of test, we want to merge the coverage of each one, and then push the coverage report to coveralls or whatever. It’s now possible using the Client API.
    • Supported types of export: lcov (used by Coveralls.io,…), coverage (dump the coverage object in a file)
    • And we can import a coverage file
  • Improve performances, but still 10x slower to load the app when coverage is active than when it’s not

Next step:

The next step is to fork a Meteor test runner and use the API provided by my package to save the coverage report between types of test and then upload the result to Coveralls.io


#7

Looks great. I am looking to add coverage to my boilerplate app, but at a loss a the moment still.


#8

I’m working on a fork of spacejam that can execute all the test-packages of a meteor app and then export the lcov file:


For the moment, the report contains the coverage of all files prototyped (~ 150 files), I need to clean that and figure it out what is required by coveralls.io to accept a coverage report.
Right now, you can add my package to your meteor app and looks the web report at localhost:3000/coverage , tell me if it’s ok for you.


#9

Tried… failed :frowning:

The behaviour I was expecting…

meteor add lmieulet:meteor-coverage
meteor test --driver-package practicalmeteor:mocha --port 3100
http://localhost:3100/ to see the test run
http://localhost:3100/coverage to see coverage ... no luck

Other behaviour I was expecting

meteor add lmieulet:meteor-coverage
meteor 
http://localhost:3000/ to see the application
http://localhost:3000/coverage to see coverage ... no luck UI router redirects to the login screen
logged in
http://localhost:3000/coverage to see coverage ... no luck UI router redirects to not found page

#10

My package is not enabled unless the environment variable COVERAGE=1 is set up
On Unix, you can test with the following :

COVERAGE=1
COVERAGE_VERBOSE=1
COVERAGE_APP_FOLDER=/path/to/your/meteor/app/ meteor

Windows:

set COVERAGE=1

meteor

Then, you can look the report: http://localhost:3000/coverage
The last things you can try is to send the client coverage to the server. Open the console tab of your browser and execute that line:

Meteor.sendCoverage(function(stats,err) {console.log(stats,err);});

Now on the report http://localhost:3000/coverage you see the client coverage.

And I just tested with a basic application (meteor create app) and it works as expected with meteor add practicalmeteor:mocha && meteor test --driver-package practicalmeteor:mocha. It outputs me the following global stat of lines executed:

30.57% Statements 9024/29520
16.41% Branches 2991/18227
25.36% Functions 1373/5415


#11

Cool, got it working. Though I only see the server side code being covered and the PATH is kind of off (SET/ENV/COVERAGE_APP_FOLDER/OR/READ/README/) :wink:

It’s a good start.

I think for non-production we should be okay to have /coverage enabled and have it turned off when Meteor.isProduction much like how I did it http://www.trajano.net/2016/04/minor-performance-boost-for-angularjs-and-meteor/

I did do the Meteor.sendCoverage(function(stats,err) {console.log(stats,err);}); and now I see the the client coverage. Too many things there though. I think a reasonable default would be just anything that is in packages, .meteor, meteor home and node_modules should be removed from the list.

As far as COVERAGE_APP_FOLDER=/path/to/your/meteor/app/ perhaps you can do it by relative path much like how I did it here http://www.trajano.net/2016/04/publishing-meteor-application-api-documentation/ I just put a console.log(process.cwd()) to find out where I am then just traversed from there.


#12

Cool, got it working. Though I only see the server side code being covered and the PATH is kind of off (SET/ENV/COVERAGE_APP_FOLDER/OR/READ/README/) :wink:

Do you have setup COVERAGE_APP_FOLDER ? It looks like the package cannot find process.env.COVERAGE_APP_FOLDER :confused:

I think for non-production we should be okay to have /coverage enabled and have it turned off when Meteor.isProduction much like how I did it http://www.trajano.net/2016/04/minor-performance-boost-for-angularjs-and-meteor/1

This package is debug only:

The debugOnly property tells Meteor’s build process to exempt a package from your production code.

So it’s even better than testing Meteor.isProduction !

I did do the Meteor.sendCoverage(function(stats,err) {console.log(stats,err);}); and now I see the the client coverage. Too many things there though. I think a reasonable default would be just anything that is in packages, .meteor, meteor home and node_modules should be removed from the list.

Work in progress !

As far as COVERAGE_APP_FOLDER=/path/to/your/meteor/app/ perhaps you can do it by relative path much like how I did it here http://www.trajano.net/2016/04/publishing-meteor-application-api-documentation/1 I just put a console.log(process.cwd()) to find out where I am then just traversed from there.

This cannot be reproduceable every where on every OS, sometimes Meteor stores the build outside of the main project directory (I have no reproducible environment to provide, but I observed it several times). There are several usage of the COVERAGE_APP_FOLDER inside the package, and right now I cannot say to you if it’s possible to eliminate this dependency.

With the fork of spacejam I’ve created, this environment configuration is created automaticaly, so you just ask spacejam to test your application and print the coverage in a file.


#13

Changes 0.5.0

  • edit the Travis CI conf file
    • Using a fork of spacejam, it runs tests, saves the coverage inside a lcov.info file and using node-coveralls it sends the coverage to coveralls.
  • add the coverage img to the repo ! Coverage Status
    I think this is the first meteor package that has one :slight_smile:
    Note: the server code of this package is not covered at all because on runtime the server reads and executes the meteor-coverage server js and after that it starts to instrument every single js that will be read.
  • fix file paths when meteor test-packages inside a package (necessary for covering itself)
  • clean & other fix…

Next

Check if any meteor package that uses Tinytest and Travis can run their tests on travis and export coverage.
Support for meteor app coverage.


Adding Quality to Meteor
#14

Changes 0.7.1

  • remove underscore dependency: I tryed to instrument a real package and had severe issue with this dependency. The simple fix was to remove all reference to underscore.
  • works well with test-packages
  • add eslint

So here is real example of meteor package coverage:


Coverage Status

Here is an example of .travis.yml:

language: node_js

node_js:
  - 0.10.40

cache:
  directories:
    - $HOME/.meteor
    - $HOME/.npm

before_cache:
  - rm -f $HOME/.meteor/log/*.log

before_install:
  # Download Meteor
  - PATH=$PATH:$HOME/.meteor
  - if [ ! -e $HOME/.meteor/meteor ]; then curl https://install.meteor.com | sh; fi

  # Install spacejam
  - npm install -g https://github.com/serut/spacejam/tarball/master
  - npm install -g https://github.com/nickmerwin/node-coveralls/tarball/master
  - npm install -g eslint

script:
    # Check clean code
  - eslint .
    # Run packages tests
  - spacejam test-packages ./ --coverage && cat lcov.info | coveralls

You can try it with your package, but keep in mind that this .travis.yml file is not the final version.

Next

Do the same thing for meteor test & meteor test --full-app


#15

16% coverage? not too impressed… j/k

I see you use spacejam in your travis, but you’re doing test-packages, can it work with the main app?


#16

Right now spacejam doesn’t support the meteor test command, but the pull request is there : https://github.com/practicalmeteor/spacejam/pull/51.


#17

oh suX0r

r0x0r. Hopefully it gets merged soon.


#18

Fantastic work, @serutan. I added the coverage package to the Meteor Chef Base, and it picked up the .test[s].js files with no problems.


#19

Changes 0.7.2

  • Handle the server coverage of this package when testing so now it’s covers 100% of the meteor-coverage files (client and server side)

I create a sample repository to show how to test using spacejam and this package: https://github.com/serut/meteor-coverage-app-exemple
It supports:

  • Test package using tinytest
  • Test package using mocha
  • Test app using mocha
  • Test full-app using mocha

I made several pull request to spacejam: https://github.com/practicalmeteor/spacejam in order to merge my changes into the main repository and let developpers embrace coverage soon and easily.


#20

do you know how spacejam relates to meteor test?


#21

spacejam test
==> meteor test --driver-package=test-in-console --port 4096

spacejam test-packages
==> meteor test-packages --driver-package=test-in-console --port 4096

spacejam [] --driver-package=practicalmeteor:mocha-console-runner
==> meteor [] --driver-package=practicalmeteor:mocha-console-runner --port 4096

If you want further information about spacejam, run it with the option --loglevel debug. I will print a lot of informations.