Using istanbul/nyc for code coverage in Meteor

testing
1

I really need some code coverage tools and the existing packages I’ve found for Meteor aren’t working with Meteor 1.6

It seems like the rest of the Node world is using nyc to instrument tests.

It works like magic but is kind of a hack imho and meteor breaks the hacks that it uses. It is possible to get it working by specifying a path directly to the meteor tool index.js like so: meteor npx nyc meteor node $HOME/.meteor/packages/meteor-tool/1.6.0-beta.13/mt-os.osx.x86_64/tools/index.js test --once --settings test.json --full-app --driver-package meteortesting:mocha

Another thing experiment I did today was using the istanbul babel plugin to get coverage output on the server side with some short patches to meteortesting:mocha. I’ll publish this in a fork shortly if anyone is interested in checking it out, but it has minimal configuration options currently.

What’s missing? One thing is the ability to have different babel environments in .babelrc since you wouldn’t want to run istanbul babel plugin in production.

Also I didn’t manage to get it working for client tests yet but it shouldn’t be too hard.

There’s really two situations where coverage would be nice:

  1. For CI I’d like to output coverage reports and upload to coveralls/codecov. the NYC method of running Meteor test is actually a good way to go here, I think what would be required is a reliable way to invoke meteor that works with nyc that doesn’t break the magic meteor does either. I may take a stab at this later and try to understand how Meteor could work with the nyc shim.

  2. Interactive coverage reports: this is where I want to add more tests to improve my coverage. What’s important here is the console output and HTML coverage generation from meteortesting:mocha. Getting it working for both client and server tests is ideal.

Anyway, I will continue on and let you know what I figure out.

3 Likes
2

Well, after playing with nyc for a bit I realized that while I can get it working just great for server tests, I will still need something else for the client.

I think that means modifying meteortesting:mocha to collect the coverage data and send it back to the server at the end of the tests, and then to save that data in a way that can be merged with the server side coverage.

Don’t want to totally re-implement https://github.com/serut/meteor-coverage but it seems very specific to both practicalmeteor:mocha and the old version of istanbul. Ideally re-use as much of nyc as possible and create a thin shim that would integrate with meteortesting:mocha

Also I created a PR for Meteor to allow babelrc env support. This allows me to add babel-plugin-istanbul so that nyc can generate it’s reports properly during testing.

There seems to be a failing test but I thought all tests passed locally. Will have to look into this again shortly.

3

Update on nyc and Meteor. I was able to get it working nicely with server side tests with no modifications to meteortesting:mocha and about 10 new lines of code in meteortesting:browser-tests

The other modification needed is to run the node & meteor tool directly without using the wrapper script. I believe by studying nyc it would be possible to fix this so that we could use meteor + nyc without any changes as well.

2 Likes
4

I’m going to try and use this in the next week or so. did you figure out a way to use this for clients?

5

more or less, it should be pretty easy as the heavy lifting is done by istanbul babel plugin and nyc for reporting.

I have a fork of the meteortesting:mocha with a POC for client side coverage.

Not using it much currently however as I ran into some issues running my tests when coverage was enabled (server only)

To be specific, a certain specific mocha test would intermittently time out when coverage was enabled. Despite adding a bunch of debugging code I couldn’t figure out why it would time out. Maybe just needs more timeout allocated but my instinct wasn’t saying that. I’ll try again when I have some more time.

6

Here’s what I did to enable logging the coverage data from a client side test. I didn’t too much testing of it but it seemed to work fine.

Some point soon I’ll try to refactor into something that could be accepted as a PR upstream.

7

thank you for the pointers… i’ll try to play around with it and see how it goes.

8

Here’s some information on how I use istanbul on the server side.

  1. You need a custom .babelrc that uses BABEL_ENV to selectively enable istanbul. You don’t want to run it all the time for hopefully obvious reasons.

  2. You need a custom Meteor invocation because nyc relies on being able to override certain parts of the child node process and Meteor tool kind of messes that up.

  3. Once you’ve generated the coverage data you can either run a report using nyc or you can upload it to coveralls/codecov

2 Likes
9

Here are my initial results -

we currently only have server side unit tests so i removed the --full-app option from the execution script. with that it seems to not work…

|----------|----------|----------|----------|----------|----------------|

File % Stmts % Branch % Funcs % Lines Uncovered Lines
All files Unknown Unknown Unknown Unknown
---------- ---------- ---------- ---------- ---------- ----------------

Im still digging through it

10

What test runner are you using, what is in your babelrc, how did you invoke the test runner?

11

i used your run-coverage.sh script - with the same test runner - except for --full-app option. I only have unit tests. I tried switching to other drivers as well but did work. babelrc is also the same.

I was looking through nyc and there was some additional option for “presets”: [“es2015”]. that didnt help either.

12

hello @hexsprite ! Nice idea to use nyc, I’m using it too with webpack-mocha and that’s just perfect ! The future of js coverage… And maybe someday it will be really easy to export coverage on Meteor 1.6 ! :smile:

Your solution is quite interesting on the server side, as every piece of code executed on the node thread is instrumented and covered, but it also means that the entire meteor stack is covered.
That may be an issue, as there is more files in the coverage report than in your project, so it can lead to mistake in coverage report : paths are not truncked to your app folder as there is the meteor stack covered too, but source maps nyc reads are relatives to your root app folder.
nyc is quite nice but there are usecases it isn’t aware of.
So the question is, when you look at the report, do you see the coverage correctly mapped on all server files, and meteor files ignored ?

That’s totally nice to use the babel plugin to instrument the client side ! Good job with the .babelrc pull request !!

However, it means you need an headless browser to run client tests, collect coverage and export it to the developper.
That’s the job of your fork of meteortesting/meteor-browser-tests runner.
I advice you to send the coverage dump to the server in order to create the complete coverage report of your app, a metric that you send to sonar, coveralls, your customer…

You should consider to create a repository that contains a minimal app with some client and server files to cover, to show how to use it and what are the limits.
That’s not an issue if you run the hack directly on a test server like travis, but it would help to see the stack working on a server.

As the author of lmieulet:meteor-coverage, let me explain few things

  • meteor-coverage is totaly agnostic of the test framework that you use. In fact, you can even covers your app while you’re browsing your app with a browser.
    I’ve developped a probe that lives inside the app, starts to covers when the server part of the module is loaded and that provides an Http API used by the automated browser to creates reports.
    In my opinion, a module that starts to cover your app from the inside offers more control and possibilities than nyc, but is a lot more error prone.
  • on the other hand, there is only one test runner that supports meteor-coverage, my spacejam fork. If you look at the diff between my fork and the original one, there is only that part that concerns coverage manipulation
  • and yes, meteor-coverage is not ready for node v8 and meteor 1.6 ! The issue serut/meteor-coverage#54 makes hypothesis that are not insurmountables, we will see how it turns out.
13

@serutan hey!

Actually, since I’m using the babel plugin (not nyc) to instrument Meteor its seems to only instrument my application code, not the Meteor code itself. It only shows coverage from the application, not Meteor.

All the paths are rooted at the application directory.

Yes, the coverage is being generated on the client and then exported to the server. It’s pretty easy as its just a JSON blob.

What I like about your solution is that it is interactive – you can make changes, your tests run and the coverage is updated.

However I had problems with spacejam getting all my tests to run correctly and thus have been using meteortesting:mocha for running all my tests which is not supported.

1 Like
14

Thank you very much, your post is very helpfull.
Previously, I am unable to use lmieulet:meteor-coverage for meteor 1.6 because I have empty client apps.
I want to just test server apps only with meteor 1.6 and your post is the answer.

15

Because I want to use all js, I rewrite your run-coverage.sh and replace it using gulp task.
And since I do not fully understand the command, I use exec several times to mimic the shell command.

Additional requirement inside package.json (istanbul configuration, additional script to run, and additional packages [dependencies or devDependencies]).

"nyc": {
    "reporter": [
      "html"
    ],
    "exclude": [ "gulpfile.js" ],
    "report-dir": "./coverage"
},
"scripts": {
    ...
    "coverage": "gulp test:coverage"
},
"dependencies": {
    ...
    "gulp": "^3.9.1",
    ...
},

The gulpfile.js replacement for run-coverage.sh

/**
 * @file
 * gulpfile.js
 */

// Require gulp.
const gulp = require('gulp');
// Require execSync.
const { execSync } = require('child_process');
// Require del.
const del = require('del');
// Require path.
const path = require('path');
// Require package.json.
const config = require('./package.json');

// Create gulp task to clean up coverage.
gulp.task('clean:coverage', () => del([config.nyc['report-dir']]));

// Create gulp task to do test coverage.
gulp.task('test:coverage', ['clean:coverage'], () => {
  // Define shell command environment.
  process.env.TZ = 'UTC';
  process.env.BABEL_ENV = 'meteor:coverage';
  // Get node for meteor.
  // Alternative: const nodePath = process.execPath;
  // Only if called using: meteor npm run coverage.
  const shellCommandGetNodePath = 'meteor node -e "process.stdout.write(process.execPath)"';
  const nodePath = execSync(shellCommandGetNodePath).toString().trim();
  const meteorIndex = path.resolve(nodePath, '../../../tools/index.js');
  // Change the --settings value.
  const shellCommandCoverage = `node_modules/.bin/nyc ${nodePath} ${meteorIndex} test --full-app --settings __tests__/settings.json --raw-logs --once --driver-package meteortesting:mocha --port 9500`;
  const result = execSync(shellCommandCoverage).toString().trim();
  console.log('Result:', result);
});

Now we can run test coverage with this command:

meteor npm run coverage

Or if we have installed gulp-cli:

gulp test:coverage

Hope this usefull.
Thanks.

16

@hexsprite’s script works beautifully for my server files, however they seem to skip over my client files. Is that happening to anyone else?

Also it doesn’t seem to work at all when I run this inside a docker container.

17

Yes, I did manage to get the client side coverage working with a custom fork of meteortesting:mocha but I didn’t totally finish it yet and it wasn’t accepted for merge. You can find it on my Github though if you want to play with it.

18

@hexsprite: is this diff the correct one?

19

yup. it also has a patch for nightmare to log exceptions to the console.

20

Anyone knows an open source Meteor app having a babel configuration and some tests ? The package meteor-coverage is almost ready to embrace this new paradigm, I just need to test everything before shipping the V3 release but the proof of concept is already working.

1 Like