Where’s the best place to document additional utilities?
Clinical Meteor has gone live last month with a release-track wide QA infrastructure on Circle CI using Gagarin and Nightwatch (both of which are built on Mocha or have Mocha support). It’s working perfectly, and PR requests are starting to ramp up across the entire distro.
We’re currently extracting code samples across 2 dozen packages and 3 or 4 reference apps and will be providing a QA cookbook for apps that need to go under FDA regulatory review.
I just tried exactly that and I got hot code push with all tests re-running. Could it be that the way you added console.log led to a client exception so that tests stopped running?
If you can describe a repro, please file a GitHub issue. But at the moment I think hot code push for tests works fine.
I tried it again. The problem is in the UI of the reporter. When I click on one of the right arrows of the test scenario, the url changes to http://localhost:3000/?grep=todos%20leaves%20createdAt%20on%20update, but still all tests show. When the tests rerun or I refresh the page, only the tests show that match the filter todos%20leaves%20createdAt%20on%20update. I didn’t see the URL change and the delayed effect before, so that was confusing.
Is this work included in the latest 1.3-beta.11 release? We are presently writing all our unit tests in mocha and executing them in our automated build system using spacejam and the practicalmeteor:mocha-console-reporter.
Upon upgrading to the latest beta.11 phantomjs started throwing an error at us when run using from spacejam:
spacejam: spawning phantomjs
phantomjs: Running tests at http://localhost:4096/local using test-in-console
phantomjs: TypeError: 'undefined' is not a function (evaluating 'RegExp.prototype.test.bind(/^(data|aria)-[a-z_][a-z\d_.\-]*$/)')
When I run the same tests directly using meteor test-packages with either the console or web reporter it works fine.
Just wondering if others have seen that.
Also, we’ve been looking forward to being able to create app level unit tests in mocha such that we can run them in our CI environment. Is this possible? The guide seems pretty skinny on the subject.
After a bit more digging, I think may be an issue with phantomjs not supporting Function.prototype.bind(). I tried resolving it using the ‘phantomjs-polyfill’ package.
Basically I added it to the npm.depends() section of my package and then simply required it at the start of my test app. But I’m still seeing the issue.
We’ve been using spacejam/phatomjs successfully for a while now so I don’t know if we added a test case that is tripping it up now or if it was the switch to b.11. I’ll walk back through our other changes to try and isolate the issue.
I solved the phantomjs bind() polyfill issue… we had inadvertently removed the es5-shim package which was supplying the polyfill.
However I am still having trouble getting our mocha tests to run using spacejam. I’ve noticed we get an error in the console that the runTests export is missing:
Error: Test driver package practicalmeteor:mocha-console-runner missing runTests export
I also see this error when running meteor test-packages directly (with both html and console reporters) however there the test cases still execute and report their results. I suspect phantom may be choking on the error and bailing early on the results.
I saw there was a modified version of the practicalmeteor:mocha driver that implements the missing export but this startup code can’t seem to find it and throws the error:
Package.meteor.Meteor.startup(function() {
var testDriverPackage = Package["practicalmeteor:mocha"];
return;
if (!testDriverPackage) {
throw new Error("Can't find test driver package: practicalmeteor:mocha");
}
if (!testDriverPackage.runTests) {
throw new Error("Test driver package practicalmeteor:mocha missing `runTests` export");
}
testDriverPackage.runTests();
});
}
But somehow the tests still seem to run in the browser…
I have an app fully converted to modules and mocha, and it’s working well with meteor test-app --unit locally. Now is there a way to make it work on CI yet? spacejam only works with packages afaik
I have a fork of mocha-console-reporter that works (see above) but I’ve not attempted to modify spacejam to support test-app – it would be more involved…
Thanks Tom, I will give the console reporter package you point to a try… However note that I am getting the runTests error even with the normal mocha reporter.
We are currently only running mocha unit tests at the package level - I presume the app units will run similarly in the browser? Will meteor be supporting CI environments directly or will we be forced to use spacejam’s phantom output piping hack forever?
The interactive browser runner is slick for test case development but it is slow and we have a large number of tests to run. Running verification and even integration tests in the browser against your live meteor server makes sense but not so much for units which are generally a bunch of quick but isolated tests.
I guess I’m coming from the nodejs world where the runner isn’t so tied into your application… it feels heavy.
@tmeasday, we can’t use your mocha-console-reporter until your change to remove the hard-coded avital:mocha driver package is released. Are there plans for another beta release anytime soon? I have it working for CI with console reporting by copying avital:mocha into /packages and then replacing the HTMLReporter code with the ConsoleReporter code, and using a custom node CLI I wrote instead of spacejam. I can share this stuff and help document CI usage, but it will be easier with a new beta release.
Yeah we are going to do another beta release probably Monday US time, so this problem should be resolved. Sorry it took a while to get a version out where you can specify --driver-package, I was hoping it would be earlier in the week last week.
Nice work @aldeed! Appreciate your efforts and I like how simple and clean it is. Hopefully we can get our package units working soon based of this or an updated spacejam.
@tmeasday ,
Regarding whether this is built into the core or a separate npm package… it doesn’t matter a whole lot as long as there is a working MDG blessed story for package and app unit testing in both the CI and interactive development environment.
As I mentioned earlier, the CI environment is top priority for us… the interactive html reporter is nice but our automated build tools need to check for regressions and alert folks if the build is broken. We can’t simply trust that every developer will always run all the test cases impacted by their change.
I’m glad to see you have integration testing in your guide and it will be interesting to see how that pans out. Dependency injection would be nice to see at some point as it will enable better unit and integration tests. In nodejs we used mocha with tools like nock and rewire on the server and karma injection in the client. I’m curious to see how we’ll do these things in meteor.
Finally, we are just starting to develop end-to-end validation tests and I’m curious to hear recommendations from MDG or others on that. Currently we’re using the starrynight nightwatch/phantomjs package - again because it supports our CI environment. So far so good with that but I’d like to hear what others are doing too.
Thanks again for all your efforts thus far… I realize its a ton of work and we do a lot of grumbling and complaining on these forums but I’m really happy to see the focus on this topic.
I decided to try beta12 and I was wondering why my tests were not working, as with beta11 they were running fine, and realised that test-app is not working anymore, and now it’s just test. So this works now: meteor test --driver-package avital:mocha. I think you should change everywhere the documentation, because it’s misleading everyone with so many usage changes.
Second thing I would like to comment is that given this command description from the shell:
$ meteor test --help
Usage: meteor test --driver-package <driver> [options]
meteor test --full-app --driver-package <driver> [options]
Runs tests against the application.
Will start a special app based on a test driver (specified with
--driver-package -- read more about driver packages at
http://guide.meteor.com/testing.html#driver-packages) which handles the
task of running tests and displaying the results in the browser when you
visit it.
In normal test mode, no files in your application are eagerly loaded, aside
from test files (files named *.test[s].* placed anywhere in your application).
You can import your app's modules from within your tests and test them in
The last statements which says placed anywhere in your application it’s not really what it says, because I place my tests under app/tests/unit/server/methods.tests.js and doesn’t load at all that file. Maybe a bug or maybe it’s not intended to work as this, who knows that?
Thanks for pointing this out @menda, I’ve updated the documentation you’ve referenced, hopefully I got all of it. Apologies for that oversight, I hope you weren’t too confused.
One potentially confusing thing here is that folders named tests/ remain (as they always have) completely ignored by the build tool. This includes if you put tests in them.
One reason for this is so you can put acceptance tests in them. Another is that it’s confusing (both in terms of writing the build tool, and explaining the rules) to say “files inside tests/ folders are completely ignored, unless they are called *.tests.* and you are running in test mode, etc.”
Is this a big problem for you? You can of course rename the folder to anything else to avoid this behaviour.
How about renaming the tests dir to exclude instead, which makes it explicit and allows a user to consciously put anything in there, including a tests or features sub-directory.
I would also like to see the new test modes check both for *.tests.js and *.specs.js files as they are both very common.
Thanks for all the great work.