As of Feb 2017 it seems like there is no one tool available to run client and server tests in the console ( and thus CI services ). Is it just me, and my implementation?
Here’s my current take on the ecosystem:
practicalmeteor:mocha - Is what I am using, but only does web reporting. The last commit was in Jul 2016, and the owner has said he isn’t available to maintain this package (which I completely understand) . There is a design constraint: testing in browser looks great, but not productive in a number of workflows. Also there is an implementation issue: it runs client and server tests in parallel with the same state. So any ‘integration’ test that needs a DB state will cause a parallel test to fail.
practicalmeteor:mocha-console-runner + spacejam - This is what you would use to test practicalmeteor:mocha in the console. It is failing silently: https://github.com/practicalmeteor/spacejam/issues/67 . There are different points of failure having to install an npm module plus a large meteor package to run mocha tests. @jsep
xolvio:chimp - Can’t tell if it is intended to run full stack tests, or only acceptance. Anyhow I couldn’t get server integration and client unit tests to run. @sam
I’m not sure where to invest my effort in patching components, considering the fragmented state of testing.
@MDG@benjamn@debergalis@sashko Can we get some MDG resources on this? Testing is inconsistently documented, fragmented, and more importantly, broken. We need your help here if people are going to write quality apps with Meteor.
I sincerely doubt devs are testing consistently with recent versions of meteor, and packages. I’m relying on practicalmeteor web reporting which can’t run client and server simultaneously.
The issue you’re encountering with dispatch:mocha-phantomjs has been fixed (see recent discussion). There may be other issues (phantomjs is a bit flaky!), but dispatch:mocha-phantomjs is what we currently use for Meteor’s own CI tests, e.g. the modules test app.
@benjamn thanks. I don’t think the issue I raised is actually fixed. Please take a look at the Meteor Todos app, and try the tests after updating. That’s this issue here: https://github.com/meteor/todos/issues/198
The main problem is that the dispatch:mocha-phantomjs issue is a phantomjs issue, and phantomjs is pretty dead. I might try to swap in electron instead. Selenium webdriver is another option but I believe that would require separately installing binaries.
That said, the Todos tests run fine for me all the time, and there are workarounds explained in that issue if you happen to be on a system that experiences the issue. So I would not say that testing is impossible.
But to the general point, maybe we could centralize the primary working test driver packages in a single Meteor org with some additional maintainers. If anyone is interested in maintaining, let me know.
I’d vote for Chromedriver rather than Webdriver, since Chrome is based on V8, which would make it isomorphic to Node and Mongo. Ties the implementation a little bit more to the ChromeOS philosophy, in the same way that Mongo did; but Chromedriver has been the most stable driver in my experience.
@aldeed thanks for chiming in. I’m grateful for your work on dispatch.
I agree we could consolidate the non-redundant packages into one org.
Though in practice I think it would be more productive to invest in one reference mocha package which runs on electron/chromedriver. More eyes and testers on one package. I think dispatch:mocha-phantomjs is nearly there.
( One technical note, I don’t think its possible to have a headless chrome/v8 browser in windows. Doesn’t affect me, but it would be a non-stater if that was the case.)
Also importantly, I would like MDG to contribute to the design. If they don’t have any buy-in then we’re going to end up with competing implementations that are unmaintained (like the docker situation).
Contrary to PhantomJS, SlimerJS is not headless: you see windows and it needs a graphical environment. You can use a tool like xvfb on a Linux/MacOS system to have this headless feature and to execute SlimerJS on Linux boxes that do not have xorg installed.
OK, I’ve made some changes that I think are in the right direction. I published https://github.com/aldeed/meteor-browser-tests to replace https://github.com/DispatchMe/meteor-phantomjs-tests. The new package requires you to install the necessary browser NPM packages yourself, which makes it more extensible and easier to maintain. I added support for PhantomJS, nightmarejs, and selenium chromedriver initially. Anyone who wants to add others, it should be pretty easy by looking at the current implementations in the “browser” folder.
Then I updated dispatch:mocha to support client tests using any of these browsers by depending on the new package I just mentioned. Check out the updated https://github.com/DispatchMe/meteor-mocha readme for details.
So in summary:
aldeed:browser-tests replaces dispatch:phantomjs-tests (You don’t use this directly. You depend on this if you want to make a new test driver package.)
dispatch:mocha can run client tests now so it replaces dispatch:mocha-phantomjs
dispatch:mocha-browser is still separate for now. There is a possibility this could be folded into dispatch:mocha as an additional browser option, but it needs some thought because of the extra CDN files and such.
@lpgeiger@awatson1978@benjamn Please try out switching from dispatch:mocha-phantomjs to dispatch:mocha and let me know if it works for anyone besides me.
@robfallows It should be pretty easy to add your slimerjs implementation to aldeed:browser-tests, too, if you want to attempt a PR.
And everyone: If you are interested, please volunteer to help maintain browser-tests and I will add you to the repo.
@aldeed always can count on you! Thanks for the updates. I’ll be testing now. I’ve been working on another test runner in parallel. I’ll be happy to help do some maintenance if you want to add me (keyscores on github).
