Velocity is the official test runner of Meteor. What does that mean exactly?


#1

I’m currently working on a project called Meteor Fresh, and am in the process of trying to figure out a testing framework to go along with it. To be honest, I’m having quite a bit of trouble with Velocity. I’ve tried every different package available (jasmine, mocha, cucumber) and I have always run into some kind of problem out of the box be it the tests not running, difficulty in testing packages, etc.

On the other hand, Tinytest seems to “just work” and feels much more integrated with the Meteor core, but is lacking features that come with something like Jasmine or Mocha (spies, before/after hooks, etc). Plus I’m under the impression that it might be phased out soon in favor of Velocity due to being announced as the “official test runner” of Meteor.

What does that mean? Should we stop using Tinytest? Is the MDG planning on fully supporting Velocity? The Meteor testing community appears schizophrenic at the moment, meaning it’s having trouble figuring out what it’s supposed to be. This is my impression at least after reading other forum posts and even the documentation on many packages. I’d love some suggestions on how a Meteor starter template like Meteor Fresh should handle testing.


#2

I’m struggling with the same question, and I too tried all the Velocity testing options and gave up. TBH that was quite a while ago, and it’s probably about time I tried again. This blog actually makes sense to me. I feel I “should” be using Velocity.

In the meantime, Tinytest’s lack of spies and set-up/teardown functions was annoying. But then I discovered practicalmeteor:munit, which is built on Tinytest and just works (actually, there was some jiggery-pokery needed to test with DOM elements).


#3

I have yet to try munit, but it looks very promising. Especially that it is based on tinytest, the testing framework that Meteor was written with. That’s why I’m a bit confused at what it means for Velocity to be the “official test runner” when the MDG themselves don’t use it…it seems a bit like a chef not eating his own food. munit appears to even come with chai and sinon, which are both very active packages. All those packages together create a testing framework that can do anything you would ever need, and it’s based on what the MDG uses to test.

Velocity aims to be compatible with many many different testing frameworks, but I don’t believe that’s what we need. Testing should be opinionated. It should be predictable when applied to the technology that you’re using. So instead of having a single very strong solution, we have 5 solutions that are developed by different people and clearly have many issues. Maybe the testing community should band behind a single solution?


#4

As a bump, I have an issue open on the munit github page about this. It looks like he has plans for a mocha package that uses spacejam to run tests from the terminal with mocha syntax. I’ve got munit up and running with almost no issue whatsoever, so I can only image that this mocha package will be awesome!

Here’s what he had to say.

My plans for the future is to finish a mocha package that I’ve been working on that will support running mocha package tests from the command line using spacejam, as well as support all current mocha reporters. As soon as that will be ready, we will depreciate this one [the munit package].


#5

Interesting question. I’ve been holding off on responding because reasons, but am pretty much in complete agreement with your conclusions @csauer: TinyTest extensions like munit are the way to go.

For what it’s worth, I tried voicing these issues within Velocity team meetings on multiple occasions, but my input fell upon deaf ears. After spending hundreds of hours trying to use Velocity and follow their API and architecture recommendations, I eventually cut my losses when it was apparent I had spent over $10,000+ of time and effort, and the mirrors and test-proxies and other unit-testing specific functionality were still breaking my client’s acceptance tests.

I came away from that experience with the exact same conclusion as you: a package based architecture is the wrong approach, and integration is the way to go. If there were hundreds or thousands of testing libraries, then package-based architecture might make sense (out of necessity). But there’s really only a dozen or so testing libraries that are relevant. And from what I’ve observed, the broader Meteor community is more interested in a consistent and integrated API than modularity.

A few months ago, I quit working on Velocity and started looking closely at SpaceJam instead, and seeing where that would lead. Because you’re absolutely right: Meteor core is built on TinyTest, and I and my clients need the core package tests for auditing purposes.

In the past few months, I’ve managed to port the core SpaceJam functionality over to StarryNight/Nightwatch, and swapped out the PhantomJS instance with a Chrome browser controlled by Nightwatch. My experience has been that the getLogs() function has been more reliable with Nightwatch than SpaceJam’s Phantom page.onConsoleMessage() function. End result is that TinyTest results are now available via the Nightwatch runner.

Also, for those of you who haven’t been following the latest Nightwatch developments, it now supports client and server side unit-testing, and ships with a basic Chai example. So, between Nightwatch and practicalmeteor:chai, we have Chai as a basic isomorphic unit-testing API across client, server, and packages (when using Nightwatch/TinyTest together). There’s also a verbal commitment from Andrei for adding mocha support to Nightwatch. If Ronen can get mocha integrated with TinyTest, it looks like mocha’s describe/it syntax will be available isomorphically as well.

I’m also looking at porting the TinyTest API to Nightwatch. Probably as default Nightwatch commands that ship with StarryNight. But I’ve been in contact with Andrei; so maybe within Nightwatch core.

With regards to acceptance testing APIs, the most recent version of StarryNight (3.2.61) now provides Nightwatch/Selenium/Webdriver API isomorphically across client, server, and packages. It can do that by scanning the filesystem and autogenerating the nightwatch config file. And with the Nightwatch+Cucumber example, we have a clear plan on how to add BDD style Given/Where/Then syntax across client, server, and packages. So, as far as consistent APIs go, there’s active development in creating such acceptance testing functionality, and it’s pretty much wrapping up at this point.

Anyhow, can’t speak for the Velocity team, but there are folks in the Nightwatch community who are in complete agreement with you, and have been working on building out tools that work to extend TinyTest rather than replace it. We’re totally on board with the idea of a consistent, isomorphic, integrated testing API. Using TinyTest, Mocha, Chai, Selenium Webdriver, and Cucumber syntax, even. But it requires integrating the libraries into something new. Not keeping them isolated and separate via packages.

So yeah. The package based approach, with its focus on unit-testing, and trying to replace TinyTest instead of extend it… it just didn’t work for us or our clients. So we went in a different direction.


#6

@awatson1978, this is a great reply! I had to take the time to read it 5 times over and thoroughly research every aspect before responding.

Nightwatch looks like the silver bullet to all my testing woes. After reading through your StarryNight github repo and source code, it’s looking very much like a “complete” testing solution. It’s refreshing to see people sharing my idea of what testing in Meteor can (should) be: an isomorphic, predictable solution across all borders (client, server, packages).

A couple things, how does testing packages with StarryNight work? You mentioned:

So, between Nightwatch and practicalmeteor:chai, we have Chai as a basic isomorphic unit-testing API across client, server, and packages (when using Nightwatch/TinyTest together).

What do you mean Nightwatch/Tinytest together? Can I write my unit and integration tests within a package using Nightwatch (v0.6) syntax and have them run successfully?

A second concern, it looks like you’re choosing to include lots of different syntaxes and frameworks after reading through some source code. Mocha, Jasmine, Cucumber syntax, etc. This makes me a little anxious. I really think the Meteor community would benefit from a few people saying “this is the right way to test” and laying down some standards, and that means only using a handful of frameworks (in my opinion). I’d love to hear your opinion on this.

Ideally, I’m wanting to include a completely isomorphic-across-test-type testing environment in Meteor Fresh so that I can start telling people who use it, “this is how we unit test, this is how we integration test, this is how we test end-to-end”. That’s my dream :smile:

Will report back after taking StarryNight for a spin.

UPDATE:
Now that I’ve exercised the utility for a bit, I have to say that your documentation is rather scattered. Different resources say different things and I’m a bit confused. I can definitely get it to run Nightwatch acceptance tests, which is great, but I can’t quite decipher the docs to get it to do much else. I tried to follow the examples posted on the website, but most of them don’t work. In fact commands like “starrynight create --package” don’t even set up working packages…the paths in the packaje.js file are incorrect.

It’s also a bit hard to tell from the site what exactly is and isn’t implemented yet. Am I looking at the right site?


#7

There’s a few different questions and subjects going on here. I hope I can clarify a few things

What does “Official Test Runner” mean?
It means the MDG has let us, the community, drive the testing solution for version 1.0. We (the Velocity team) have been experimenting and path-finding new ways to help discover what it means to test Meteor apps.

When we first embarked on this journey, the MDG did not have testing support very high on their roadmap. This fact has not publicly changed and currently only the MDG know their priorities.

So when they gave the Velocity team the torch to do the path finding, they announced us as official solution for 1.0 and said they would help us. This help led us to having the debugOnly flag on packages and also the --test command in the core and they have given us the ability to do what we like in there (within reason of course!).

No other testing framework - mentioned in this thread or elsewhere - currently has this level of support from the MDG and this is why it’s called the “official test runner”.

Will TinyTest be phased out?
No official word from the MDG about this, nor do the Velocity team want this! It’s worth understanding what TinyTest really is.

TinyTest is a very simple MDG-written testing library. It is the default library in the Package.onTest testing mode. It’s a means for testing packages and packages only. If EVERYTHING you build is a package like the MDG, then it might serve you well like it has them. Of course, it’s a framework that lacks any advanced testing capability like mocks and spies and that’s where munit was first to patch Package.onTest and add Mocha to it.

The less known fact is that Velocity frameworks like Jasmine and Mocha already support Package.onTest the same way munit does. Here’s how you use sanjo:jasmine and mike:mocha for package testing.

If you ask my opinion (and this is not the opinion of the MDG at all) then I see the Package.onTest option is here to stay. With munit, mike:mocha and sanjo:jasmine all using this mode, it would be a smart move to extend this testing option to allow more advanced testing of packages. I wouldn’t be worried that Velocity will replace this mode, especially since it’s being used by Velocity compatible frameworks!

What’s the point of Velocity?
If you want to test your app as a whole and not as separate packages. You have two options. Integration or End-to-End.

You can forget about the Package.onTest mode as this does not allow you to access your app. Perhaps there will be version in the future that does that, which is exactly what Velocity mirrors do :slight_smile:

Next you’ll need to:

  1. Create a test environment to run your test against
  2. Start a framework of your choice
  3. Run tests

In point 2, if you want the integration framework to have access to the running app variables then you’ll need to wire it up to do that, making sure to have Meteor.bindEnvironment etc. That’s exactly what sanjo:jasmine and mike:mocha do. They deeply integrate jasmine/mocha into your running Meteor environment.

Now consider that if you do the above on the main running app (default port 3000), then your tests will interfere with the main app every time you change your code. This works if you’re a 100% TDD person and know exactly what you’re doing (I am actually considering supporting a noMirror mode for such users). But if you want to keep the default functionality of Meteor and see changes you make to the code live without tests interfering, you need to have a separate test environment. This is what Velocity does with mirrors.

Velocity also takes care of parts 1 and 3 for you at development time, and on CI.

Finally, Velocity provides a common reporting API. We’re still working on this, and the aim is to have a single consolidated report from your package, integration and end-to-end tests.

So what’s the problem?
The problem is less about the technical solution that we have and more about documentation to educate users. As such, we’re in the process of redesigning the velocity website and have focussed our efforts on centralizing the documentation. So instead of going to GitHub pages for individual frameworks, you now go to Velocity site where you get guidance on how to use the frameworks.

On another note, you’ll find the issues are usually with the frameworks and not Velocity itself. In fact, in the past month only 1 bug has been filed with the Velocity core. If you want to be sure you get the tip-top support, then you should use Jasmine as that’s the recommended and most supported Velocity-compatible Framework.

Final Thoughts
Velocity fills a huge gap in the Meteor testing space that other solutions don’t come close to touching. It’s still version 0.7.x and the more issues that frustrated people file, the sooner we can get it there. I’m always willing to remotely pair with users if they hit a genuine Velocity/Cucumber bug that they’re unable to resolve.

I hope to have shed some light on the current state of play for you and for anyone else that reads this thread.


#8

@sam This is enlightening. This support from the MDG is really great, but I kinda wish they would give you guys a little extra “oomph” and move testing up the priority scale.

The less known fact is that Velocity frameworks like Jasmine and Mocha already support Package.onTest the same way munit does. Here’s how you use sanjo:jasmine and mike:mocha for package testing.

This is wonderful! I just got Jasmine package tests working in 2 seconds. Before I was trying to shoehorn Jasmine into testing my packages from outside the packages directory, being an advocate of the all-package approach this felt “wrong” in that I couldn’t isolate my package under test. I think this should work wonderfully in Meteor Fresh coupled with an acceptance test runner, possibly even running simultaneously on a separate port from the package tests.

If you want to be sure you get the tip-top support, then you should use Jasmine as that’s the recommended and most supported Velocity-compatible Framework.

Do you have a similar acceptance test runner that you can recommend?

Thanks for your reply.


#9

Glad it helped! I wish we had 3 full-timers on Velocity!

xolvio:cucumber is the e2e framework to use. I made it so I’m biased, but it has gone through many iterations of real world users. You should give the tutorial mentioned above a shot if you haven’t already and as always, just file an issue on the github repo and I’ll be glad to help.

The thing that will help you is the @dev tag. It will start a mirror but only rerun tests with that tag.

I recommend you look at outside-in testing techniques using cucumber. You basically start with a feature definition, then automate at a feature level (happy path) then you drive the development of all related packages using jasmine. This will give you feature level coverage plus package level - and all of it in a just-in-time fashion.


#10

How does testing packages with StarryNight work?

There are two ways we’re experimenting with. The first is a scrape of the TinyTest runner, which is the same method that SpaceJam uses. TinyTest runs the package tests, but reports it to the browser console, which Nightwatch then reads.

The second method uses Nightwatch to do the unit testing for packages, and involves building a config file with the necessary directory paths in it. Building the config file is Meteor specific, and slightly outside the purview of Nightwatch (which is a Node end-to-end runner), which is what the StarryNight utility is for.

What do you mean Nightwatch/Tinytest together?

starrynight run-tests --framework tinytest-ci

There’s also experimental support for running both Nightwatch and TinyTest in the same command. If you want to tinker with running them serially vs in parallel, the ‘multi’ framework is for that. Right now it’s disabled, but it’s in active development, so that’s liable to change at any time.

starrynight run-tests --framework multi

Can I write my unit and integration tests within a package using Nightwatch (v0.6) syntax and have them run successfully?

Right now, you can write unit tests using Nightwatch syntax within the package, but not integration tests. This requires the StarryNight --autogenerate and --autogenerated flags. (In time, we’ll link that process together so it’s fully automated.)

For integration tests, you would need to add the --framework tinytest-ci argument, and pick up the TinyTest results from the console browser and report them via Nightwatch.

It looks like you’re choosing to include lots of different syntaxes and frameworks after reading through some source code. Mocha, Jasmine, Cucumber syntax, etc. This makes me a little anxious.

Yes, the intent is a complete integration of the different syntaxes. They each serve a different purpose, and from the discussions I’ve had with my clients, there’s a need to have them all in one place without multiple packages. Now, keep in mind that most libraries only introduce a few novel syntaxes. describe/it syntax in Mocha; Given/Where/Then syntax in Cucumber; should.be.foo syntax in Chai Expect; verify.foo() method chaining in Nightwatch, etc. Each syntax is optimized for a different use: Mocha is great for unit tests, Cucumber for behavior-driven development, Nightwatch for test-driven-development, and so forth.

Beyond the syntactical differences, the remaining functionality in the various frameworks is a mish-mash of overlapping functionality. People reinventing the wheel with regard to runners and reporting. And that’s where the bulk of the overhead comes from.

I’ve been in contact with Andrei, and he’s expressed similar anxiety about including Mocha and Cucumber directly. Nightwatch already has a runner and a reporter, so why add Mocha’s? It just creates redundancy in the code base. He proposed simply adding the describe/it syntax, and the Given/Where/Then syntax directly to Nightwatch. Which would provide the core API syntax that people find useful, but do it via a tight integration, and without the overhead of multiple libraries.

The idea right now seems to be that Mocha and Cucumber syntax will get precompiled down to Nightwatch’s TDD syntax, in a similar way to how CoffeeScript gets compiled to Javascript, Stylus/Less gets compiled to CSS, and Jade gets compiled to HTML.

Now that I’ve exercised the utility for a bit, I have to say that your documentation is rather scattered.

For sure. StarryNight is considerably newer than Velocity. We’re currently making workflow diagrams, more architecture diagrams, and many more tutorials (with screenshots). These past two weeks I’ve been on-site at two clients who are underwriting this development, which has slowed getting documentation out the door. But it’s on it’s way.

In fact commands like “starrynight create --package” don’t even set up working packages…the paths in the packaje.js file are incorrect.

This is an interesting point. As mentioned above, MDG ‘handed the torch’ to Velocity, and has only substantially contributed help via the ‘onDebug’ flag and ‘–test’ flag. For my client’s needs, this is woefully insufficient for the kind of work we want to do. We have dozens of commands and functions related to testing we wanted to see in the meteor tool and on the command line. Not just one or two. The most recent big-ticket item that came directly from a client was package refactoring from an existing component. Basically, they wanted this:

meteor create --package --from /path/to/component

But it’s fairly impossible to submit such functionality via the issue tracker or pull requests. We’ve tried issues, pull-requests, following Velocity architectural recommendations, contacting MDG members, and have gotten nowhere. So, we basically cloned the core structure of the meteor tool, renamed it to starrynight, and began implementing extensions we wished were in the meteor tool ourselves. So, the reason that starrynight create --package doesn’t work, is because that functionality is available via meteor create --package. Could/should that have been documented? Perhaps.

Alternatively, could we add pass-through command functionality so that it works more seamlessly? For sure. We recently implemented that for Nightwatch with argument arrays, so we have the pattern, and it would be relatively simple to implement for the Meteor command as well.

It’s also a bit hard to tell from the site what exactly is and isn’t implemented yet.

That’s fair. As I mentioned, I’ve got sponsoring from three clients right now to develop this technology and am working on it pretty much full-time. It’s very much in active development. And there’s a bunch of documentation in the pipeline. There’s just not enough time in the day to get it done when everybody is asking me to fly around for on-site consultations. :wink:

It’s early yet; but we have a clear vision of where we’re going. Isomorphic API, extending TinyTest with Selenium via Nightwatch, tight integration rather than a package based approach; command line tools.


#11

Thanks for the reply. I really believe StarryNight is a healthy project for the testing community, and you can bet I’ll be following it closely. I’ve subscribed on Github to watch for further development :wink:


#12

In case you missed the latest announcement from Meteor. See this line:

"Testing improvements. We plan to move Velocity into core and simplify and streamline it" :whale:

I think this should sufficiently answer what “official test runner” means


#13

@sam Will the unit tests get any faster? It seems like the reload speed of Meteor is limiting the speed for client tests? (i’m using Jasmine).

For React component unit testing I ended up using a separate karma in the tests/karma folder running outside of Velocity and it’s much faster. Using Velocity Jasmine client unit, it was taking 5 seconds to run 1 or 2 unit tests (from save to result). The same is almost instant with the default karma.
I hope it gets faster in the future, def. willing to drop the external karma!

FWIW, I feel like the integration test speed is acceptable :smiley:

Currently my testing setup is:

  • Integration tests: Velocity Jasmine
  • Package: Velocity Jasmine
  • Unit: External Karma/Babel/Jasmine
  • Acceptance: Capybara with Rspec (using Meteor Capybara)

So far it’s been working out really great. Until we get async-await I don’t think I can switch to JS based acceptance testing (personal pref. of course).


#14

Unit testing will be so much easier as soon as we have ES6 modules. This will allow us to load code and easily isolate it

With regards to acceptance testing, have you seen the chai-as-promised integration in xolvio:cucumber? I’m not familiar with async-await but I think that’s what you mean?

WebdriverIO 3.0 just added real Promise/A+ support. Combined with chai-as-promised this gives a VERY fluent syntax.

I’m interested in your usecase, but that should probably go under another thread.


#15

@sam : About Velocity and package testing, is it possible to have the meteor application running, and all the packages unit tests running at the same time (after each reload due to code modification), like it does for application unit tests ?
This is something that cannot be done with TinyTest, you have to switch back and forth between “meteor” and “meteor test-packages”, and for me that would be a reason to switch to velocity for packages unit testing.


#16

Yes, you can. But I think you can also do it with Tinytest. You just have to define another port. Like meteor test-packages --port 5000. Let me know if I miss something.


#17

I didn’t knew that ! Thanks.


#18

Ah that makes sense! I can’t wait! Thanks for clearing this up.


[quote="sam, post:14, topic:5709"] With regards to acceptance testing, have you seen the chai-as-promised integration in xolvio:cucumber? I'm not familiar with async-await but I think that's what you mean? [/quote]

I’ll check it out! When I used CasperJS or WebDriverJS a few years ago it was like pulling teeth and callback hell.

Synchronous Ruby helps make it as painless and terse as possible. I’m not sure if ES6 Async - Await is even on the roadmap for those libs but it seems like it would help make it read better (although the promises in chai-as-promised doesn’t look too bad).


[quote="sam, post:14, topic:5709"] I'm interested in your usecase, but that should probably go under another thread. [/quote] Let me know! Feel free to PM or email me.

#19

I’m working on running a synchronous version of WebDriver.io in Fibers. The same way as you write Meteor server code. :smile:


#20

Whooaaa… You da best!