You did not mention Velocity in your reply. Is Velocity still the “Official Testing Framework”, as announced here:
http://xolv.io/blog-posts/2015/11/8/xkybpw2cdvg6ytncjbh7hmvu5f9jbw
and Chimp is “just” your recommendation for a certain type of testing (i.e. ‘end-to-end-testinng’)?
Or did MDG change their plans in the meantime? Sorry, but I am a bit confused, because the OP asked about the future of Velocity in particular.
My answer has multiple parts, and it goes like this:
I believe our plan is to build/recommend just enough for Meteor 1.3 that it covers all of the requirements for testing an app - unit tests, integration tests, and end-to-end tests for every part of a relatively normal Meteor app.
On a personal level, here’s my thought: Velocity had very ambitious goals like connecting a variety of different test runners and reporters, reactivity for all parts of the system, merging the results of all kinds of tests into one output, and more. I think if it had more limited goals it would have been easier to build and maintain. That’s why I think Sam’s new project Chimp is great - it looks like it is focusing on getting just one part of the testing story right, and in a way that is minimally Meteor-specific.
Well, as chimp is E2E testing, what else you expect it to do than execute calls from client/server Meteor context and simulating browser ?
For unit and integration testing, velocity-cli way was best working for us.
Wow, that was fast. Thanks for your quick and detailed reply!
Looking forward to reading the testing guide, especially the part about unit and integration testing, as this is what I am currently needing most.
TBH, I am using no testing at all at the moment, but I am feeling quite uncomfortable with this situation. Refactoring is always a challenge if you don’t have any tests. I tried Velocity when I was still on Meteor 1.1, but I could not get it work, so I abandoned it. Now I wanted to give it a second chance, as I am on 1.2 now; just to learn that Velocity has been dropped in the meantime. Guess Meteor is just too fast for me…
Chimp looks nice, but as far as I can tell it is not as tightly integrated with Meteor as Velocity was. But maybe I just have to dig a bit deeper into it.
Yes, you’re right. I thought Chimp was some kind of successor to Velocity; but I now learned that this is not the case. I’ve never tried the CLI version of Velocity so far, but maybe this will work. I am quite curious what @tmeasday will eventually recommend in his testing guide - and can’t wait to read it.
It is still unclear what tools are available to unit-test an app. Velocity was great with that. Does anybody know a good unit-testing tool (app level, client + server) for Meteor?
Before Meteor 1.3, I think the best thing is to split your app into packages and use test-packages. That’s what we are doing so far in the Todos example app we’re building for the guide: https://github.com/meteor/todos/blob/master/packages/lists/lists-tests.js
In Meteor 1.3 I think it will be easier to do this without going through the hassle of building Meteor packages, since you will be able to unit test individual ES2015 modules. Before we release the guide with 1.3 we’ll migrate the Todos app to that structure, so there will be a good set of example tests to look at.
We’re also writing a whole ton of tests internally for the Galaxy UI, which won’t be open source but will be a good testbed for the new testing features we’re working on for 1.3.
With end to end testing you will test the application at the user interface level typically by having a robot fill in inputs and press buttons and then it asserts that the application state has changed (such as text on the page)
Typically this will test all layers of the application including the database so it’s very close to what a real user is experiencing.
Most end to end frameworks don’t have any knowledge of the web framework you’re using but some include hooks to setup and tear down the app so that its in an expected state.
I was afraid of the test-packages answer …
Do you mean we’ll have to write unit-tests inside our own modules? How will the tests be automatically run? And what if we don’t use use modules? I’m even more confused, could you give us a short example?
A test-driven Todos-app tutorial (just like in Phoenix) would be really great by the way 
If you don’t use modules in 1.3 then you’ll have to test the same way you do today.
However, if you use 1.3 modules (and you should), you can use any JavaScript library you’d like to unit-test your module. You can just write a test file and in that file import your Meteor module, then you can stub/mock out any dependancies outside of the module as needed. Test runners like Jasmine, Mocha, Tape, etc… have watchers that can run in the background and will finish in milliseconds.
I’m planning on recording some screencasts migrating a 1.2 project (react-ive meteor) to 1.3 while adding unit tests, end-to-end tests and a few integration tests. Stay tuned!
I’ve just found in this one month old post that there are lots of great ideas being implemented (well, I hope so).
Also this very interesting article is worth reading about the importance of unit-tests “vs” end-to-end tests:
http://www.alwaysagileconsulting.com/articles/end-to-end-testing-considered-harmful/
This is spot on, UI level testing is harmful. Most e2e frameworks try to focus too much on the UI and end up building a huge pile of technical debt. These tests look more like test-scripts rather than executable specifications. Test scripts come after the coding as an afterthought and almost always end up getting disabled because they go out of date and nobody wants to update them. Executable specifications on the other hand are a development level technique and drive development. The latter is a long lived practice that is self maintaining in committed teams.
I strongly recommend you watch this presentation that shows some pioneering work on Modeling by Example. This technique allows you to apply the exact right number of tests at the e2e, acceptance and unit levels (it can be done with Mocha/Jasmine not just Cucumber).
Another good read on this subject is Google’s testing blog post on end-to-end testing where they say
Google often suggests a 70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests. The exact mix will be different for each team, but in general, it should retain that pyramid shape.
We have built Chimp on the exact principles of the testing triangle and it currently allows you run e2e and acceptance (integration) tests.
For unit testing, then @SkinnyGeek1010 is absolutely right with this statement:
Unfortunately even with a fresh skillsmatter account I get “Because of its privacy settings, this video cannot be played here.”
Sounds great 
I’m really looking forward to seeing this because I’m not sure how to do unit testing with Mocha outside the dying Velocity.
Meteor (with CoffeeScript) made me switch from the Python/Django universe to the Javascript universe. Modules are everywhere in Python, so are unit-tests (assert is a Python core keyword).
But in this new Javascript universe I’m still wondering what way I should remix my Metor apps into modules. I can’t wait to see the following in the Todos app: [quote=“sashko, post:30, topic:15084”]
you will be able to unit test individual ES2015 modules. Before we release the guide with 1.3 we’ll migrate the Todos app to that structure, so there will be a good set of example tests to look at.
[/quote]
[quote=“awatson1978, post:17, topic:15084”]
Typical. This kind of language is what drives women away from the tech industry.
[/quote] I lacks words… Why is this necessary?
So I guess I am looking for a unit testing framework. Sorry, but I am not too deep into these testing terms 
Relax. I also did not get this. 
who say’s I’m not relaxed? and I totally get it. I just think this kind of ranting about gender/tech belongs in it’s own post, as it has nothing to do with the stated topic.
to get back to the subject - What do you guys think Meteor will advocate for their testing section in the Meteor guide?
Searching for .mp4 in the source and pasting that link in a new window solved it here.
(There’s a related blog post here.)
I’ve just started using Chimp with Cucumber and WebDriverIO (Selenium) for a few complicated, automated UX tests. Works great for that and was very quick to set up even though I only have (basic) experience with Mocha (using mike:mocha(-package)).
With just those few tests in place, I’m already wanting to decouple test logic as much as possible, to reuse within that and other testing solutions.
On the value and scope of testing (i.e. pyramid structure), I think everyone agrees it’s very context-dependent. Maybe 70/20/10 is a good starting point until your project informs adjustment. Here’s a timed link to a lucid, mere mortal summary of the DHH, Martin Fowler and Kent Beck hangouts, which touches on some of that.
Hmmm… I must be the only one whose sarcasm detector was going off the charts.
edit: I stand corrected 
This is the same video here: https://vimeo.com/149564297 - it’s very similar to the blog post, but it shows a LOT more detail of how to do it in reality.
Emily is absolutely right and has a better idea on the problem than DHH does in my opinion. I really like the TextTest tool she created. You can achieve some very similar results by using subcutaneous testing to capture the data that will be used by the UI, instead of testing the UI itself. This can’t work if you put a ton of logic into the view, however if you write your code using UI components, this is awesome as you can unit test the UI components like crazy (with something like React Test for example). Subcutaneous testing draws the boundary above units and below the UI therefore they are considered integration / acceptance tests, without the penalty of being sensitive to UI changes.
Emily was also right in saying that the education needs to happen both with novice and advanced developers to show how acceptance testing is properly done and how it’s different to end-to-end testing.
Just my 2c!