Velocity is the official test runner of Meteor. What does that mean exactly?

rjakobsson · August 7, 2015, 2:14pm

I would love it if Velocity would ship out of the box with all of the example apps, as it would prioritize the developer experience with Velocity.

waldgeist · September 26, 2015, 6:48pm

Tried to set-up Velocity yesterday – and finally gave up. In principle, it worked, but it did not accept any of my ES6 classes. Seems as if it ignored my package dependencies. Also, I did not get my package tests to show up in the Velocity console. The documentation only refers to a command line mode. When I tried this command line, it just did not show anything. No error message, no progress.

My personal conclusion was: Seems to be a very nice package, but really lacks good documentation. If the goal is to sell the testing book - well, having such a scarce documentation might be a valid choice. But if the Velocity team wanted to gain more attraction for the framework, they should add a bit more sample code and tutorial stuff.

kookoskar · September 26, 2015, 9:57pm

@waldgeist Were you using the velocity CLI tool? Did you install it with the correct permissions (e.g. if you installed it with sudo, but tried to run the tool without sudo, you’d experience the case where nothing happens).

The testing book itself sucks. A lot. Do not buy it, it’s tantamount to throwing your money away. I still do not understand how anyone thinks it’s OK to advertise it so much when the book itself has only a few chapters that cover the absolute basics of testing; it barely even touches testing anything in Meteor (which is why you’d buy it in the first place).

csauer · September 26, 2015, 10:41pm

@kookoskar what is the difference between using the CLI tool, and the velocity Meteor package with the HTML reporter?

timfletcher · September 27, 2015, 12:11am

Thanks for the review. I’d been on the fence about buying it but all the interesting chapters on testing specifics e.g. Methods / publications are yet to be written and it’s been like that for the 10 months or so that I’ve been working with meteor.

I’ve managed to glean enough from various sources to write reasonable tests. Velocity’s sample Letterpress app I’ve found good.

pcastrom · September 27, 2015, 12:13am

I gotta agree with @waldgeist, I actually bought the Meteor testing book almost a year ago, I knew I was getting a beta book, but I couldn’t imagine that after almost a year I would still see a lot Not Ready chapters (the ones i was most interested about…). Nevertherless I started using velocity and mike:mocha on two proyects, unfortunately it hasn’t been pleasent at all, let me explain some of it:

We wanted to focus on client-server integration, so we focused on testing meteor methods (one of the not ready chapters from the book) but our tests run and pass most of the times, but we sometimes get errors, particularly when making changes quickly, meteor restarts two or three times, and now our tests are red and can’t seem to recover unless we stop and restart the meteor process again.
While mike:mocha seemed like a good choice, it now feels like jasmine and cucumber are getting more attention from the velocity developers (just my impression) did I pick the wrong one?
A couple of times we even had inconsistent results using meteor --test which again, could be my fault, but we haven’t found anything any good reason to believe so, since triggering a new build withouth any changes fixed the problem.

Bottom line, I think we should be able to fully trust our tests, if our tests gives us false negatives, then it’s even worse than not testing at all. I honestly feel dissapointed on the Meteor Testing book’s progress and now that packages based apps are getting more attention, I’m thinking that Tinytest may be more than enough for most testing needs.

sam · September 27, 2015, 9:55am

As the author of the book and a core Velocity member, I have to respond to all this!

You’re all absolutely right to call out the lack of updates on the book. I recognize this acutely and I’m working on the chapters as we speak. The order they are coming out in is as follows:

Test Types
Testing Best Practices is being renamed to “How to Structure your App for Testing”
Testing Meteor Methods
Testing Collections
Testing Packages

I also owe everyone a reason and I will be sending this out in the next update (which I promise is coming in the next week or so). The reason for the lack of updates is that Jonas and I are the only two active people left on the Velocity team. We have the following responsibilities:

Maintain & Support Velocity for free

Maintain & Support Meteor Jasmine for free
Maintain & Support Meteor Cucumber for free
Develop our paid service Simian.io
Do consulting work to pay our bills
Write the book

We would absolutely love to work full time on Velocity and related frameworks, and it really is a full time job. The reality is that we have to juggle priorities and lately we’ve had to the prioritize points above point 6.

@waldgeist Did you post an issue about your ES6 classes not working? They should work no dramas in Jasmine. Cucumber support is coming soon.

@csauer the velocity-cli npm tool is a shortcut so you don’t have to write crazy long commands for things like package testing.

@timfletcher Letterpress was developed for the book not Velocity. It’s well ahead of the book as I tried to solve all problems with it prior writing content. It’s still acting as the path finder in that way and is always a good resource to go see the latest development.

@pcastrom you would have better luck with jasmine for sure. We use it every day for our own product and for client work.

I’m really grateful that you’ve all taken the time to write these comments and I genuinely take the above as constructive criticism. Ultimately you want better content and I owe it to you so stay tuned and if you find issues with Velocity / frameworks, please post them on Github. We do jump on any issues that are blockers quite quickly.

hwillson · September 27, 2015, 12:39pm

Community take note - this is how you respond to criticism. Well done @sam - and thanks for your hard work on all of this!

kookoskar · September 27, 2015, 2:02pm

@sam Yup, good response. Although I still don’t think it’s fair to advertise the book in the manner it is being advertised if it’s only half-done. It’s advertised like it’s the book that’s supposed to solve all your problems with testing, but in reality… you get very little out of it (or, I should say, I did). Either add disclaimers or don’t advertise it at all, otherwise people are bound to end up disappointed.

sam · September 27, 2015, 5:24pm

@kookoskar noted and a valid point. I’ll update all the blurb on the places the book is advertised.

On the getting something out of it aspect. There are people that come to the book knowing nothing, I’ve heard directly that they get a lot of value as all the chapters are currently aimed at beginners. The experienced folk that I talk to (I’m assuming you are) are waiting for the more advanced stuff. I’ve also had to delay that prior to 1.0 due to changing APIs, and then again in 1.2 and it looks like there will be some significant changes coming soon with ES6 modules on the horizon.

That being said, there are definitely things to write about and like I said, I’m working on it right after this thread post so expect something very soon!

rjakobsson · September 30, 2015, 7:50am

I want to mention that Sam and Jonas now offer premium support for Velocity as well:
http://xolv.io/services/velocity-premium-support/

I wish that Velocity would be counted as a core part of the Meteor experience, and that no new version of Meteor would be considered whole if Velocity wasn’t ready for it yet.

warehouseman · October 1, 2015, 5:20pm

I’d add, in your defence, Sam, an additional point to your responsibilities :

7 - Try to keep up with the accelerating stream of changes and improvements in the whole ecosystem!

awatson1978 · October 1, 2015, 5:39pm

Velocity architecture is very unsuitable to many types of testing, and has cost us tens of thousands of dollars of development time with its architectural assumptions and project management. It’s like announcing Angular integration when other people are using React. Sure, let’s get testing frameworks into the platform, but mUnit/SpaceJam and Nightwatch/StarryNight should be considered also.

sam · October 1, 2015, 7:40pm

Thank you @warehouseman.

In the interest of making sure new comers to this thread that see this message, some clarification is in order:

This is a statement shows a misunderstanding of how Velocity works. Velocity has current support for the following:

Unit Testing
- Jasmine supports this for both client and server
Integration Testing
- Jasmine supports this for both client and server
- Mocha supports this for both client and server
End-to-end Testing
- The Jasmine currently supports this
- Cucumber supports this
- Robot supports this
- CasperJS supports this
Package Testing
- Jasmine supports this for both client and server
- Mocha supports this for both client and server

Velocity is suitable for all the testing types above and works today.

This is true, it has cost us all a lot of time and we continue to contribute and give this for free to the Meteor open-source community.

Yes, they should be considered. Here are some facts to support people’s decisions:

mUnit/SpaceJam : Built onTinyTest and supports package testing. It works great, especially when used with Sinon. If that’s your preference and it works for you, you should absolutely use it.

Nightwatch/StarryNight: An opinionated and ambitious project that tries to solve scaffolding, testing and system architecture domains all at once by giving you a command line interface. If that’s your thing, you should absolutely try it.

###Some Facts
Velocity and related frameworks (especially Jasmine & Cucumber) have been battle-tested and are being used in production by some of the biggest Meteor applications around today that power real businesses. Here are the supporting stats:

Issues closed on GitHub that were not created by the author

Velocity Core: 251 tickets - (downloads not relevant as it’s a core package used by the frameworks below)
Jasmine: 179 tickets - 132,000 downloads on atmosphere
Cucumber: 136 tickets - 71,000 downloads on Atmosphere
Mocha: 94 tickets - 99,000 downloads on Atmosphere

spaceJam: 14 tickets - 19,000 downloads on npm
StarryNight: 4 tickets - 3,600 downloads on npm

I hope this data makes it easier for readers of this thread to see the landscape a little better.

Happy testing (then coding)

Sparser · October 1, 2015, 8:36pm

Velocity is working great for us, now.

At first we ran into all sorts of trouble. It turned out to be the proxy settings we use were interfering with it.

warehouseman · October 1, 2015, 9:37pm

Sam,

Where can I learn about using Velocity in continuous integration; eg, in CircleCI or Travis?

sam · October 1, 2015, 10:01pm

Try this for Circle

github.com

xolvio/Letterpress/blob/master/circle.yml

machine:

  java:
    version: oraclejdk8

  environment:
    # Enable this to debug builds
    #VELOCITY_DEBUG: true
    SETTINGS_FILE: "../environments/build/settings.json"
    STRIPE_SECRET_KEY: $TEST_STRIPE_SECRET_KEY

dependencies:

  cache_directories:
    - "~/.meteor"
    - "~/.npm"
    - "~/_opt"

  override:

This file has been truncated. show original

I’m working on Travis for a client and I will share when I’m done if they agree

warehouseman · October 1, 2015, 10:10pm

Great thanks!

(Quite a bit of head busting went into that, I’ll bet)