Why use Cucumber AND Jasmine?

pobrien · July 17, 2015, 1:38pm

I’m playing around with Velocity and the various testing frameworks, but feel I’m missing something when it comes to the meteor testing manual recommendation of using cucumber and jasmine together. Looking at the 7 testing modes of Meteor Jasmine seems to offer everything.

In Cucumber I might define a scenario like

Scenario: New doge goes to home page
  Given I am a new doge
  When I navigate to "/"
  Then I should see the title "such title"

but, couldn’t I just do that in a Jasmine client integration test

describe("home", function(){
  describe("template", function(){
    it("shows 'such title' title", function(){
      expect($('title').text()).toEqual('such title');
    });
  });
});

Is the only difference that Cucumber allows me to approach the test with conditions lick “as a new user”? It goes without saying that the recommendation to use Cucumber & Jasmine together comes from people with MUCH greater knowledge than me, so I’m just trying to see what I’m missing.

UPDATE

Having just read around a bit more, I think I’m seeing where these frameworks fit together. This idea of ‘outside in’ testing, mentioned by Josh Owens here and a nice blog post here (albeit about Rails rather than Meteor).

Sanjo · July 17, 2015, 4:38pm

Cucumber is a tool that allows you to do Specification by Example. You can find the key ideas here: http://specificationbyexample.com/key_ideas.html. The book is also a good read. Because Specification by Example involves all people that are involved in the project, Cucumber separates the textual specification that everyone can read and write from the technical automation concerns.

If you don’t need that you can just use Jasmine to write end-to-end tests with Webdriver.io. Right now the support is not that first class. But I have it on my roadmap.

joshowens · July 19, 2015, 2:19pm

For me, I write tests in Cucumber because it forces me to think in plain language about what the user can see and do. As far as end-to-end testing goes, I’ve seen a number of examples where I developer might create an interface that allows user input, but then they just check the database to make sure it got there. Your user can’t check the database, they need an interface to see/retrieve their input when they need it.

Similarly, I think there are times when you need to get more complex with code and you just want to test that small piece of it. Perhaps you fetch data from twitter and do some complex calculations on it, and then save that to the db for later use in your UI. I find Jasmine can really shine in helping you write unit tests or integration tests that prove your code works how you want it to.

I guess for me, you can do Cucumber with the project stakeholders and you can do jasmine on your own.

ryanswapp · July 19, 2015, 4:42pm

@joshowens I’ve used Cucumber a little bit and it seems pretty straightforward. However, it took me FOREVER to get my tests working and they were super brittle. I had to keep playing around with the millisecond parameter for the webdriver waitForVisible and waitForExist methods in order to get them working (I also had to mess around with the screen width so that certain elements on the page were visible to the webdriver). In addition, when I added more tests my app slowed down significantly (I guess this pain point has been partially alleviated with the @dev indicator). I’m curious as to whether you run into these issues as well? Or if I’m just doing it wrong.

joshowens · July 19, 2015, 6:08pm

Have you tried 0.11 or higher? @sam has switched Cucumber to use Chrome as the default driver and it seems much better now.

sam · July 19, 2015, 7:25pm

All great responses. I’ll chip in my 2c

Doing BDD is a good way to ensure your team(s) can scale as it can help you communicate effectively, save time on rework and it involves everyone from designers to developers.

There are two main problems people face when doing BDD. They are:

People don’t know the difference between specs and tests. Typically this ends up with developers getting annoyed that free text is not refactorable and they abandon ship!
There are so many moving parts in doing end-to-end automation. A developer typically has to start a selenium server, connect some webdriver bindings, manage sessions, setup some test data, wait for the right conditions to happen, and so on. When something goes wrong they find the whole experience brittle and they abandon ship!

xolvio:cucumber (which uses our chimpjs under the hood) aims to solve problem 2. It deals with all the problematic and mundane aspect of setup and wiring, and allows you to get on with the interesting automation parts. It’s been a rocky road trying understand it all and through deliberation and constant user feedback, we’ve improving and steadily achieving this goal and is now better than ever. Here’s what it does today:

The @dev tag as you mentioned has been a huge win for performance
You have a button in the reporter than runs the whole suite when you want
Using Chrome as the default browser has made tests a ton more stable (phantom is now for the brave)
Full CI support with automatic screenshot grabbing on failures
We have created a fully tested app that runs on CI with awesome real feedback. Check out Letterpress
Chimp has very good process management so the perf issues related to hanging processes have been dealt with
The browser sessions are reused now using a Session manager so you get ultra quick response when you save a file
We have made a ton of Meteor stubs packages to ease the automation for things like email, Github/Twitter OAuth and even Stripe
We promisified everything so that you don’t have nested callback hell. Still, this area is not 100% as we like it (See fibers below)
We have created mechanisms and patterns for setting up and managing test data

On the roadmap we have:

Fiber based calls to Webdriver and server DDP from within steps so you can do synchronous style coding like Meteor
We have an experimental parallel testing mode that currently requires big hardware, but we’re have a streamlined version planned
Automatically generating boilerplate code for undefined steps
Any other problems we find whilst we’re using it / hearing from others

We are also working on solving problem 1 with Simian.io to help people write better specs and collaborate with others. xolvio:cucumber will soon send results to Simian so you can see and share progress of your feature completion.

As you can see, we’re in love with this problem doing BDD, writing good specs and automating the world and we’re not stopping any time soon. The more than join this party and give feedback, the better this party will get.

It may not be everyone’s cup of tea, but if it is then we are sure serving the best out there!

ryanswapp · July 20, 2015, 3:42pm

@joshowens I think that I was running 0.7 or 0.8 the last time I really tried to get it working in an app. It was right around the switch to the this.server and this.client syntax. From what Sam just explained, it sounds like it’s improved substantially since then! I’ll have to give it another go.

@sam Thanks for the detailed explanation! I’m grateful for your and the velocity team’s efforts to make the Meteor testing story better and better. It sounds to me like the most recent changes have made the xolvio:cucumber package a lot easier to work with so I’ll be sure to give it another shot.