Xolvio:cucumber - Latest version breaks a my understanding of it

testing
1

Hi everyone,

I have a nice accumulated quite a few tests for my current project and i am happily relying on them by now.
Sadly the newer version of xolvio:cucumber (v0.11.x and v0.12.x) break my tests in a weird way that i can’t wrap my head around, so i need some help to migrate my tests to the newer version.
Right now i am running in to false positives which is very scary.

The main thing i don’t understand is, why my feature steps are showing up out of order in my cucumber.log.
I think if i could understand why that is happening i will be able to fix my tests. I’ve already spent more then a day on figuring it out, i know it has somehow be related to the promise style syntax/logic in all this but I’m kind of new to that.

I’ve read in the change-log that the Webdriver.io API was updated to v3 and found in some discussions that this brought major changes. But i can’t seem to find any documentation on how to handle this. I feel like most of the cucumber tutorials are not up to date with the speed of development…

So for now I’ll have to stay with the old v0.10.0 of the xolvio:cucumber package.

Let’s jump into a small example I’ve stripped out of my real world project:

I have the following small feature making sure my background layer on my leaflet map is loading:

790×248 47 KB

If I break my feature intentionally this is how cucumber reports the failure:

738×299 98 KB

When i run the exact same test with the newer versions of xolvio:cucumber, here is what happens:

705×279 65.6 KB

Now you surely want to see my step definitions:

  @Given /^I view the map$/, (next) ->
    @client.url(process.env.ROOT_URL)
      .waitForVisible('.map')
      .call next

  @Then /^I should see the background tiles loading$/, (next) ->
    @client.waitForVisible('img.leaflet-tile-loaded')
      .call next

The steps are written in coffescript.

If i start executing more tests so it takes longer after some time the

I should see the background tiles loading

step fails eventually but still step execution is totally out of order and more steps then actually available/set-up are executed.

I’ve also tried these variants of the steps with no success:

  @Then /^I should see the background tiles loading$/, (next) ->
    @client.isVisible('img.leaflet-tile-loaded')
    .should.become(true)
    .and.notify next

  @Then /^I should see the background tiles loading$/, (next) ->
    @client.isVisible('img.leaflet-tile-loaded')
    .should.eventually.be.true
    .and.notify next

  @Given /^I view the map$/, (next) ->
    @client.url(process.env.ROOT_URL)
      .isVisible('.map')
      .should.eventually.be.true
      .and.notify next

Hope to get some decent hints on how to understand the situation!
Thanks for your time!

2

I’m going to guess that the issue here is that coffeescript has implicit returns, which can play havoc with promises.

Try this:

  @Given /^I view the map$/, () ->
    @client.url(process.env.ROOT_URL)
      .waitForVisible('.map')

Jonas and I have talked a lot about the problems people are having with callbacks and promises, so we’ve made a new version of Chimp (and therefore meteor-cucumber) that uses fibers like Meteor, so you have 0 callbacks and 0 promises. It’s on a branch that Jonas finished last night so fingers crossed, we’ll have it ready for release in the next week or two. Once we fiberize webdriver + ddp calls within steps, these issues will evaporate!

Thank you for bearing with us and let me know how the above works

3

Hey Sam,

thanks for the immediate reply and the good news about adding fibers support to chimp!

Although your suggestions seems to fix the first step, it still really hard to find a statement that behaves reliably and predictably for the second step. Also i did not test the case where the ‘.map’ element would not become visible since it is more about the timing of the components and not an actual test.

I also seem to get an error in the consecutive runs of cucumber. Somehow my leaflet map breaks, although it’s working the first time cumber runs, after I startup meteor. That’s also not happening on the v0.10.0 release.

I think for know I’ll wait for your fibre version of chimp. But I’d be eager to try an development release of it.

4

which browser are you using?

5

I assume you mean for Selenium? Most of the time I use Chrome, although I’ve tried the other options.
I’ve tried the new release from August 2nd, but that does not include the fiber version of chimp yet, right? Still that version seems to be working a little better for me.