How does testing packages with StarryNight work?
There are two ways we’re experimenting with. The first is a scrape of the TinyTest runner, which is the same method that SpaceJam uses. TinyTest runs the package tests, but reports it to the browser console, which Nightwatch then reads.
The second method uses Nightwatch to do the unit testing for packages, and involves building a config file with the necessary directory paths in it. Building the config file is Meteor specific, and slightly outside the purview of Nightwatch (which is a Node end-to-end runner), which is what the StarryNight utility is for.
What do you mean Nightwatch/Tinytest together?
starrynight run-tests --framework tinytest-ci
There’s also experimental support for running both Nightwatch and TinyTest in the same command. If you want to tinker with running them serially vs in parallel, the ‘multi’ framework is for that. Right now it’s disabled, but it’s in active development, so that’s liable to change at any time.
starrynight run-tests --framework multi
Can I write my unit and integration tests within a package using Nightwatch (v0.6) syntax and have them run successfully?
Right now, you can write unit tests using Nightwatch syntax within the package, but not integration tests. This requires the StarryNight --autogenerate and --autogenerated flags. (In time, we’ll link that process together so it’s fully automated.)
For integration tests, you would need to add the --framework tinytest-ci argument, and pick up the TinyTest results from the console browser and report them via Nightwatch.
It looks like you’re choosing to include lots of different syntaxes and frameworks after reading through some source code. Mocha, Jasmine, Cucumber syntax, etc. This makes me a little anxious.
Yes, the intent is a complete integration of the different syntaxes. They each serve a different purpose, and from the discussions I’ve had with my clients, there’s a need to have them all in one place without multiple packages. Now, keep in mind that most libraries only introduce a few novel syntaxes. describe/it syntax in Mocha; Given/Where/Then syntax in Cucumber; should.be.foo syntax in Chai Expect; verify.foo() method chaining in Nightwatch, etc. Each syntax is optimized for a different use: Mocha is great for unit tests, Cucumber for behavior-driven development, Nightwatch for test-driven-development, and so forth.
Beyond the syntactical differences, the remaining functionality in the various frameworks is a mish-mash of overlapping functionality. People reinventing the wheel with regard to runners and reporting. And that’s where the bulk of the overhead comes from.
I’ve been in contact with Andrei, and he’s expressed similar anxiety about including Mocha and Cucumber directly. Nightwatch already has a runner and a reporter, so why add Mocha’s? It just creates redundancy in the code base. He proposed simply adding the describe/it syntax, and the Given/Where/Then syntax directly to Nightwatch. Which would provide the core API syntax that people find useful, but do it via a tight integration, and without the overhead of multiple libraries.
The idea right now seems to be that Mocha and Cucumber syntax will get precompiled down to Nightwatch’s TDD syntax, in a similar way to how CoffeeScript gets compiled to Javascript, Stylus/Less gets compiled to CSS, and Jade gets compiled to HTML.
Now that I’ve exercised the utility for a bit, I have to say that your documentation is rather scattered.
For sure. StarryNight is considerably newer than Velocity. We’re currently making workflow diagrams, more architecture diagrams, and many more tutorials (with screenshots). These past two weeks I’ve been on-site at two clients who are underwriting this development, which has slowed getting documentation out the door. But it’s on it’s way.
In fact commands like “starrynight create --package” don’t even set up working packages…the paths in the packaje.js file are incorrect.
This is an interesting point. As mentioned above, MDG ‘handed the torch’ to Velocity, and has only substantially contributed help via the ‘onDebug’ flag and ‘–test’ flag. For my client’s needs, this is woefully insufficient for the kind of work we want to do. We have dozens of commands and functions related to testing we wanted to see in the meteor tool and on the command line. Not just one or two. The most recent big-ticket item that came directly from a client was package refactoring from an existing component. Basically, they wanted this:
meteor create --package --from /path/to/component
But it’s fairly impossible to submit such functionality via the issue tracker or pull requests. We’ve tried issues, pull-requests, following Velocity architectural recommendations, contacting MDG members, and have gotten nowhere. So, we basically cloned the core structure of the meteor tool, renamed it to starrynight, and began implementing extensions we wished were in the meteor tool ourselves. So, the reason that starrynight create --package doesn’t work, is because that functionality is available via meteor create --package. Could/should that have been documented? Perhaps.
Alternatively, could we add pass-through command functionality so that it works more seamlessly? For sure. We recently implemented that for Nightwatch with argument arrays, so we have the pattern, and it would be relatively simple to implement for the Meteor command as well.
It’s also a bit hard to tell from the site what exactly is and isn’t implemented yet.
That’s fair. As I mentioned, I’ve got sponsoring from three clients right now to develop this technology and am working on it pretty much full-time. It’s very much in active development. And there’s a bunch of documentation in the pipeline. There’s just not enough time in the day to get it done when everybody is asking me to fly around for on-site consultations. 
It’s early yet; but we have a clear vision of where we’re going. Isomorphic API, extending TinyTest with Selenium via Nightwatch, tight integration rather than a package based approach; command line tools.
