Introducing the StarryNight utility

awatson1978 · April 7, 2015, 8:11pm

Hi all!
In celebration of Meteor Cookbook’s 1000th star (thanks everybody for all of the support!) I’m excited to release a little utility today that’s been in the works for the past year or so: StarryNight.

StarryNight is an application life-cycle management utility for Meteor, somewhat similar to rails generate, spacejam, abee, mrt, and similar utilities. The difference being that a) I wrote it, and b) it’s a bit more focused on things like scaffolding new applications, launching test frameworks, using cookbook recipes, and the like.

You can find the project homepage and npm repository at:
http://starrynight.meteor.com/

StarryNight is the culmination of a year of research on the best way to launch testing frameworks like selenium and nightwatch (which is way more complicated than you might initially expect), and two years of writing the Meteor Cookbook. It also sort of marks my foray into the world of node tool development, and not just doing app development.

It took me a long time to figure out how to refactor the run_nightwatch.sh script into a node utility; and once I did, I had learned so much that it made sense to go ahead and include a bunch of other utility commands that I felt have been missing from the Meteor ecosystem… repository cloning, file permission audits, component generators, etc.

So, long story short, Meteor Cookbook now has a utility tool.

Anyhow, the tool is very new and only has a few basic scaffolds, and the api is under heavy development. But if you want to kick the tires, feel free to do so.

Also, for everybody who’s been using the clinical:nightwatch and velocity:nightwatch-framework packages, those packages are now slated to be deprecated and become abandonware. All future nightwatch development will be with the StarryNight utility. So better to publish it and announce it, since all future nightwatch work of mine will be there.

Enjoy!
Abigail

warehouseman · April 9, 2015, 12:54pm

Does it leverage the scaffolding ooomph of Yeoman?

Rich · April 9, 2015, 1:24pm

I had taken a look at Meteor Cookbook before, but somehow I must’ve missed how great of a resource it is. Thanks for all the work you put into it! Looking forward to playing with StarryNight

awatson1978 · April 9, 2015, 2:50pm

@warehouseman… thanks for sharing about Yeoman! It wasn’t on my radar, but looking through it… yeah, they are very similar tools. The big difference is that StarryNight uses Atmosphere and Meteor instead of npm and grunt for it’s package management and build tools. StarryNight is early enough in it’s development cycle that the generator implementation is still just a basic clone-and-copy-subdirs (the -pattern option), so the yo command could find it’s way into the tool for help with the generators. I’ll do some research to see if it makes sense to include it.

@Rich… thanks for the kind words! Yeah, the Cookbook isn’t as flashy as Discover Meteor or MeteorHacks or some of the other resources out there; but I like to think it provides high-value. There’s a website in the works, but I’m taking my time in building it.

warehouseman · April 9, 2015, 2:55pm

Dear Abigail

I have installed StarryNight and I see nothing about Yeoman. (I had seen comments from others that if any does a scaffolding tool for Meteor it ought to be based on Yeoman, but I don’t really know enough about Yeoman to have my own opinion.)

Following through the steps on the StarryNight tool’s web page I have hit a snag.

yourself@qtst:~/projects/helloworld$ starrynight -survey acceptance
Launching StarryNight.  Analyzing meteor environment...
Detected a meteor instance...
Launching nightwatch bridge...

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: spawn ENOENT
    at errnoException (child_process.js:988:11)
    at Process.ChildProcess._handle.onexit (child_process.js:779:34)

yourself@qtst:~/projects/helloworld$ meteor update
This project is already at Meteor 1.1.0.2, the latest release.
Your packages are at their latest compatible versions.

yourself@qtst:~/projects/helloworld$ cat /etc/issue
Ubuntu 14.04.2 LTS \n \l

yourself@qtst:~/projects/helloworld$ npm --version
1.3.10

Any ideas?

warehouseman · April 9, 2015, 3:26pm

@awatson1978

There’s a whole controversy about the right task runner for Node based projects. Here is a sample from HackerNews : Why we should stop using Grunt and Gulp In the light of that, you might want to get feedback on the right way to go. I am new to Meteor and the Node ecosystem, really – so not qualified to expound my own opinions.

At the same time I am very interested in building myself a minimum effort development environment, so I am playing with Meteoris at the moment.

If I can get my head around StarryNight I might switch,

Sincere regards,
MHB

deanius · April 15, 2015, 3:10am

@awatson1978 - I love the Cookbook, and this is an effort I want to check out. But the language around it seems to be posing StarryNight in opposition to Velocity (single tool vs collection of packages, etc) Is MDG not happy with the direction of Velocity because it doesn’t ‘embrace the existing testing tools used by MDG’ ? Could be clarified, if not… Personally, I’d love if my package test runs (tinytest, or mocha) would be more accessible under CI, but haven’t figured out if I need to PR Velocity for that or try a tool like Starry Night ?

Also, if you’re interested, I’ve worked on bootstrapping new packages: current state of the art is here at http://package-kitchen.meteor.com - I think helping people get started is rad, and I applaud (most any) attempt to ease people into the Meteoric way

awatson1978 · April 15, 2015, 3:36pm

Thanks for the kind words, @deanius. And great to see your package-kitchen utility! Definitely the kind of tool that we need more of.

I can’t speak for MDG, since I’m not part of it. But let’s just say that it was me who wasn’t happy with the direction or architecture of Velocity. So I quit a couple of weeks ago, after the Velocity team voted out some of my contributions for the nth time, and decided to take my efforts elsewhere.

Managed to accomplish more in a week of coding than I had with 6 or 9 months of trying to integrate Nightwatch with the Velocity architecture or the Meteor Tool.

StarryNight is a very different vision of what testing with Meteor can/should be. Much more along the lines of SpaceJam, in that it embraces TinyTest as a core testing framework, and is its own command line tool. (By contrast, Velocity still doesn’t support TinyTest after 9 months). Velocity is… well, I don’t know exactly what it’s trying to be. It’s own Continuous Integration SaaS solution, maybe? StarryNight is much more focused on app-lifecycle management.

And it’s working well enough that I’m planning on putting the full brand of the Meteor Cookbook behind it, and/or converting the Cookbook into the manual for StarryNight. To date, I’ve ran somewhere around 400,000 tests through the Nightwatch framework over the past year, and the StarryNight utility is the culmination of that experience. (By contrast, I was barely ever able to get 5,000 tests ran through Velocity, with its mirrors and file watchers and package based architecture.)

Also, I’m planning on doubling the number of commands next week, and have nearly 2 dozen commands in the works. So, I’ve got a ton of features planned, and am rocking-and-rolling with this new utility. Now that I don’t have Velocity slowing me down, I’m not looking back!

sircharleswatson · April 15, 2015, 9:45pm

@awatson1978 - first of all, great last name

Second of all: I am just starting to look into testing meteor applications. It is the area I lack the most knowledge. I just listened to the Meteor Interviews podcast episode with Sam Hatoum where they talk about testing with Velocity. I am interested in your package and will have to keep an eye on it.

What are the key differences between StarryNight and Velocity? What would you say to someone new to testing about which direction to take?

jimmack1963 · April 16, 2015, 6:50am

What an inspiring quote from a valuable resource:

I can’t imagine that sentence ever occurring again! It feels recursive?

awatson1978 · April 16, 2015, 9:23pm

Hi Charles! I noticed you had a great last name, too!

So, Velocity is sort of a drop-in continuous-integration server which you add to your meteor build pipeline. It’s comprised of packages from within the Atmosphere ecosystem, and has a modular design allowing people to add custom testing solutions. If you have very specific testing requirements and say ‘I must test with such-and-such process or technology’, then Velocity gives you a way to integrate that into your build pipeline (in theory). In practice, I found the architecture heavily biased towards a specific style of unit testing, and to ignore the needs of people doing acceptance testing, load testing, and other forms of testing. Velocity is trying to introduce best-practices from the greater testing world into the Meteor ecosystem. Yet, oddly, sort of ignores TinyTest in the process, since everybody is trying to find a replacement for it. Oh, Velocity has a snazzy reactive UI that one can load into the browser.

StarryNight on the other hand, assumes you’re going to use a third party continuous-integration service, such as Travis, BrowserStack, or SauceLabs. It’s much more pragmatic, and is concerned with documenting and automating scripts for getting your app under QA, so it can go into production. Rather than using packages, StarryNight is built using node, runs from the command line, and exists as a tool that you run in addition to meteor. It’s a bit more suited for folks who don’t care how they get testing started, and would rather use whatever has been shown to work well with Meteor. StarryNight also supports TinyTest, which is used by MDG for testing core meteor packages, and is trying to extend that process with more tools rather than replacing it.

tl;dr - StarryNight is a utility for building apps and getting them under QA asap, rather than an attempt at building a new type of continuous-integration technology.

awatson1978 · April 16, 2015, 9:30pm

Also, many thanks to @hwillson on and @meonkeys for some initial pull-requests, help in cleaning up documentation, and getting the first update ready!

Just published version 0.1.10 to npm, which includes dynamic pathing, and should (in theory) provide support for Linux, Windows, and any other situation where node might get installed to a custom install path.
https://www.npmjs.com/package/starrynight

Updated documentation just got pushed to the project page:
http://starrynight.meteor.com/

I’ve also seeded the Issues and Milestones with a basic roadmap of where the utility is headed in the upcoming weeks:

awatson1978 · April 29, 2015, 7:13am

StarryNight 0.2.0 just released. Updates include:

New Commands

display-env
run-tests
run-framework
extract-ids
extract-classes
extract-tests-for
generate-travis

New Frameworks
tinytest
tinytest-ci

Experimental Frameworks
spacejam
mocha
pioneer
multi

You can now use syntax such as the following to both run tests and for continuous integration servers:

$ starrynight run-framework nightwatch
$ starrynight run-framework tinytest-ci

The experimental frameworks have the various libraries installed; but don’t have the necessary configuration files or sample test files in place to be scaffolded in.

awatson1978 · June 16, 2015, 12:20pm

StarryNight 0.2.61 released this morning (codename: Constellations).

New Features

package refactoring/creation utility
filesystem scanning
isomorphic validation testing API
better tinytest integration
chromedriver
filesystem compact and cleanup utility

New Commands
create --package namespace:foo --from /client/components/foo
generate-autoconfig
run-tests --framework nightwatch --autogenerated
compact

See the snazzy new documentation at starrynight.meteor.com

Edit: Sorry for any inconveniences with that small window where 0.2.61 wasn’t published to npm yet. It’s now published and available for download.

robfallows · June 16, 2015, 3:55pm

It installed on Linux no problem , but I’m still struggling to get it onto Windows. I’ll keep you posted.

awatson1978 · June 17, 2015, 7:24pm

Also, bump to 3.2.61, to reflect previous repositories and development efforts.

Revision History

Versions 0.x.x
awatson1978:selenium-nightwatch
Bash script based launcher. Had problems with installing and attaching to Selenium.

Versions 1.x.x
awatson1978:clinical-nightwatch
Package based architecture, as recommended by Velocity team. Had problems with installing executables after Meteor 0.9 which didn’t provide +x executable permissions.

Versions 2.x.x
velocity:nightwatch-framework
Package based architecture, as recommended by Velocity team, using unit-testing API hooks. Had multiple problems with mirrors, test-proxies, and unit-testing architectural assumptions. Abandoned after developer quit Velocity project.

Versions 3.x.x

Node.js utility based architecture.

robfallows · June 18, 2015, 1:21pm

TL;DR: I got StarryNight installed, but couldn’t get it to run. YMMV.

So, I finally have something to report wrt Windows. I stripped Windows back to a bare installation to find out what a Windows developer might need to install prior to installing StarryNight.

Prerequisites:

nodejs and npm. Don’t do what I did and install the latest version of node. Things will break. I eventually installed our old friend version 0.10.36. I used the msi file from http://nodejs.org/dist/v0.10.36/, which also installs npm 1.4.28.
python. I installed version 2.7 from https://www.python.org/downloads/release/python-2710/. Version 3.4.3 might work - I didn’t try.
You need a C compiler. I installed Microsoft Visual Studio Community 2013 from https://www.visualstudio.com/products/visual-studio-community-vs. Other C compilers might work, I don’t know. The errors I saw prior to installing this referenced Visual Studio, so I took the route of least pain (but probably longest ever installation time). You should check the licence if you are using it commercially.
Check your PATH settings. node and npm seemed to set this up for me. So did Visual Studio, after a reboot, but python didn’t. If you don’t know where to check/set, click the Windows button, right click on Computer, select properties -> advanced system settings -> Environment Variables (Win7).

From here, you can now open up a command prompt and install starrynight:

npm install -g starrynight

During this I got a handful of warnings:

warning C4506: no definition for inline function 'v8::Persistent<v8::Object> v8::Persistent<v8::Object>::New(v8::Handle<v8::Object>)' [C:\users\***\AppData\Roaming\npm\node_modules\starrynight\node_modules\pioneer\node_modules\selenium-webdriver\node_modules\ws\node_modules\bufferutil\build\bufferutil.vcxproj]

warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc [C:\users\***\AppData\Roaming\npm\node_modules\starrynight\node_modules\mrtbulkrelease\node_modules\sync-prompt\build\sync_prompt.vcxproj]

However, the installation completed ok , although on my core i5 laptop it took 3 hours !

Unfortunately, when I tried to add the nightwatch scaffolding:

starrynight scaffold --framework nightwatch

I got the following error:

Starting scaffolding of nightwatch tests...
{ [Error: ENOENT, lstat 'C:\Users\rfallows\AppData\Roaming\npm\lib\node_modules\starrynight\scaffolds\sample-tests\nightwatch']
  errno: 34,
  code: 'ENOENT',
  path: 'C:\\Users\\rfallows\\AppData\\Roaming\\npm\\lib\\node_modules\\starrynight\\scaffolds\\sample-tests\\nightwatch' }

The issue seems to be with no lstat command in Windows. I have checked cygwin, GnuWin and MinGW and cannot find a Windows equivalent. Currently stuck!

awatson1978 · June 18, 2015, 1:48pm

Epic installation! Good reminder of what all we take for granted when working within a single OS environment.

I’ll create an issue to track this, and we can get things sorted out.

chmac · June 22, 2015, 3:34pm

StarryNight looks interesting. A little tricky to find on Google, but I got there.

Is there a getting started, or beginners walk through? I’m struggling to grasp the concept, never mind how I’d actually use it in practice. I’ve read most of this thread and the readme and so on, but I’m not quite getting it yet…

PS> Or an example project using the tool?

awatson1978 · June 22, 2015, 3:49pm

There’s a section in the Readme called Examples that has a fairly extensive tutorial that starts with a helloworld app, adds database and server/client functionality, and then testing scripts. That’s the beginner’s walkthrough. I’m going to add screenshots and break that tutorial out into a separate section next week.