Decouple velocity CI execution from meteor application

testing Velocity Core
1

I know velocity is tightly integrated with meteor to provide the awesome development experience which we all like, but I think specifically for CI there should be a way to run the “hub” part of it outside of the hosting application.

A quick recap on how velocity currently runs CI with the cucumber plugin to the best of my knowledge:

  1. user starts the process by calling VELOCITY_CI=1 meteor --test
  2. the application under test is started at port 3000 and acts as the hub
  3. a mirror process (or multiple ones if specified) is started from the hub
  4. the mirror starts its own chimp process
  5. the mirror connects to the hub (DDP connection)
  6. the mirror starts running tests (either all at once or the ones the hub tells it to do in case of parallel execution)

Actually the only difference between the CI mode and the development mode is that in CI mode all tests are run and that the hub and all its mirrors are shut down, once the tests are finished.

I can understand that the design of the architecture was driven by the wish to create an awesome development experience, but I find it kind of weird that when running tests in CI mode my application is running under the default port and does basically nothing else then orchestrating tests on its mirrors.

I stubbled across this because I obviously need to allocate dynamic ports when running tests for more than one application at the same time on the same machine, for example with a Jenkins setup or even locally.

This post is meant to start a discussion about the architecture of velocity between the community and its creators/maintainers.

2

This architecture issue of a decoupled CI hub has been brought up repeatedly by myself and others.

We even spent the time and effort to contribute packages that could support a decoupled CI hub via Nightwatch. There were folks on the Velocity team who became defensive and territorial, however, and rigged a voting process so they could get those contributions removed.

So, this architecture issue you bring up, about a decoupled CI hub … its exactly the underlying cause of Nightwatch getting removed from Velocity, spinning off, and the StarryNight project getting started.

Just a quick update on the state of decoupled CI hubs and testrunners for Meteor:

  • We got funding at the beginning of the summer to continue StarryNight development and write a bunch of tutorials for Nightwatch, which have been going online at starrynight-beta.meteor.com.

  • We’re finishing up the first batch of tutorials and examples this week, and will be moving the beta site into production next week.

  • We’re using a .meteor/nightwatch.json configuration file, which can be set up to use dynamic ports with a little shell scripting. It controls the hub, and can be configured with as many environments as you want.

  • We also had a community contributor on the Nightwatchjs mailing list provide a Nightwatch/Cucumber integration, which will be going into Starrynight 3.4.0.

  • I’ve been coordinating with the maintainer of Nightwatch, and he’s bringing in official mocha integration in version 0.8.

  • I’ve been finishing up a Nightwatch/TinyTest integration strategy (which has been heavily inspired by SpaceJam/mUnit), which will involve a fork, cleanup, and rebranding of munit. We’re currently considering which namespace it should go in.

  • We’ve also been identifying how to streamline the APIs across Nightwatch and TinyTest so they become isomorphic.

  • We’ve also been identifying how to clean up the APIs so they’re appropriate for institutional and corporate users (human resource departments frown on spying, mocking, innuendos, etc)

  • We’re preparing to upgrade the core TinyTest packages this fall by way of the clinical:METEOR track.

  • And we’ll be rolling many/most of the StarryNight commands into the meteor tool by way of the clinical:METEOR release.

1 Like
3

Hey @sgroebler

Thank you for taking the time to post this continuation of our discussion in the Velocity chat rooms. The more eyes we can get on it the better the discussion.

###The reason for the hub
As you identified, the hub is the central controller and all test frameworks are started from here and report back here, it manages all the collections and provides publications for consumption like the html/console reporters.

This functionality requires a Meteor app as it has 2 main dependencies on Meteor. They are:

  1. The communication channel are Meteor.methods.

If we wanted to do the above in an NPM package we’d need a DDP server to make 1 work. And we’d need to start a mongo database to make part 2 work. Meteor does these two things pretty well!

###Can the hub be reused?
Absolutely and this is possible today on a per-framework basis. A framework does not have to request a mirror and can easily run against the hub. I plan to allow xolvio:cucumber to be runnable against the main hub with an option. It’s a little easier for E2E style frameworks to do this since they run against the app. Integration frameworks run within the app so that would be a little more involved.

Even with this option, I don’t think it addresses your specific problem. What you’re asking for is to get a dynamic port instead of 3000. The hub currently uses freeport to start mirrors on random ports. If you run against the hub directly, then the port selection is still down to you. See next point!

###Can you allocate dynamic ports for the hub?
Sure! Here’s a script I use on Letterpress that runs Velocity on CircleCI. It is a node script that starts Meteor with some arguments. All you would need to do is add freeport as an npm dependency, wire it up and add a line that says:

args.push('--port');
args.push(theFreePort);

There, now you have the main hub starting on a random port without doing an NPM. Perhaps I could generalize this script a little and release it as reusable NPM module. In fact, I think I will!

###CI Steps Clarification
As an aside, let me clarify a couple of things. The steps you describing above are specific to the xolvio:cucumber, I can tell from step 4 as Chimp is only used by Cucumber. The reason this is worth noting is because on CI cucumber only needs to be run with VELOCITY_CI=1 meteor --test.

If you use Jasmine / Mocha then you have to use this command on CI meteor --test --release velocity:METEOR@1.1.0.2_3.

The latter tracked version of Meteor does a couple of interesting things:

  1. It tells Meteor to also watch / build files under /tests/<framework> and
  2. it outputs the files to a different location to .meteor/local.

Step 1 stops you having to do crazy symlinking to get tests included and step 2 allows you to run the main app and mirror using the same source directory without overwriting each others output files.

###Open to Suggestions / Improvements
I’m interested to hear opinions and feedback on how we can improve Velocity and related frameworks. The most valuable type of feedback is real-world use cases

1 Like
4

Hi sgroeber,

yes, currently the Velocity Core Hub runs inside the app instance. In the last two week sprint I’ve written down a design proposal to separate the Velocity Hub into it’s own app. Since then haven’t gotten to implement it because there were always more important problems to solve. If you want to drive this change, this would be great. I’m available for questions in Slack. Just write me directly or create a new channel and invite me.

Here is my design proposal from the last two week sprint (4 months ago):

Velocity CLI tool

  • Starts a Velocity Hub app that is the “controller” for running tests
    • It receives the test results
    • It includes the test reporter
  • The Velocity Hub app is a Meteor app.
  • The Velocity CLI tool starts the Velocity Hub app via child_process.fork (bypassing the meteor Command line tool)
  • The Velocity CLI tool and the Velocity Hub app can communicate via the node.js communication channel.

EDIT: The Velocity CLI tool will be part of the Meteor command line tool. So not separate as originally written.

5

Hey @awatson1978,
sounds like I hit a nerv there. Thank you for bringing up StarryNight and Nightwatch, I will definitely check it out and evaluate it as a potential future alternative. Since I am kind of a cucumber fan-boy, I would probably have to wait until that part is implemented.

6

@sam, just to clarify. my post was not about having a problem (technically or conceptually) with allocating ports dynamically. I just wanted to discuss why this is necessary in the first place and yes the steps I explained are specifically about the cucumber integration (see second sentence).

thank you though for the more detailed overview of how velocity works and how it is dependent on meteor.

after briefly scanning the StarryNight page, I must say that it is probably a good idea to provide a system architecture diagram/overview for velocity, which I guess would make contributing a bit easier.

7

hey @Sanjo,

just to clarify you and @sam are official maintainers of velocity and in parts of their integrations, right? I am a bit confused, because both your answers seem uncoordinated and also I was technically writing with you on gitter but actually talking to sam (extremely confusing).

your suggestion sounds like the way to go. is their consensus about that in the velocity team?

I would love to be involved in those changes, but I am not so confident that I can manage to allocate enough time.

Anyhow, in order to discuss this in more detail how should I get in touch with you? Did you mean gitter when you said “slack”, because afaik. slack team signup is not open for public but either invitation based or email domain restricted.

8

just to clarify you and @sam are official maintainers of velocity and in parts of their integrations, right?

Yes

I am a bit confused, because both your answers seem uncoordinated

I haven’t read Sams post before I wrote mine. I think we had a different understanding of your raised concern.
As Sam has written it, the Velocity Hub must be a Meteor app. That doesn’t mean it cannot be a separate Meteor app that is just the Velocity Hub :smile:

I was technically writing with you on gitter but actually talking to sam (extremely confusing).

In Gitter Sam answered you. Gitter has this confusing UX bug where it is not clear who answers you from Slack. We have connected a Slack channel with Gitter.

your suggestion sounds like the way to go. is their consensus about that in the velocity team?

We have discussed this a bit 4 months ago and Sam, Mike and I generally agreed to this idea. In general we try stuff out and if it doesn’t work we try something else :smiley: We still should lay out the use cases where this design has benefits for the user and where maybe our current design gives a better UX. We can also have both options for the user if that makes sense.

Anyhow, in order to discuss this in more detail how should I get in touch with you? Did you mean gitter when you said “slack”, because afaik. slack team signup is not open for public but either invitation based or email domain restricted.

I meant our Velocity Slack. Just send me your email and I can invite you. Our Velocity Slack is for communication between people who are a bit more involved in the Velocity development.