How do you test a packages-for-everything app?

testing
1

Continuing the discussion from Is packages-for-everything the solution to all Meteor problems?:

@spicemix: Thanks for starting an intelligent discussion around the app-in-packages methodology. I’m interested to hear opinions around testing in this paradigm.

I’m at my 3rd Meteor based startup, been working with the platform for more than 2 years, built a dozen or so apps in that time. In my most recent project, I finally adopted the all code in packages paradigm. My primary motivation was better testing support.

We’re using the mike:mocha-package package, and we’re running our tests on CircleCI like meteor test-packages --driver-package respondly:test-reporter. We have packages like app-logging, app-init, app-jobs and so on. There’s code in our packages which executes when run. As in, we don’t simply declare a function and expose it, the package runs code. It often hooks into Meteor.startup(), but also executes directly.

The challenge we’re now facing, and the philosophical point I’m grappling with, is how we run our tests, without running our code. I’ve tried searching on the topic of testing the app as packages model, but I can’t find any in depth discussion around it.

I had a couple of ideas:

  • Defer all code to run via Meteor.startup(), and then stub Meteor.startup() when running tests.
  • Only expose functionality from our packages, and move the calling of that functionality into the app itself. So package foo exposes Foo() then the app calls Foo.run() to “start” or “execute” the package.

Do either of these ideas make sense? Is there a better solution?

This challenge really only appears with background jobs. We don’t want the background jobs to run every time we run our tests. I can’t figure out how to determine if Meteor is running tests or running normally, but even if I could, conditionally disabling code based on that seems very ugly.

In Go, this is a non issue. With Go, you define a main() function, which gets called when your compiled binary is run. When you load the package for testing, main() is never called, so the application doesn’t start. Simple. You also get access to package scoped variables in your tests (not true in Meteor). I’m wrestling with the question of how to achieve something similar in Meteor.

Thanks in advance for any suggestions or feedback.

2 Likes
2

Perhaps you could have a jobs runner package which role is to schedule the running of the “jobs” or functions.
Put those functions in (an)other(s) package(s), and then test those functions with yours tests.
If you want to test the jobs runner, you test the jobs runner package and then stubs the jobs.
What do you think ?

3

If I understood you correctly, I think that’s what we are doing. We have an app-jobs package which loads @vsivsi’s excellent Job Collection package, and calls .startJobs(). If package foo depends on app-jobs, in my foo tests I can stub app-jobs and run my tests. That works great.

The challenge is, when I want to test the app-jobs package, in the package.js file’s onTest() block, I say api.use('app-jobs');. When that gets executed, the job server gets started, and so my background jobs start running.

Maybe the key lies in stubbing every package from every other package. Currently, in my foo package, I have both api.use('app-jobs'); and api.use('foo');. Otherwise, the code inside foo throws exceptions. It would be more leg work, but I suppose instead of using api.use('foo'); I could instead do something like Foo = sinon.spy();, so that all calls to the foo package hit a stub instead of throwing an exception.

4

I’m not familiar with the Job Collection package, but if you put your jobs runner in package A, job 1 in package 1 and job 2 in package 2, when you are testing the A package, you stub package 1 and package 2, and then when your runner is executing, it runs the stubs and not your actual jobs.
I didn’t understand the part about foo, what package is it representing ?

5

StarryNight provides a testing framework that doesn’t rely on the package system itself to install, and therefore doesn’t need to have Meteor started and running to launch. That lets it grab code in isolation when doing server-side and package unittests. The directory scanning has been working great, and we’re able to keep both acceptance tests and unit tests within our packages.

3 Likes
6

@vjau Maybe the Job Collection thing is complicating the issue. Imagine we have a package which sends an email on startup. It’s called EmailStartup Something like (coffeescript):

Meteor.startup ->
  Email.send email

When I try to test this package, in package.js I put api.use('EmailSend'); and then the package gets loaded, and my email is sent. The email has already been sent before my test code runs. So in my test code, I could write sinon.stub Email, 'send', but it’s too late. The email has gone already.

I’m grappling with how to stop that email being sent.

7

I think the function definitions and the client code (instantiationg or calling) should be put in separate packages.
You have your package “email sender” in package A and the code that triggers it in package B.
So when you test package A, yours tests will trigger the sending once it has been appropriately stubbed.

8

So in my EmailSend package I expose a method like EmailSend.run() and then I create a second package like EmailSendRunner which calls EmailSend.run()? Is that what you mean?

Even if I do that, you can’t stub EmailSend.run() before loading the EmailSendRunner package via api.use('EmailSendRunner') inside the onTest() block, as far as I know. That’s the crux of my challenge I believe…

9

Let’s say you have a FooMail package with a sendEmail method (i like it better than run, because the concerns are more separated).
Then you have a TaskScheduler package with a method to schedule task like that :

TaskScheduler.add(func, periodicity);

The important point is that TaskScheduler has NO dependency on FooMail. TaskScheduler has not to know about FooMail since its only duty is to schedule Tasks.

Now, in a third package, let’s call it Main or App, you require (api.use) FooMail and TaskScheduler and write something like :

TaskScheduler.add(FooMail.sendEmail, 10);

(edit : actually you have to use bind, but that’s not the interesting point)

No if you want to unit test TaskScheduler, it’s easy to pass to it dummy “tasks” or what you want.

Most of the time, easier testability leads to more decoupled design, which is a good side effect :slight_smile:

1 Like
10

@vjau Thanks for persevering. :smile:

It’s getting a bit hard to discuss all this in English, so I’ve resorted to code! I’ve written up an example repo here. It’s 2 packages. fake-email which just calls console.log(), and email-on-startup which does 2 things. First, it calls Meteor.startup() with a callback to call FakeEmail.send(). Secondly, it calls FakeEmail.send() immediately as soon as it’s loaded.

Running meteor test-packages email-on-startup shows that the first call to FakeEmail.send() can be successfully avoided in the test. To do that, it’s necessary to have the api.use('email-send'); in the onTest() block of package.js.

However, the second call, which is in email-on-startup.coffee line 10 cannot be avoided. This code is executed before the test code is run, and so at that point, FakeEmail.send() has not yet been stubbed.

This exercise has taught me that the test code is run before Meteor.startup() callbacks are run. That’s very useful to know. That would mean, so long as we’re careful, we could manually stub every dependency package in our test files, and then the real packages should never get called.

However, that’s a very fragile, inelegant solution. If we miss any methods, then the real methods will be run. It also means we have to run everything via Meteor.startup() which may not be ideal in all situations.

Is there a better alternative that I’m missing?

11

Some related reading :
http://misko.hevery.com/2008/07/08/how-to-think-about-the-new-operator/

I refactored you example separating :

  • the email sending facility (unit testable)
  • the startup executing facility (unit testable)
  • the app instantiating facility (not unit testable, like explained in the link).
12

One way I’ve successfully stubbed packages with other packages, is by getting the stubbing package to depend on the original package. In so doing, Meteor loads the package to be stubbed first.

In your example, I would set your EmailFake to use Email. This is what I do for this xolvio:inbox-stub, which is an app-level email interceptor.

Hopefully that helps you

13

@vjau Great, thanks for the code. Now I know exactly what you mean. :smile: The app instantiating facility is not unit testable in your example because of limitations within Meteor, right? There’s nothing inherently “untestable” about that functionality as I understand it.

What I’m wondering, is what to do about this. Do I simply not test this part of the code? Or do I create some alternate construct or pattern that allows me to test this? If I do, I’m introducing more fragility into my code, because I will probably end up with test related code inside my production app.

@sam: Nice, I like this approach. Maybe the ultimate solution is to implement something ugly like if process.env.TESTING is true then stub the package, and always run my tests like TESTING=true meteor test-packages....

This could get pretty complex though. I’d end up with something like my real package foo-core which is a dependency of an intermediary foo package. In that intermediary package, I conditionally stub or load the actual foo-core package. Again, I’m including test related code in my production application, which increases fragility.

I think that this challenge fundamentally comes down to the fact that in Meteor, it is not possible to load a package under test without actually running the code in that package. There is no inbuilt solution for this. So I can either build something myself, or the code in my packages will always be run when my tests are run.

I suppose after this discussion my question comes down to whether or not I can detect the difference between meteor and meteor test-packages. I started a new thread to ask that question here:

14

Not quite :smile:

You can set the debugOnly flag to true, and this means any package you build this way will not be built in production. This is how we do it with Velocity and the MDG helped us by implementing that flag. I’d give that a try first

15

Hey @chmac you could use something like Space.Application to start your code exactly when you want to (basically the same as having a main method).

The scenario you described is actually the very reason i started building the Space framework, I needed more control over the configuration and bootstrapping process. The Space framework gives you two phases you can control and hook into your app: 1. configuration and 2. startup. You can even do things like overriding complete subsystems with stubs during testing simply by hooking into the configuration step of your application.

If you want to test your sub-packages in various scenarios you can also create stub-apps for your integration tests that load and configure the modules in a certain way before running.

2 Likes
16

@sam Aha, that’s a nice approach. It doesn’t solve the specific tests running vs meteor running in development issue, but it’s a nice way to make sure stubs never go to production.

@CodeAdventure Interesting approach, thanks for the link. Not sure if I want to go so far as to use Space, but it sounds like an interesting development. I’d love to see something like this in meteor core. Some sort of main() function would be a real help for being able to write more testable code.

17

Hey @chmac, I guess you misunderstood what I was trying to say: Meteor has a main function! It is called Meteor.startup and it is your responsibility to structure your code in a way that there is only one place that calls it. All the tools are there, now be creative and build a great structure around them :wink: I guess if you never try a more solid way to structure your code, you will always end up with coupled spaghetti and typical hard-to-test Meteor code that is floating around a lot.

1 Like
18

@CodeAdventure OK, fair point, we could always hook into Meteor.startup(). I suppose if we always exposed the function that was called by Meteor.startup() then we could also stub and test it.

When we run meteor test-packages, the Meteor.startup() hooks are always called, so it’s not quite the same as the main() in Go, but we could leverage it to get pretty close.

The dependency tree gets pretty complex though. If package C depends on B which depends on A, then to test C in isolation, I need to stub all the calls to Meteor.startup() in both B and A. That could get really messy, and it’d be nearly impossible to test that we had actually stubbed all of those startup calls.

Anyway, you’re right, there are options available via Meteor.startup(), thanks for the clarification.

19

@chmac there’s one catch: never use Meteor.startup in packages unless you make them completely configurable beforehand. I just had to ask @arunoda to add a hacky FlowRouter.wait API to allow me to start routing when my app is ready, not when the routing package thinks everything is ready. So I would even go so far to say that you actually should only use a single Meteor.startup call for the whole Meteor application if possible.

If you have package dependencies then your code should reflect that explicitly and not indirectly via hard-to-control Meteor.startup callbacks and hard-coupled globals. Either you make each package configurable, similar to iron-router or you will always end up with these problems.

Honestly, at this point in the discussion you should try the Space framework I showed you earlier :wink: it was specifically made to solve all of these dependency-tree / injection problems and to make your code unbelievably easy to structure and test. You can define modules and have other modules / the main app require them in a really, really simple way. It also does not matter if you use a package-only or “normal” Meteor app structure, it just works.

I also have to say that while it might looks daunting at first to use something like dependency injection, it really makes your code much more readable and the dependency graph explicit, which will save you many hours of pain in the long run.

If you want to read up the well established design patterns behind solve these problems:

  1. Inversion of Control
  2. Dependency Injection

These patterns are pretty much standard nowadays and not even Meteor can save us from writing our software with solid architectural principles in mind. Meteor provides a great build platform and many amazing packages to deal with realtime-sync and display of MongoDB data. Everything else is still up to you as application architect (and that’s a lot!). I like to see Meteor as an implementation detail that should not leak into every line of code of my application.

2 Likes
20

@CodeAdventure Thanks for the follow up. I’m reluctant to add a package such as Space as a dependency into my codebase. Then I’m subject to any bugs / issues in that package, and my code simply won’t work without it. It becomes as much of a dependency as Iron Router, maybe even moreso. I’d want to see wide community support, broad usage, and so on, before doing that.

Ideally, I’d like to see this sort of control within Meteor’s core. It looks (from initial glance) like the Space framework is a great solution, but a great solution that’s implemented alongside or on top of meteor, rather than within it. I’ll take a closer look at Space on the next Meteor project. :smile: