Adding Quality to Meteor

announce
1

I recently released a blog post about some ideas that i’ve noticed in other languages and frameworks. I’d love to hear feedback on these suggestions!

13 Likes
2

Really great post. I agree with all of the points 100%, and it’s unfortunate that we dragged our feet on it for so long.

However! These things are currently being addressed as part of the Meteor Guide project. Let’s go through them:

  1. Doc blocks - I agree this is crappy in JS land, but we’re successfully using JSDoc to generate docs.meteor.com, so it’s not totally a lost cause. The guide doesn’t address this. Do you think this is important for app developers? How can we make it easier to use JSDoc?
  2. Generating HTML of docs - see above.
  3. Testing - this is being directly tackled in the guide. I think the words “testing” and “velocity” have been deeply entangled in the Meteor mind space, but I think with package testing, CommonJS modules coming in Meteor 1.3, and a commitment to describing all of the different kinds of tests an app needs in the guide article, we can improve the situation significantly. Testing isn’t rocket science, we just need to make it work. See the outline here and please comment and PR: https://github.com/meteor/guide/blob/master/outlines/testing-outline.md
  4. Generating tests by default. 100% agree - I think we should upgrade meteor create to give you something that follows some semblance of good structure. This will be much easier once we have (3) above, since we can just copy-paste the example code from the guide article. See the issue here: https://github.com/meteor/guide/issues/59
  5. Standard project structure is being actively worked on. Here’s the outline (it’s not quite done yet, we need to rework it for Meteor 1.3) and here’s an example app we’re almost done with: https://github.com/meteor/todos
  6. File generators… I think these can only come after (3, 4, 5) above are done, since we need to write down a standard structure before we can generate it.

I’m excited to see this post because it means we’re working on exactly the right thing with the Meteor Guide. It’s a pain I’ve felt since joining Meteor, and I strongly believe that this project will have a huge positive effect on all of these points.

Yes yes I know I know…. MDG feels these things are better left up to the community.

Not anymore - now we can all build a standard set of answers for all of these questions. However, we can’t figure out all of this by ourselves, since we might miss something. Please take a look at the discussions on the guide and let us know what is missing.

9 Likes
3

Thanks for the thoughtful reply @sashko !

Doc blocks - I agree this is crappy in JS land, but we’re successfully using JSDoc to generate docs.meteor.com, so it’s not totally a lost cause. The guide doesn’t address this. Do you think this is important for app developers? How can we make it easier to use JSDoc?

So if you would have asked me this before my weekend elixir hacking I would have said they’re nice but not a big deal. However, now, seeing their effect on tooling it ends up making developers write more of them so they can take advantage of seeing function docs in the repl, auto doc generation and hosting, and ease of use.

For instance ‘hypothetically’ if Meteor had the same ability we could do h(Meteor.subscribe) in the shell and it would spit out the docs without opening up google. Editors can have a shortcut to make this pop up as well.

Here’s an example of how these doc blocks look on Hex (although this is a more popular package): https://hexdocs.pm/ecto/Ecto.html Pretty cool that they’re auto hosted!

As far as how it can be easier… i’ve always given up after 20 mins of fiddling with getting it to work and build properly. I’m not sure why but I can try again to see what the pain points are.

Testing - this is being directly tackled in the guide. I think the words “testing” and “velocity” have been deeply entangled in the Meteor mind space,

I agree. Thinking about it a bit deeper, I think what I was trying to convey was that when I got started testing in Elixir it was super easy and there was only one test runner that felt a lot like Rspec and it just worked really well. (i’ve tried other langs like Haskell and Lisp too but the testing was just average).

I find it hard to express how Meteor testing is different… I guess it’s just so hard to write integration tests. I agree ES6 modules will help make unit tests trivial and end-to-end will also be easy.

However, i’m being nitpicky! Overall it’s good but just not quite excellent.

Generating tests by default. 100% agree

Awesome! I think it’s just a pure psychological play but it works! 10x when there is a structure/generator!

File generators… I think these can only come after (3, 4, 5) above are done, since we need to write down a standard structure before we can generate it.

Yep definitely. Since Meteor is so easy to get started having generators will help continue this momentum (and learning when first starting).

Sounds like MDG is on it! :beers:

2 Likes
4

[quote=“sashko, post:2, topic:13525”]
Doc blocks - I agree this is crappy in JS land, but we’re successfully using JSDoc to generate docs.meteor.com, so it’s not totally a lost cause. The guide doesn’t address this. Do you think this is important for app developers? How can we make it easier to use JSDoc?[/quote]

Could we just get a sample script or recipe of how it gets run at MDG? Is there a meteor jsdocs command that we can run?

+1 for practicalmeteor:mocha and spacejam making it into the first round of contributions. That satisfies the Verification Testing requirement of 21CFR820.75, since it includes the TinyTest scripts from the existing core packages.

In conjunction with the UC Santa Cruz Dept of Engineering, we’d be happy to submit a PR with StarryNight and Nightwatch contributions to provide the Validation Testing requirement of 21CFR820.75. I’m budgeting time in our next contract for this work, so I’ll be on this later this month and into December.

We have StarryNight commands that extract acceptance/validation tests from Blaze templates. Also, we’ve been experimenting with extending the meteor create command and providing more package structure. We have usecases, examples, and templates to share.

List of packages looks reasonable. Would like to see the files in the packages organized according to the traditional client/server/lib folder structure. The flat file structure doesn’t give much context about what the package does for an app.

We’ve been making a lot of progress on this with StarryNight, since we have a fairly well defined target (21CFR820.75 regulatory approval). StarryNight is moving towards using Meteor.io, and will probably be converting all of our file templates and generators to use it. Again, we have a ton of usecases and resources to share.

4 Likes
5

Could we just get a sample script or recipe of how it gets run at MDG? Is there a meteor jsdocs command that we can run?

Actually, this meteor-jsdoc project has its roots from Meteor’s own generating tool. I now use it in all my projects, and it works really well.

3 Likes
6

So i’ve been thinking about this a bit more and the biggest win is having them for packages. Having them for apps too is nice but not quite as important (though making it as easy as meteor build --docs would be nice!

Having all the atmosphere packages include docs would be amazing.

If the developer could just have the dockblock in their code and the documentation was generated on atmosphere this would be huge.

3 Likes
7

Great article. Testing++++! :smile:

@sashko I really want to learn more about the Velocity about-face. I’m honestly confused about why you guys are trying to “untangle” the words “testing” and “velocity” when it was “the official testing framework”.

I get some of the specific criticisms or technical issues but what I’m confused about why you guys think they’re so large and can’t be addressed.

Anyway, I really think that the testing issue needs more than just guide articles. I’m glad you agree about #4, though it didn’t happen over the last year while Velocity was the official testing framework.

2 Likes
8

Sashko’s will surely chime in with additional thoughts; although in the meantime, I’d hazard to say that the key words from his post are “all of the different kinds of tests an app needs in the guide article”.

To give some backstory on the Velocity project, there were a couple of individuals who were very enthusiastic about testing and wanting to be the leads on the project. Mozel tov. They contributed many fine packages to the community. However, in their excitement, they started scrubbing the project of any contributions that didn’t fit their particular vision of what testing could/should be.

In doing so, they made some architectural decisions that favored unit testing and integration testing. However, when other testing professionals, some with decades of experience, tried to contribute to the architecture so it better supported core TinyTests or acceptance testing, those contributions were soundly rejected.

Specifically, Velocity was architected as a series of packages with realtime file watchers that rerun tests on file change. This is fine for folks who are beginning projects or have a small number of tests to run. But it doesn’t scale well. To keep things under 60 seconds, they had to go down this rabbit hole of mirrors and parallelization.

However, for people who assume that they’re going to have thousands or tens of thousands of tests, and are going to take half an hour or an hour to run on a CI server regardless, the idea of realtime and rerunning tests on file change is naive. So, while the Velocity folks were preparing for 100 meter dashes; there were marathon runners being left out.

In the end, Velocity has become a very good unit/integration testing solution, particularly for beginners, folks with a small number of tests, and folks who want behavior driven development.

But there are many different kinds of testing, as Sashko’s has pointed out. And while that was all happening, the marathon runners (who are never going to win the 100 meter dash of package downloads) have been concentrating on their own infrastructures which focus on supporting and extending the core TinyTest framework, and running extremely detailed acceptance/end-to-end scripts. Specifically, I’m referring to SpaceJam and StarryNight/Nightwatch; both of which have Node architectures, rather than Meteor package architectures.

These architectures run as their own node commands, are tailored to Meteor and CI environments, and while they don’t have the most number of downloads, I think you’ll find that they are running some of the apps with the largest coverage surface area, with thousands or tens of thousands of tests in their test suites.

Anyhow, I’ve tried to write this as respectfully and politely as possible. There have been some hard feelings with regard to the way the Velocity project was handled, and it’s great to see this change of direction, with SpaceJam making its way into the testing guide.

It’s a great start to meeting the requirement of Verification testing, as defined in 21CFR820.75 (the US FDA regulation for biomedical/food related software). If we can get StarryNight/Nightwatch added, we should have the requirement for Validation testing as well.

4 Likes
9

Just wanted to show an example of how easy i’d like Meteor to be for testing (including a nice test runner). Here’s a few second clip of creating a project and running tests:

https://youtu.be/plojuN-UT4I

2 Likes
10

Ok so I spiked a quick solution to get self testing examples in doc blocks going… is this something that people would be interested in? Would tinytest be the best for these? Jasmine?

It would have to be manually ran before the tests were run (perhaps it wouldn’t’ be too hard to make it a build step).

The bonus is that you’d get a free happy path test just by documenting it (and docs of course).

with jsdoc

CalcUtils = {
  /**
   * A trivial calculator that can add *two* digits (yes that's markdown)
   * Example:
   *   > CalcUtils.add(1, 3)
   *   >>> 4
   */
  add(a, b) {
    return a + b;
  }
}

although I wish someone would make a nicer syntax like this:

// @moduledoc A module used to calculate things
CalcUtils = {
  // @doc 
  // Takes two numbers and adds them together
  // Example:
  //   > CalcUtils.add(1, 3)
  //   >>> 4
  // add: Number, Number -> Number
  add(a, b) {
    return a + b;
  }
}
2 Likes
11

Here’s a link to the jsdoc that is generated for meteor-boilerplate

http://site.trajano.net/meteor-boilerplate/

I also added tests, but the only report you can see is in the https://travis-ci.org/trajano/meteor-boilerplate hopefully I can also add coverage in coveralls like