New package: mdg:validated-method - Meteor methods with better scoping, argument checking, and good defaults

I think the guide mentions flux kind of in secret. I noticed that it suggest getting data from a topmost template and pass it down as properties and contexts in contrary to direct data retrieval access from any template. Furthermore, it mentions in a few places that reactivedict and even local collections can be used to manage a state object store.

I think these are steps towards a flux architecture while keeping the terminology strictly within blaze, tracker and minimongo to avoid confusion, if that’s the intention.

2 Likes

@serkandurusoy, I think you’re on to something…

Based on Geoff Schmidt’s talk the other night, putting it mildly, it seems Meteor might one day soon closely resemble the Facebook (FB) stack (with the addition of DDP)

It doesn’t make sense to adopt React as the official view layer, but not all the underlying tech that FB builds around it, the tech that synergies and complements it.

@aadams my initial reaction to the react announcement was negative on the premise that enterprise software development does not take it too well when things don’t get multiple years of solid support and shifting gears too quickly might not always be a good thing.

Then there were talks about keeping Blaze as the officially endorsed layer while using react as the “rendering engine” for an upcoming blaze upgrade. I was sceptical about that as well.

But to be honest, seing that the guide is preparing us first with a migration of the mental model, showing that the current toolset can in fact be used in a way that resembles the best practices and patterns from a new technology, without changing anything other than how we (can better) architect out software - and that is a good thing - actually made me believe that MDG will deliver on the promise of a seamless transition.

Now at this point, if MDG can better communicate these developments and their vision in terms of actionable and tangible items that we can see and touch today, then I would not mind changing underlying technology (with the ability to do it at some point of my own chosing). I think that does actually deliver on the 7 principles and the original vision.

I also want to commend @sashko for his ability to (often) keep his cool under such heat and the efforts he is putting in transparency and leadership. One thing to add though, that perhaps the gems (in terms of communication) hidden in disparate places like multiple github issues and documents could be better promoted here on the forum as well, at least with some key takeaways. Afterall, github is for the progress while the forum is for communicating that.

3 Likes

This probably isn’t worth changing but maybe rename “_execute” to “test” or “callWithContext”? Does the underscore imply that it could change in the future? Also execute is pretty generic, isn’t obvious what it does.

1 Like

Just a question: Is this stable?

Hmm, I wouldn’t expect any super major changes, but I’d wait until Meteor 1.3 and the first version of the guide land to really rely on it. On the bright side, it’s so few lines of code that it wouldn’t be a big deal to fork it to your own version if you need to.

1 Like

Cool! So this would be part of 1.3 as well… npm loading and this? Anyway, I would look forward to using it. I guess for now I’ll stick with Meteor.methods

I’d say we consider this method package is a subset of the Meteor guide, which will be a really big part of the 1.3 release. We need to get some more feedback from production users about it before everything is finalized.

Hmmm… well I was hoping I could use it on my production project but it might have breaking changes if it switched to Meteor 1.3

Great to see some thought on how to use Meteor for large-scale applications!

Please forgive me for dumping some slightly unstructured thoughts here :smile:

CQRS

First let me suggest a perspective: looking at Command Query Responsibility Separation as an inspiration for API design for Meteor. I’m always kind of surprised that the term CQRS hardly ever shows up in discussions of architecture in Meteor, because Meteor is a pretty awesome modern CQRS framework. Let me briefly describe the important parallels:

A ‘traditional’ non-CQRS has the server going through ‘domain logic’ code for all its requests. An app with an Active Record style for example, has you using Post objects loaded with query logic in your update / create / remove stuff and it has you loading those same objects with update / create / remove logic even when you’re just displaying them in a list. CQRS tells us to separate our app logic into a command part and a query part. And guess what: Meteor methods are command handlers, and Meteor publications are a thin read layer. BOOM, you’ve got CQRS! But like a rocket-powered version, because the publications use livequery and get pushed from the server to the client for auto-updating goodness, oh my! (I often feel that an understanding of CQRS is why I’m often a lot less confused about Meteor’s architecture than a many of my colleagues) Also, MongoDB is an excellent database for denormalised data storage that goes very well with this architecture.

Now given that Meteor apps are basically CQRS apps, this means we can use lessons learned in the CQRS world to improve Meteor! Some examples:

Middleware

Given that Meteor Methods correspond to the POST / PUT / DELETE part of an application, it would be nice to use the middleware pattern to for purposes like logging, access control, maybe early validation.

Domain Driven Design + Optimistic UI

While CQRS is a natural fit with Domain Driven Design, the proposed validated-method makes you separate validation and behaviour, rather than coupling them like you would in a DDD app. I’m still searching for a pattern to couple this and still use Meteor’s client simulation.

Example: I have a form. Upon submission I would like to run domain code to perform domain behaviour and throw errors if necessary. Obviously I want all this to run on the server for security reasons, but for a quick response I also want to run it on the client. Now, on the client I want to catch potential errors from the simulation and the server, and use them to create user feedback on the form view. But oddly, when I listen for errors, Meteor decides to wait for the server response rather than trusting the simulation? From this point on I could just as well have left my domain code in /server instead of in /lib

The Meteor guide considers the optimal Meteor method to be one that waits for client validation until it goes to the server. However, for the applications I build, the correctness of my code (hence DDD) is more important than a bit of server resources.

In the guide, I see the usage of two options returnStubValue and throwStubExceptions, but I don’t see them in the Meteor documentation. They sound like they would allow me to make methods behave like I had expected their default behaviour to be. Are these 1.3 features or undocumented 1.2 features?

Persistence

I’ve learned that when working with CQRS, NoSQL and Meteor: it can really pay off to start thinking about persistence very differently from how I used to think about it when I still used web frameworks that prefer SQL. Especially when you start getting comfortable with data duplication, a lot of possibilities open up. A couple things I learned:

  • Database schemas should not handle all of your app’s validation, mainly because of transactions (looking at you, Collection2)
  • Denormalizing data is ok!
  • You don’t need an ORM
  • For some apps Event Sourcing is the way to go
  • Even though we use MongoDB: Make the schema explicit

I know that data duplication / denormalization feels really awkward for people coming from a SQL background; Which today is lots and lots of people. This is why you get the ‘Where is the ORM in Meteor?’ type questions. I found that as long as you make your schema explicit on the boundary between your domain and your storage, you’ll be fine. This boundary could be made with the Domain Event pattern or with the Repository pattern, for example.

A simple example of the repository:

var PostsCollection = new Mongo.Collection('posts');

PostsRepository = {
  findOne(id) {
    return PostsCollection.findOne(id);
  },

  insert({title, author, content}) {
    return PostsCollection.insert({title, author, content});
  }
}

This allows you to encapsulate the details of object persistence, and so I know exactly where in the code the database schema is made explicit.

Things that bug me about persistence in Meteor:

First, I can’t create the same collection in two places in my codebase, because Meteor will automatically try to create the /collection/insert etc. methods, and considers this a conflict.

Second, publications only take cursors. This means I often need to write two difference ‘findOne’ methods on my repository:

// lib
PostsRepository = {
  publishOne(id) {
    return PostsCollection.findOne(id);
  },

  findOne(id) {
    return this.publishOne(id).fetch()[0];
  },
}

// server
Meteor.publish('post', PostsRepository.publishOne);

// client
Template.blog.helpers({
  post(id) { return PostsRepository.findOne(id); }
});

Not sure if it’s possible to fix this, obviously the client needs to know which collection to add the document to…

Final thoughts

So ummm, concluding: I love Meteor’s low barrier to entry, and it’s brilliant pub/sub implementation. I’d love to see that in becoming more mature, Meteor will continue to play to it’s strengths, as it did before with using Cordova and MongoDB with isomorphic JavaScript. Thanks for all the great work on such a fantastic platform!

2 Likes

Yes, we could add a feature to ValidatedMethod to allow you to avoid passing those options. They are undocumented options that have existed for a while, presumably we should also document them.

This is not true - if the simulation throws an error, it immediately calls your client-side callback. That’s actually the main goal.

So do you want client-side errors to matter, or not? What should happen when the method simulation throws an error, in your view?

We do something similar, nice to see an official package.

foo: T.makeMethod('something.foo', function(bazz) {
  foo.validate.message(bazz);

  foo.authorUserId = Meteor.userId();

  return Bazz.mutate.bar(bazz, {fizz: 'something'});
}),

This has significantly less boilerplate but it seems ValidatedMethod does more. We wanted a simple wrapper.

This just defines the method and calls it. Doesn’t that already handle the stubs for client?

Then when we want direct server access we just call the mutator directly. The above would only be called for untrusted front-doors.

ValidatedMethod has more code because the above code seems to take advantage of foo.validate and Bazz.mutate.bar, which would both be part of the validated method definition when using this package.

1 Like

Yeah, I see the enforced structure and keeping it packaged all in one area. That’s a lot of boilerplate to assemble that but maybe method enforcement structure is a good idea. I’m not sure. So far doing what we have feels ok and if we need flexibility we have it too, but that can go either which way.

This is the source btw:

T.makeMethod = function(name, fn) {
  //install the Meteor.method
  Meteor.methods({[name]: fn});
  //return the Meteor.call wrapper
  return function() {
    var args = _.toArray(arguments);
    args.unshift(name);
    return Meteor.call.apply(Meteor, args);
  };
};

We became accustomed to defining our mutators and validators in their own objects.

We’re going to review ValidatedMethod and see the pros and cons of making the switch. As we’ve said before, we like to stick to the Meteor way ;).

1 Like

Can you tell me more about what your validators and mutators look like, and why you think it’s good to put them in a separate place? For example, I think it’s very valuable to be able to say myMethod.validate(inputs), instead of thinking “which validator am I supposed to call here?”

1 Like

Because validation isn’t always bound to a function body. It can be bound to schema integrity and other checks (security?).

We don’t worry so much about which validator as we have a pretty set standard of what to use 80% of the time since it’s a shared validation on integrity.

We have two tiers of validation 1 on the front-door (within this makeMethod call) and then 1 on the mutator for strict schema integrity.

Examples:

Foo.validate = {
  message(message) {
     if (!Meteor.userId()) {
       throw new Meteor.Error('not-authorized');
     }
  }
};

Foo.mutate = {
  post(message) {
    check(context, Match.Optional(Object));

    const newMessage = _.extend(_.pick(message, 'type', 'userId'), {
      createdAt: new Date(),
    });

    check(newMessage, Foo.Schemas[message.type]);

    const messageId = Foo.collection.insert(newMessage);

    if (Meteor.isServer) {
      Hooks.doBackgroundJob();
    }
    return messageId;
  }
};
1 Like

It appears as though you have two tiers of validation in ValidateMethod. You have one defined as validate and the other tier is embedded in the run method. It would be nice to have the second tier of validation as a seperate method. It might look something like this:

// Attach your method to a namespace
Lists.methods.makePrivate = new ValidatedMethod({
  // The name of the method, sent over the wire. Same as the key provided
  // when calling Meteor.methods
  name: 'Lists.methods.makePrivate',

  // Validation function for the arguments. Only keyword arguments are accepted,
  // so the arguments are an object rather than an array. The SimpleSchema validator
  // throws a ValidationError from the mdg:validation-error package if the args don't
  // match the schema
  schemaValidate: new SimpleSchema({
    listId: { type: String }
  }).validator(),

  validate(...) {
     // `this` is the same method invocation object you normally get inside
    // Meteor.methods
    if (!this.userId) {
      // Throw errors with a specific error code
      throw new Meteor.Error('Lists.methods.makePrivate.notLoggedIn',
        'Must be logged in to make private lists.');
    }
    const list = Lists.findOne(listId);

    if (list.isLastPublicList()) {
      throw new Meteor.Error('Lists.methods.makePrivate.lastPublicList',
        'Cannot make the last public list private.');
    }
  },
  mutate({ listId }) {
    Lists.update(listId, {
      $set: { userId: this.userId }
    });

    Lists.userIdDenormalizer.set(listId, this.userId);
  }
});

This allows me, if I want to, to bypass validation altogether if I am calling .mutate from an already protected area of code.

2 Likes

The other thing is you may want to call one mutator from several different function bodies so decoupling it can be nice.

Although I guess you could extract that mutator out anyway and just reference it within any of this new boilerplate code generation.

2 Likes

Continuing the discussion from New package: mdg:validated-method - Meteor methods with better scoping, argument checking, and good defaults:

Thank you for clarifying!

To be specific: I was referring to the Method’s asyncCallback. The documentation says:

This sounds to me like the callback is not run until the client receives a response from the server, but maybe I’m misreading the meaning of ‘when the method is complete’.

For one of my apps, for which I host a dev version on a free Heroku account, it would explain why a certain part of the UI is often very slow to respond to form submission. Specifically, I clear the form fields when I don’t get an error in the method call’s asyncCallback:

Meteor.call('foo', 1, 2, function (error, result) {
  if (typeof error === "undefined") {
    successMessage(result);
  } else {
    validationErrors(error);
  }
} );

This clearing often takes a few seconds while the new entry gets added to the list view instantly; which would be a result of the simulation. I’m assuming: free account = slow server.

I would expect the asyncCallback to run based on the simulation, whether the simulation returns an error or not. Or should I write that as follows?

try {
  const result = Meteor.call('foo', 1, 2);
  successMessage(result);
} catch (e) {
  validationErrors(e);
}

Meteor claims to have optimistic and correcting UI, so this is also how I would expect method results behave. If the server’s method result turns out to disagree with the simulation, it should somehow get ‘corrected’. I’m not sure if this is technically possible though; there’s no ‘livequery’ for methods. Maybe it could be done with Redux? Ha! :smile:

Even if ‘correcting’ is not possible, I would still expect it to be ‘optimistic’ by default.

I can see this go wrong with developers who are not aware that this requires them to make all the data needed for validation available on the client. For example: a CMS’ optimistic UI would answer ‘is this slug taken?’ based on the subset of slugs it has available. Developers would need to publish and subscribe to have all the system’s slugs in the client’s MiniMongo if they need a correct answer (even though outdated info could still cause trouble).

My Heroku-hosted app which I mentioned above seems to have this weird in-between behaviour where the method callback waits for the server, but the database writes are simulated. (once again, maybe I’m just using it wrong) It should be either one or the other: waiting if no simulation is available and optimistic if it is. Developers should learn to figure out per scenario which one fits and place their domain logic in /lib or /server accordingly.

I hope you excuse me for being so lengthy. :innocent:

Yes, this is one of the things that ValidatedMethod changes - it calls the callback with the error immediately if the simulation throws an error.

The pattern you have where you do form validation by calling the method is exactly how ValidatedMethod is intended to be used, and should be super fast. But it’s not fast with “normal” Meteor methods.

This is one good reason to use ValidatedMethod over a “normal” Meteor method.