New package - jam:easy-schema - An easy way to add schema validation

jam:easy-schema is an easy way to add schema validation. It’s lightweight and fast.

I was looking for an easy way to define a schema in one place, use it for validation on Meteor Methods, and use it to create a Mongo JSON Schema for the db. Rather than cobble together a bunch of packages, some of which are heavy and seem outdated, I built this one.

It also automatically validates against the schema on the server before hitting the db to provide better error messaging since the Mongo errors can be a little lacking.

It could serve as a replacement for SimplSchema and Collection2, depending on how you use those packages.

It’s highly configurable so you can customize to your liking.

If you end up taking it for a spin, let me know! If you have ideas on how to make it better, feel free to post below, send me a DM, or open up a Github discussion.

8 Likes

Well done the package is very impressive :+1:t3:

1 Like

Looking really good! Have you thought about adding typescript support or how would that work?

1 Like

I gotta be honest with you, I really like the schema package a lot more than the methods package. The fact that it generates a JSON schema and the check-like interface makes it a pleasure to work with. Good job :+1:

1 Like

No Typescript support?

I’ve thought about it. I think for the Typescript aficionados I’d rely on zod to create the schema. There’s one missing piece to the puzzle here though which is a nice zodToMongoJsonSchema function. I’m hoping that this package will fill in that gap. Then I would likely release a separate package maybe called easy-schema-zod. Let me know if you have any thoughts here.

Other than that, if something like this proposal is ever introduced, it’d be trivial to add types to the package as is I think. :slight_smile:

Continuing from the jam:method thread as it’s true all my questions are schema related! :see_no_evil:

I don’t have plans to support autovalues in jam:easy-schema at this time. Like you said, I’m sort of against them. Having said that, can you give an example of how you use them and why you find them useful?

We have a base schema that we build other schemas upon which basically handles a few common variables. Here’s what it looks like. There are a handful of other places but as a general rule we try avoid ‘magic’ and things happening outside the main execution path.

import SimpleSchema from "simpl-schema";
SimpleSchema.extendOptions(["autoform"]);
export const BaseSchema = new SimpleSchema({
    createdAt: {
        type: Date,
        optional: true,
        autoValue: function() {
            if (this.isInsert) {
                return new Date();
            } else if (this.isUpsert) {
                return {$setOnInsert: new Date()};
            } else {
                //allow createdAt date to be set by migrations
                //this.unset();
            }
        }
    },
    updatedAt: {
        type: Date,
        optional: true,
        autoValue: function() {
            //both updating and inserting should modifying the updated date.
            if (this.isUpdate || this.isInsert || this.isUpsert) {
                return new Date();
            }
        }
    },
    creatorId: {
        type: String,
        regEx: SimpleSchema.RegEx.Id,
        optional: true,
        autoValue: function() {
            if (this.isInsert) {
                return this.userId;
            } else if (this.isUpsert) {
                return {$setOnInsert: this.userId};
            } else {
                this.unset();
            }
        }
    },
    updaterId: {
        type: String,
        regEx: SimpleSchema.RegEx.Id,
        optional: true,
        autoValue: function() {
            if (this.isInsert || this.isUpdate || this.isUpsert) {
                return this.userId;
            }
        }
    },
    deleted: {
        type: Boolean,
        optional: true
    }
});

I haven’t really. My initial thought: I think these are so tied to the view layer you’re using that I think it’d be better to have someone write a package for their view layer that takes an Easy Schema as an input to magically generate a form.

Yes, but as you’re generating the jsonSchema that might be an easier path as a lot of form libs support that.

In general, I think migrating from SimpleSchema to Easy Schema should hopefully be relatively straightforward. It could also make your schemas a lot more straightforward to grok. Though it will depend on how many features of SimpleSchema you rely on. Easy Schema intentionally doesn’t have all the SimpleSchema features.

It would be cool maybe to have a comparison table in the repo? To jog the memory of long time users who have maybe forgotten what magic they are using from SS. I’m just thinking about the custom validation function now and that could be a problem. Do you have any solution to interdependent fields? So if type = x then status is required or that type of rule?

Hmm, not sure I’m following :slight_smile:. Which code are you referring to?

This code here

Easy Schema does a conversion to Mongo JSON Schema automatically so hopefully your use case is covered. Can you give an example of what you’re looking for?

As it looks like it might be easy to migrate SS → JsonSchema I was just wondering if having the ability to define EasySchemas in JsonSchema might make life easier. But yeah, that loses the brevity of easyschema and is probably quite a bit of work (particularly infering the check side of things from the jsonschema) so probably makes no sense for this package.

Cool, thanks for sharing that example. I could see how that would be handy. This type of functionality kinda feels more in the realm of collection hooks. My understanding is that the collection hooks functionality will be brought into Meteor core in the not too distant future. When it is, I believe jam:easy-schema will be able to validate automatically before those write operations without needing to change anything but I’ll be sure to test it when Meteor releases collection hooks.

jam:easy-schema is in line with your philosophy of trying to avoid too much “magic”.

Interesting. I haven’t spent any time looking into that but maybe I should. Thanks for the suggestion. If you have specific ideas on how this could work, let me know.

Good idea. I might add something like this if there’s enough interest from people currently using SimpleSchema. Generally, jam:easy-schema seeks to be compatible with Mongo JSON Schema and tries to avoid a lot of the “magic” that SimpleSchema has which could be handy but also led to a heavier, slower package (there’s always tradeoffs). Having said that, I’m always open to ideas on ways to improve the package and if there is critical functionality missing, let me know.

Not at the moment. If this is crucial, let me know and I can take a closer look at what it would take to add it. I believe this would be equivalent to using dependencies in Mongo JSON Schema.

Ah ok, I see what you’re saying. Funnily enough, I actually started creating this package using JSON Schema as the starting point. As I dug into it more, I realized that I didn’t really love its verbosity. I liked starting with the brevity of a check-like syntax that I could then convert to Mongo JSON Schema so that I could have the best of both worlds. It also enables this type of scenario:

Let’s say you’re building something new but you’re not sure what shape your data may take

  1. Start prototyping with check to get something in a customer’s hands
  2. Once things have solidified a bit, easily move to jam:easy-schema for better data integrity
1 Like

One of the things I love about zodern:relay is that it provides the return types of subscriptions and methods in client side code. And, as long as I do my part, the types pass through the pipeline too. This is an astonishing improvement in DX.

From what I’ve been reading in this thread, your package looks really great, but not having types is a major deal breaker for me, personally.

Regardless, thank you for creating this. :+1: I’ll be watching!

Thanks for sharing. I’ll likely take a closer look at adding typescript support. I noticed that the check package does support type narrowing so hopefully I can piggyback off of that. If you, or someone else reading this who’s a typescript expert, are interested in lending a hand, I’d welcome it. :slight_smile:

I just added a new feature: where – a custom validation function which includes support for interdependent fields. Here’s a quick example of making status required when there’s text:

{
  // ... //
  text: Optional(String),
  status: {type: Optional(String), where: ({text, status}) => {
    if (text && !status) throw EasySchema.REQUIRED // you can throw a custom message here if you'd like
  }},
  // ... //
}

Thanks to @marklynch for suggesting the feature and providing additional help!

1 Like

A couple of recent updates :tada:

  1. jam:easy-schema is now Meteor 3.0 compatible. It’s featured in @fredmaiaarantes’s Simple Tasks which recently migrated to 3.0.

  2. You can now customize error messages and it comes with even nicer default error messages.

// this will use the default error messages
const schema = {
  email: {type: String, min: 1, regex: /@/}
}

// customizing error messages is as easy as doing this
const schema = {
  email: {type: String, min: [1, 'You must enter an email'], regex: [/@/, 'You must enter a valid email']}
}
6 Likes

A few recent updates to jam:easy-schema :boom:. It now has:

  1. Better typescript and code completion support
  2. An optimized jam:method integration
  3. A changelog

It also has a tiny footprint if you choose to use it client-side. It’s roughly 1/10th the bundle size compared to alternatives. :slight_smile:

6 Likes

I just released a new update to jam:easy-schema :dizzy: with two new features:

  1. Throwing all errors for a better UX. If your user is submitting a form and it has multiple errors, you can display all errors to them rather than forcing them to go one by one through the errors, resubmitting the form each time (ugh). Maybe you’re thinking: the package didn’t already do this? In earlier versions of this package, this feature simply wasn’t possible because Meteor’s check didn’t support it. Thankfully my PR to check to enable it was merged into Meteor 2.16 :tada:
  2. Support for Mongo.ObjectID, which of course includes support for it with Mongo’s JSON Schema. Here’s a quick look on how to use it:
import { ObjectID } from 'meteor/jam:easy-schema';

const schema = {
  _id: ObjectID,
  // ... rest of your schema //
}
6 Likes

I just released a big update to jam:easy-schema :fireworks: with the following features:

  1. Fluent syntax to set conditions (optional). Here’s a quick example:
import { has } from 'meteor/jam:easy-schema';

const schema = {
  text: String[has].min(1).max(140) // string with at least 1 character and a max of 140 characters
}
  1. Set default values

Defaults can be static or dynamic. Static defaults will be set once, when the doc is inserted. Dynamic defaults will be set on each write to a doc. Pass in a function to make a default dynamic.

import { has, ID } from 'meteor/jam:easy-schema';

const schema = {
  creatorId: ID[has].default(Meteor.userId), // static
  updaterId: ID[has].default(() => Meteor.userId()), // dynamic
  username: String[has].default(Meteor.user), // static, value will be Meteor.user.username
  done: Boolean[has].default(false), // static
  createdAt: Date[has].default(Date.now), // static
  updatedAt: Date[has].default(() => Date.now()) // dynamic
}
  1. Easily configure a base schema
import { EasySchema } from 'meteor/jam:easy-schema';

const base = {
  createdAt: Date,
  updatedAt: Date
}

EasySchema.configure({ base }); // all schemas will inherit this base
  1. New ID type for Meteor-generated _ids
import { ID } from 'meteor/jam:easy-schema';

const schema = {
  _id: ID, 
  // ... //
}
5 Likes

I just released a huge update to jam:easy-schema :tada:

Several folks expressed interest in improved type inference and typescript support. Thanks to the heroic efforts of @ceigey, your wish has been granted :rocket:. Take it for a whirl and see what you think. Check the Readme on Type Inference for more details. Huge thanks to ceigey for his effort here.

This release also adds:

  • Accepting additional properties with ...Any
  • A new Double type which will come in handy if you use the Mongo JSON Schema feature

I also updated the Readme to highlight all the benefits this package has to offer. As a quick reminder, here they are:

  • Create one schema, attach it to your collection, and then reuse it across Meteor.methods
  • Isomorphic so that it works with Optimistic UI
  • Automatic validation on the client and server prior to DB write (optional)
  • Automatic creation and attachment of a MongoDB JSON Schema (optional)
  • A schema syntax that is a joy to write and read — brevity and clarity
  • Lightweight and fast
  • Native to the Meteor ecosystem – extends Meteor’s check for validation, uses Validation Error for friendlier error messages, and supports Meteor’s mongo-decimal
  • Integrates seamlessly with jam:method (optional)

Let me know if you have any feedback.

3 Likes

Great work :clap:

Just a question out of sheer curiosity, but given now that you integrate typescript into jam:easy-schema which then can be paired with jam:method. What advantages do you have over GitHub - zodern/meteor-relay: Type safe Meteor methods and publications ? In terms of type inference and cross validation.

1 Like

In terms of type inference

None. I think the advantages are realized in areas beyond type inference. If you enjoy zod + zodern:relay, then I think that’s great. At this time, I can’t say that end-to-end type safety is an explicit goal of mine. Though if others see the benefits of the jam packages and they aren’t meeting their type safety desires, I’d welcome help in improving them to get them to the point they’d like. :slight_smile:

cross validation

I’m not entirely sure what you mean by this but hopefully the examples below will help.

jam:easy-schema mostly originated from needing a schema for a particular project. I didn’t love the existing solutions – overly complicated, too heavy on the client, hard-to-grok syntax etc. I wanted something much more lightweight on the client that also adhered to Meteor’s founding principles, primarily:

  1. Latency Compensation (aka Optimistic UI)
  2. Simplicity Equals Productivity
  3. Embrace the Ecosystem

It replaces the simpl-schema + collection2 combo and adds more features – e.g. automatic Mongo JSON Schema and automatic inferred types – while being simpler (in concepts you need to understand) and all contained in a dramatically smaller footprint.

jam:method seeks to be a worthy successor to ValidatedMethod and adds more features that have been long requested of Meteor.methods while maintaining its core principles.

To be clear, the two packages can be used independently of each other, but combined things get a lot more powerful. Simply call functions. No excessive boilerplating needed.

For example, here’s how I can set things up:

// in a shared file imported into main.js on client and server
// this is optional, but helps to write less code :)

import { EasySchema, has } from 'meteor/jam:easy-schema';

const base = {
  createdAt: Date[has].default(Date.now), // static
  updatedAt: Date[has].default(() => Date.now()) // dynamic
}

EasySchema.configure({ base }); // all schemas will inherit this base
// /imports/api/todos/schema.js

export const schema = { // Create one schema to use across methods. If needed, create one-off schemas for particular methods.
  _id: ID, 
  text: String[has].min(1),
  owner: ID[has].default(Meteor.userId), // if you prefer, you can avoid using .default and set things explicitly within a method of course
  username: String[has].default(Meteor.user),
  done: Boolean[has].default(false),
  isPrivate: Boolean[has].default(false)
}
// /imports/api/todos/methods.js

import { server } from 'meteor/jam:method';

export const create = async ({ text }) => {
  return Todos.insertAsync({ text });
};

export const edit = async ({ _id, text }) => {
  await checkPrivacy(_id);
  return Todos.updateAsync({ _id }, { $set: { text } });
};

export const erase = async ({ _id }) => {
  await checkPrivacy(_id);
  return Todos.removeAsync({ _id });
};

// this one is callable from the client but executes only on the server
export const restore = server(async ({ text }) => {
  return Todos.restoreAsync({ text });
});

// ... other Todos methods
// /imports/api/todos/collection.js

import { schema } from './schema';
import * as methods from './methods';

export const Todos = new Mongo.Collection('todos');
Todos.attachSchema(schema);
Todos.attachMethods(methods); // these can also be attached dynamically to reduce client bundle size

svelte frontend

<script>
  import { Todos } from '/imports/api/todos/collection';
  import { setupForm } from './form.js';

  const { form, method } = setupForm(); // this is kind of like Superforms but much simpler and lightweight
</script>
<form use:method={Todos.create}> 
  <input name='text' type='text' placeholder='add new todo' required />
  {#if $form.errors?.text}<small class='error'>{$form.errors.text}</small>{/if}
</form>

Now let’s say you’re interested in creating an offline-first app. Well you can add jam:offline and either jam:archive or jam:soft-delete and you’re good to go because of the jam:method integration.

I think the integration between them all makes things really powerful with minimal effort.

4 Likes

Thanks a lot for the thorough explanation. It may not suit my needs atm but I definitely appreciate its value. This is great work and I like how everything is working within this philosophy of yours. :clap: :clap: :clap:

1 Like

Nice release @jam and thank you for the shout out! Glad to have helped!

1 Like