We’re heavily invested in validated method. Seems a shame to replicate some similar functionality here. Is there a way to update VM with the typescript goodness using this approach ?
3 Likes
Thanks @copleykj. Your explanation is better than what I would have written.
Using zod for the schema is probably doable with validated method. It might even be possible as a mixin, depending on how the types for validated method are written.
One main difference between zodern:relay and validated methods is zodern:relay allows the types to work across environment boundaries, as @copleykj had mentioned.
To have the argument and return types when calling a method created with validated method, you have to import the method on the client to use the call function created for it. This both makes the method be a stub on the client, and includes the file’s content in the client. For some methods that might be fine, but for others it is not.
zodern:relay is the opposite - the method is only defined on the server, and is never in the client bundle. It is designed to make it impossible to accidentally include method or publication code on the client. You can only call createMethod and createPublication in methods or publications directories and have to use the babel plugin. Otherwise, your client will crash with an error explaining what happened and how to fix it.
Even though the code calling methods or subscribing to publications is on the client, and the definitions for the methods and publications are only on the server, the types still work. That is one of the main reasons I created this package, and the idea I borrowed from @blitz/rpc.
I have been considering ways to support method stubs. It needs to be clear when and exactly what code will be kept on the client. Two of my ideas are:
- Have a separate file extension to signify when the methods inside should also be used as method stubs on the client. For example the methods in
projects.stub.tswould be defined on the server, and on the client as method stubs, while methods inprojects.tswould only be defined on the server. - The second idea is when creating a method, you can set a
stuboption to either a function to use as the stub on the client, or totrueto use therunfunction as the stub on the client. The function used as the stub must be self contained - it can not use any variables or functions defined in the root of the file, and can not use any imports from files inmethodsorserverdirectories.The only code kept on the client would be the stub function, and any imports it uses.
3 Likes
I had a bit more of a look through everything. This zod library is pretty cool.
Yeah, we prefer this way of working. We achieved this with VM by separating the method declaration and the run implementation. So our method can be used on the client but it only does type validation and permission checking and will only import and run the run code on the server. I can see why people might want stubs for some methods but would definitely argue that no-stub should be the default.
I guess the more fundamental difference between VM & relay then is the underlying schema library used, and zod is essentially what enables the cool functionality. We’re so tied to simple-schema that I don’t see any easy way to start using this.
A while ago I have seen someone (@radekmie was that you?) demonstrate how to get TS typings out of a data schema (I think that even was simpl-schema, wich is used in msg:validated-method). No Idea how that worked though and if it is viable.
This is great! Would be nice to take it a step further into the direction of this PR: Advanced meteor methods by Floriferous · Pull Request #11039 · meteor/meteor · GitHub
For example: adding hooks in methods is incredibly useful to create scalable codebases.
2 Likes
Yes, it was me and here it is: TypeScript 4.3 and SimpleSchema. It’s still possible and actually quite easy to do, even without a package. But I agree that packages like zod or typebox are better suited for that.
3 Likes
I’ve been considering various ways to handle the use cases of hooks and mixins while having type safety, and I’ve started working on pipelines: Pipelines by zodern · Pull Request #1 · zodern/meteor-relay · GitHub
Pipelines allow you to use an array of functions for a method or publication instead of a single run function. The output of each function is used as the input for the next function. The pipelines are composable - you can create functions or partial pipelines and use them for many methods and publications, and you can configure global pipelines that run before every method and publication.
For publications, some steps of the pipeline can become reactive and re-run when the results of queries change, allowing reactive joins and other uses.
The docs are complete, though some of the code hasn’t been written. If anyone wants to learn more, the docs are here. I am interested in hearing any opinions or suggestions on this. There are a number of benefits to pipelines, but there are also some parts that aren’t as nice.
6 Likes
I released version 1.1.0 today which adds the remaining features.
The name for methods and publications is now optional. It seemed unnecessary to name both the export, and the method/publication itself. Now the babel plugin will add a name if there isn’t one.
Method stubs are now supported. You define them in the same place as the method, and during build time the method stub and any imports used by the method stub will be extracted and used on the client.
import { Tasks, Events } from '/collections';
export const createTask = createMethod({
schema: z.string(),
stub: true,
run(name) {
Tasks.insert({ name, owner: Meteor.userId() });
}
});
In this example, since stub is set to true the run function will be used as the method stub. Only the body of the run function and the Tasks import will be used on the client. All other code in the file will only exist on the server. Alternatively, stub can be a function if you don’t want to re-use the run function as the stub.
This version also adds pipelines, which allows you to break methods and publications into multiple steps. When used for publications, parts of the pipeline can be reactive, allowing you to change what cursors are published as other data in the database changes. Also, you can configure a global pipeline that runs before or after every method or publication.
8 Likes
Just started to implement this and migrate my validated methods over and I came here to say this is absolutely brilliant! It’s a joy to use!
@zodern
Do you have Github Sponsors or Patreon setup? If not, I can just subscribe to MontiAPM, but thought to ask.
1 Like
Jan, given that we just discussed taking simpl-schema package into the community (if someone takes it over and maintains it), is this a good idea with @zodern package now out and being maintained by him?
Secondly, what is the overall view of using meteor/check vs Zodern:relay?
I’m using meteor-check with some project specific extensions (meaning checks that are only important for my project).
I haven’t checked the Meteor documentation lately if that’s still somewhere being recommended or replaced by another package.
1 Like
Hi, I’m trying to use it inside a meteor package. However, I am getting the following error. I have a .babelrc file in my Meteor main directory and the required npm package is installed. I would be glad if you help.
Error: Method name is missing. Do you have the relay babel plugin configured?
at createMethod (packages/zodern:relay/server-main.ts:60:11)
at module (packages/bordo:ticket-system/lib/methods/tickets/create.ts:13:22)
at fileEvaluate (packages/modules-runtime.js:336:7)
at Module.require (packages/modules-runtime.js:238:14)
at require (packages/modules-runtime.js:258:21)
at /Users/recepozen/bordo/whatsgoo-main/.meteor/local/build/programs/server/packages/bordo_ticket-system.js:369:1
at /Users/recepozen/bordo/whatsgoo-main/.meteor/local/build/programs/server/packages/bordo_ticket-system.js:377:3
at /Users/recepozen/bordo/whatsgoo-main/.meteor/local/build/programs/server/boot.js:369:38
at Array.forEach (<anonymous>)
at /Users/recepozen/bordo/whatsgoo-main/.meteor/local/build/programs/server/boot.js:210:21
at /Users/recepozen/bordo/whatsgoo-main/.meteor/local/build/programs/server/boot.js:423:7
at Function.run (/Users/recepozen/bordo/whatsgoo-main/.meteor/local/build/programs/server/profile.js:256:14)
at /Users/recepozen/bordo/whatsgoo-main/.meteor/local/build/programs/server/boot.js:422:13
I haven’t tried using it in Meteor packages. Is the package inside the app’s packages folder, or are you using METEOR_PACKAGE_DIRS? If it is the second, you might need to also configure the babel plugin in the package’s folder.
Thanks. It’s nice to hear it is being used. I do have GitHub Sponsors setup: Sponsor @zodern on GitHub Sponsors · GitHub
3 Likes
I found the issue: Error: Method name is missing. Do you have the relay babel plugin configured? · Issue #5 · zodern/meteor-relay · GitHub
@zodern Hello, how should I do error management with the zodern-relay package. Is there a way you recommend? Errors that occur with zod appear on the client as “Interval Server Error”.
There is a discussion about zod errors at Communicating validation errors to the client · Issue #4 · zodern/meteor-relay · GitHub.
Been working with this package for the last few weeks. holy moly, it is amazing. I was able to wire Zod up to Simple Schema, so now I can define my schema in one place and have validation on database writes, methods, publications, and for client-side form validation with react-hook-forms and their built in zod-resolver.
You’ve really breathed new life into Meteor. Thank you again for this AMAZING package.
4 Likes
I ended up subscribing to MontiAPM.
1 Like
Would you mind detailing your method for integrating Zod into Simple Schema?
Sure. This is still VERY much a work in progress…
// zod schema object
const zodSchema = {
_id: z
.string()
.regex(/\b^[23456789ABCDEFGHJKLMNPQRSTWXYZabcdefghijkmnopqrstuvwxyz]{17}\b/, { message: 'Not a valid Id' })
.describe('A Meteor generated Mongo identifier'),
name: z
.string()
.describe('Users Name'),
favoriteColor: z
.string()
.describe('users fav color')
};
// exported for use around the app
export const schemaObj = z.object(zodSchema);
export type ZodTypes = z.infer<typeof schemaObj>;
export const FavColors = new Mongo.Collection<ZodTypes>(favColors);
// generate schema
// collection name passed for error reporting
const simpleSchema = createSimpleSchemaFromZodSchema(zodSchema, 'favColors');
// @ts-expect-error expected error - missing types in Collection2
FavColors.attachSchema(simpleSchema);
Below is the big ugly function to create the schema that can be attached to collection via Collection2. Simple Schema is a massive tool, so this doesn’t work for everything. For example, it would take a lot of work to get this to work with autovalue in Simple Schema.
import SimpleSchema from 'simpl-schema';
import { ValidatorContext } from '/node_modules/simpl-schema/dist/esm/types';
import { ZodType, ZodTypeDef } from 'zod';
import { fromZodError, ValidationError } from 'zod-validation-error';
import _ from 'lodash';
import { Table } from 'console-table-printer';
import colors from '@colors/colors';
export const createSimpleSchemaFromZodSchema = (zodSchema: { [key: string]: ZodType }, collectionName: string) => {
return new SimpleSchema(
{ ...zodToSimpleSchema.call(this as unknown as ValidatorContext, zodSchema, collectionName) },
{
// All validation should happen in the simple schema custom function based on zod rules
// So all fields in simple schema are optional, and the optionality is checked in the custom function
requiredByDefault: false,
clean: {
autoConvert: false,
trimStrings: true,
filter: false
},
});
};
function zodToSimpleSchema(this: ValidatorContext, zodSchema: { [key: string]: ZodType }, collectionName: string) {
const simpleSchemaObject = {};
for (const [key, value] of Object.entries(zodSchema)) {
const simpleSchemaValidationObject: Partial<ValidatorContext> = {
type: SimpleSchema.Any,
custom() {
// The _id (Mongo identifier) should be required across the app,
// but not on database operations. It is added automatically on
// insert operations. For modifier operations like $set, check
// to see if it is the _id field and if we have a modifier object.
// Insert operations will return null
if (this.key === '_id' && this.value === undefined) return;
// Parse value against zod and save the result
const parseResult = value.safeParse(this.value);
// no errors
if (parseResult.success === true) return;
// Below this point the argument is failing validation checks
const details: ValidationError['details'] = fromZodError(parseResult.error).details;
generateAndLogErrorTable.call(this, details, value, collectionName);
return {
name: this.key,
value: this.value,
type: 'validation-error',
message: details[0].message
};
}
};
Object.assign(simpleSchemaObject, { [key]: simpleSchemaValidationObject });
}
return simpleSchemaObject;
}
function generateAndLogErrorTable(this: Partial<ValidatorContext>, details: unknown, value: ZodType<any, ZodTypeDef, any>, collectionName: string) {
Meteor.defer(() => {
// deferred to ensure it logs to console after any errors in database operation try catch blocks
// nasty code to recursively look through zod
// error objects and format to the console
const errorsToLog: any[] = [];
function printAllErrors(obj: any) {
for (const element of obj) {
if (element.expected || element.received) {
errorsToLog.push(element);
} else if (element.issues) {
printAllErrors(element.issues);
} else if (typeof element.code === 'string') {
errorsToLog.push(element);
for (const prop in element) {
if (_.isArray(element[prop]) && !_.isEmpty(element[prop])) {
printAllErrors(element[prop]);
}
}
}
}
}
printAllErrors(details);
const p = new Table({
style: {
headerTop: { left: '╔', mid: '╦', right: '╗', other: '═' },
headerBottom: { left: '╟', mid: '╬', right: '╢', other: '═' },
tableBottom: { left: '╚', mid: '╩', right: '╝', other: '═' },
vertical: '║',
},
enabledColumns: ['message','code', 'expected', 'received'],
columns: [
{ name: 'code', alignment: 'left', title: colors.white.bold('Code') },
{ name: 'message', alignment: 'left', title: colors.white.bold('Message') },
{ name: 'expected', alignment: 'center', title: colors.white.bold('Expected') },
{ name: 'received', alignment: 'center', title: colors.white.bold('Received') },
]
});
p.addRows(errorsToLog);
// print
console.log(colors.red.strikethrough.bold(' '));
console.log(colors.red.bold('Error Writing to the Database')); // outputs red underlined text
console.log(colors.yellow.bold('Collection: ') + colors.yellow(collectionName));
console.log(colors.yellow.bold('Value Provided: ') + colors.yellow(this.value));
console.log(colors.yellow.bold('Key/Field: ') + colors.yellow(this.key || 'none'));
console.log(colors.yellow.bold('Key/Field Description: ') + colors.yellow(value.description || 'none'));
console.log(colors.yellow.bold('Modifier : ') + colors.yellow(this.operator || 'none'));
p.printTable();
console.log(colors.red.strikethrough.bold(' '));
});
}
Interesting I’ll have to examine this closer when I have a chance. I was recently thinking about taking the opposite approach and trying to map Simple Schema definition to a Zod schema.
1 Like