There has been lots of discussion pertaining to Collection2 and its future. I’m starting this thread to avoid derailing others.
There seems to be multiple directions to go for Collection2
Integration into Meteor core. Seems the biggest point to discuss IMO. Given recent discussion brought up by @storyteller@nachocodoner there’s a direction to merge critical community packages into the core and given that something like meteor-collection-hooks is being thought of to be merged into the core, why not Collection2? It’s a point worth of discussing yet I’m honestly fearful for so many reasons:
In a way this means Meteor doubles down on a single way to manage collections in Meteor
Bloats the core with additional package
Slows down the core team due to additional cost of maintenance
Same as Astronomy but Mongoose instead. Why reinvent the wheel and waste tim? Mongoose is the biggest name in Nodejs realm when it comes to managing collections. And since @paulishca already made a proof of concept this makes it all the more easier to achieve.
Maintain the status quo and evolve independently. Probably the least chaotic decision. I bet lots of current uses would side with this one. We work on improving and maintaining Collection2 in line with community/Meteor needs providing the path of least resistance for current users and maintaining the best developer experience.
I’m thinking the biggest next step for Collection2 is to rid it of the shackles of Simple Schema especially given the latest changes. In case you’re not in the loop. The NPM version of Simple Schema isn’t compatible with Meteor so @jkuester had to hard fork the package yet now this posses challenges for async changes.
I’m deeply inspired by @jam recent work on jam:method. You’d be able to couple Collection2 with any validation library of your choice and I believe this’d be even more awesome if Autoform starts supporting bridges finally completing the trifecta Simple Schema, Collection2 & AutoForm.
I’m looking forward to hearing your opinion on this guys, cheers!
I think this should not be considered good for Meteor. There is a Mongoose package for Meteor now. Anyway with or without a package, Mongoose adoption is massive. If I could come to Meteor and use things that I already know, that would be a massive advantage.
I wonder, if something can be added with meteor add package … why this focus to put everything into the core?! One guy, one day may come and say … hey guys here is the Collection Pink or 3 etc … and then we all want that into the core. I love Meteor because I can meteor add ... and I love it even more since i can npm install ...
We make extensive use of Collection2 and it has been of tremendous help, but yes, it’s a relic of a bygone era. It was meant to make up for lack of schema enforcement in MongoDB.
MongoDB has had schema validation for years, now. Personally, I would suggest to do the minimum possible for a Collection2 migration to 3.0 and leave it at that.
All valid points. I think we should investigate how we can hook into MongoDB schema validation. simple-schema has very similar structure like the JSON Schema that is being used in MongoDB. So for me, if possible, I would like to see collection2 move towards MognoDB schema and provide a nice communication layer between that and Meteor. Which I think would be a prototype on how it could then be implemented in the core (after all it is MongoDB functionality).
The approach for me here being to reduce the number of MCP maintained packages so that we can focus more on those that are more unique looking at the whole ecosystem or whose functionality can’t be slowly moved to their main technology like MongoDB. Alternative would be having more resources to put towards maintenance and development of these packages so that they can be much more (aka Astronomy path).
Also, something that @alimgafar mentioned was that it might be better to run the schema validation on the app server before hitting DB (as that costs data transfer and then processing power on the database). Which is what collection2 is doing today.
And at that point I’d like to point again to a solution using mixins so those who want to use Mongo’s built-in shema can use it or others who want to use something else on the server can use this.
It would be great if you can still keep SimplSchema as an option. The main advantage of it is that one can have a schema attached to a collection, then reuse it in one or more forms (reactively), and even use pick or omit to reuse parts of it for validation in specific methods. It is unbeatable in that respect.
@jkuester have made SimpleSchema Meteor 3.0 “compatible” but async computations are not supported. That might be an intermediate step for Collection2 i.e. Meteor 3.0 compatible without async computations.
Then, the remaining features will wait for the full-async version of SimpleSchema.
P.S.
I am not familiar with other features of Collection2 outside the simple automated schema validation (which must become async to support async computations).
“Bring your own validation” looks like the least requirement for maintenance with the widest compatibility to existing validation tools. At the very least, a before hook with error handling will suffice.
This has not yet been brought up for discussion internally.
I will say my thoughts currently on every point.
Integration into Meteor core.
Does not sound good because of the arguments brought up and also because the whole 3.0 migration focus on bringing Meteor back to the full javascript ecosystem. A package being in the core does not mean that you cannot use other package that solves your problem but for me gives the signal that the package that is in the core, is the default way to solve this problem. In this case there are plenty of options I think, in our ecosystem and in the broader JS ecossytem.
This one I like the most, when I first started with meteor I was looking for something like Mongoose but with a focus on meteor(today I prefer just using the driver but I can see those who like using an ODM).
This one is nice. It supports change streams but I would like to see how would feel to use the real-time features for example, with astronomy works how we would in Meteor normally. Does that occur with the Mongoose package?
Maintain the status quo and evolve independently.
Maybe this is the path where is more natural? Removing the usage of simple schema sounds like the next step in the package history. Bring your own validation sounds like the today default way of having a schema-like library, many js package does this, you bring/pass the validator and the package just uses the function to validate.
Yes, we first used Collection2 + Simple Schema but never really liked it. Astronomy was a very good alternative for it, especially if you like the whole model concept. We are currently using our own model layer with Joi validation because Astronomy isn’t maintained anymore.
I would prefer Mongoose as replacement because it is widely used within the Javascript community, and if you have some micro services or other apps which share the same database like your Meteor app, it would be very nice if you could use your Mongoose model there too.
I never used Astronomy, but as for Collection2 (or universe:collection; it’s the same idea) and Mongoose – the performance impact is huge. I removed them (or recommended to) from a lot of projects (Collection2 from Meteor ones and Mongoose from non-Meteor ones). The biggest cost is the copying after fetch (transform in Meteor and its counterpart in Mongoose).
With that in mind, I’d say I’m not against those in Core (I don’t mind), but they should be opt-in, or at least opt-out.
I wanted to give this thread some time so everybody can join, so thanks a lot to everybody for sharing their opinion.
Core integration
I guess given the fact this wasn’t raised internally by the core team as @grubba mentioned and how this gets in the way of other options, then it’s safe to rule out, at least for now.
Mongoose Integration
This’d have to be ruled out too, given how this’s already implemented thanks to @paulishca, no need to repeat the efforts.
I believe Manuel’s comment sums up the case for Collection2 pretty nicely:
it’s a relic of a bygone era. It was meant to make up for lack of schema enforcement in MongoDB.
So that leaves us with other two which I’m yet undecided about, but I can assure you all who use Collection2 that the project is still maintained and compatible with Meteor 3.0.
TBH, I’ve never used Astronomy and the issue of resource allocation that @storyteller mentioned is still a thing. We’ll see.
EDIT: What’re the Meteor specific features that Astronomy offers than no other ODM library offers? @diavrank95
Please if anyone knows more about this point, share your knowledge with us.
If those features are so good to pass on then we might consider maintaining/migrating it to 3.0
the performance impact is huge . I removed them (or recommended to) from a lot of projects (Collection2 from Meteor ones and Mongoose from non-Meteor ones). The biggest cost is the copying after fetch (transform in Meteor and its counterpart in Mongoose)
Performance is a key metric for sure but not all projects are under that much of a heavy load that they’d have to squeeze out every single ounce of CPU so for some it’s ok to use an ODM.
though I guess you agree with us on not doubling down on a single option especially when it comes to the core.
With that in mind, I’d say I’m not against those in Core (I don’t mind), but they should be opt-in, or at least opt-out.
I like the idea of adding a simple before hook to Meteor core that anyone can leverage. Maybe it should be called validate to differentiate it from other hooks. This keeps things simple and adds the most flexibility. I think the trick would be to ensure that it would run before any other collection-hooks.
I would definitely take advantage of that hook for jam:easy-schema which like Collection2 validates automatically before DB writes.
Agreed, there is definite benefit here not to mention the ability to customize errors much more easily. Mongo’s JSON Schema errors are pretty lackluster imo.
Agree. An ODM is not my cup of tea but to each their own and I think it’s great if there’s a go to package for those that like it.
————
Not to take the convo away from Collection2 but I wanted to highlight a couple things for people who are evaluating which direction to go when it comes to validation:
jam:easy-schema generates a MongoDB JSON Schema automatically based on a schema with a check-like syntax that you attach to your Collection.
jam:easy-schema also has this ability. In fact, if you use it in conjunction with jam:method the pick-ing is done for you automagically.
If you haven’t checked it out yet, it’s worth a look to see if it’s a good fit for you. If you already evaluated it and decided it’s not for you, let me know. I may be able to incorporate what you need.
I think the main thing that jam:easy-schema doesn’t have at this point that was mentioned above is an Autoform integration. I don’t think it would be difficult though. I’m using Svelte as my view layer though and I have a different approach that I’ll be sharing with the community soon.
I may have been unnecessarily harsh . It is a relic, if one strictly needs just for MongoDB schema validation. But in tandem with SimplSchema, it gives you a lot of centralised control over validation. You can reuse the same schema, or parts of it, for: client forms, validating method arguments, validating the full document at the point of insertion/update.
I don’t think you have a better combo covering all these at the same time.
And I think it should be like that for as long as SimplSchema is relevant for forms.
Now you made me curious . How would it work with autoform? If not, does it work with any forms package? I looked at it when you announced it, but haven’t seen any mentions in the repo.
EDIT: I don’t know how I managed to overlook the “Autoform integration” part of your post.
After a little bit of reading Astronomy docs The best thing Astronomy has is the holistic approach to managing collections related code. To achieve a similar experience using Collection2, you’d have to pair it with Simple Schema, collection-helpers, collection-hooks, collection-instances and at least a package for methods validations like validated methods or jam:method.
There has been a previous attempt by Quave to circumvent this issue by offering one package that leverages these packages underneath to offer a more streamlined DX.
Given what I had just mentioned, I think the next logical step would be to bring all these libraries together:
into one single mono-repo/single package where by installing Collection2, you get the benefits of all three libraries. We won’t bundle simple-schema as we’ll be allowing you to bring your own validation whether it be easy-schema, zod or anything else. The nice thing about this is backwards comparability too!
I believe this is the natural next step as it offers better DX not just for you, the end user of Collection2, but also us the maintainers (@storyteller, @jkuester and me). During the 3.0 migrations @jkuester suffered a lot when he was trying to update collection-instance due to its cross-reliance on other packages. Also, this serves the point @storyteller mentioned about lessening the amount of packages we’re responsible for. And, finally we’d end up with something similar to Astronomy yet fitting for 3.0 that utilizes our collective efforts we’ve put so far.
. It is a relic, if one strictly needs just for MongoDB schema validation. But in tandem with SimplSchema, it gives you a lot of centralised control over validation. You can reuse the same schema, or parts of it, for: client forms, validating method arguments, validating the full document at the point of insertion/update.