We are using MongoDB as our data layer from the very beginning. There were some experiments with SQL by some community members and the MDG itself.
So,
I hope these are the questions a lot of us is having. I tried to analyze it find some better option.
Read Now: Why Meteor needs a new Data Layer ?
Let’s discuss.
I feel like the most contradictiory fact about Meteor is the fact that it’s reactive by default, but isn’t utilising a database built for it.
I would very much like to see some of your ideas for the data-layer implemented, especially because of Meteors marriage to MongoDB.
It’s not that I dislike mongo, but there’s all kinds of databases, and having one that does everything really well is not really an option if it has to scale.
Am personally very excited about rethinkDB.
RethinkDB runs similar LiveQuery stuff like Meteor - I believe they got the idea after seeing an early demo of Meteor. I’d bet Meteor’s approach is better here because its not implemented on the database. (ie. database does not compete with livequery for resources, and maybe Meteor can unbundle LiveQuery from application). Maybe someone else can confirm this is true or not.
Great post btw, had a hunch this is where they would be going. Hopefully, the Cache layer would continue to similar to minimongo as far as filtering data and reactive updating.
RethinkDB got me all excited until i started researching it. At first, I though it was “realtime first”, but then like you said it feels like they got the idea later and then tacked it on, and then marketed themselves as realtime first, which isn’t true. Their realtime API is very slim after all. Maybe not much is needed, but it certainly doesn’t feel like a first class priority. Someone correct me if I’m wrong. Seems like a marketing angle they jumped on.
Couple that with their functional-first interface + their own query language not based on js–both of which I’m sure will end up being great–and I’m less inspired than I was before i started analyzing it. Couldn’t they just offer these basic things, i.e. a mongo-style interface. I just dont see what they gained by ReQL not being pure js, and not having a simple finder solution like Mongo. Perhaps, I just hate having to learn something new every day of the week lately. …And then tack on what seems to be not all that great performance (at least in its early stages), it’s just like “what do i get from it?”. Supposedly, unlike Mongo, they are super implementation-oriented, and what’s under the hood is just fantastic. We are all looking to RethinkDB to be the new and improved Mongo, but without them really making realtime (and integration with Meteor, the only full stack realtime solution) a priority, I don’t quite believe it. They’ve had a huge marketing opportunity under their belt for so long–Meteor–and they have been letting 1 or 2 issues stay unresolved for months.
…here’s the remaining meteor integration issue for any interested:
They should have been the ones trying to integrate with Meteor, not the other way around. I’m apprehensive how committed these guys are to realtime.
Yes. It can be similar to MiniMongo. But I think it can be simply a lot than mini-mongo and we may not need fancy query features.
I think we should have a GraphQL based subscription system ASAP. It should complement LiveQuery, not replace it. For performance intensive apps/scenarios, you use it; for everyday prototyping continue to use Mongo/LiveQuery.
It requires a lot more boiler plate to setup all the rules and transforms for such a thing. That’s not why people initially gravitate to Meteor. However, Meteor’s gonna have to support a wider spectrum of use cases. In short, it needs to remain appealing to developers just starting out with Meteor and/or who have simpler needs, while also facilitating what experts require to scale their production apps.
If you read Facebook’s usage of their yet to be fully released internal subscription stuff for graph ql, you see exactly why they need it. Basically it allows you to address a bunch of logic without additional state–i.e. on writes within the web-server layer like a lot of people have been talkin about here–which is so important to Facebook given its size.
So being a lower level API will just take more work from application developers to setup, but that makes it a perfect “option” to have. And if it’s robust enough to make a LiveQuery driver with, hopefully it leads to community-supported integrations of other DBs and the like. But I also think there should be a simplified version of it for more basic needs of application developers. So basically a super low level GraphQL Subscriptions-based API and one a bit higher up novice developers can use to easily setup some triggers, kinda like AnyDB.
Yeah! I agree on this.
A HyBrid data layer is the way to go.
The following is a rather specific use case and would probably not be ideal for meteor as a whole, but given the topic at hand, I must throw this into the ring… I have long been considering this setup:
Redis (pubsub + as front cache) backed by MySQL for cold storage and/or mongo—I’m not too big a fan of mongo, but might seriously consider it for collections with schema that are not so rigid.
My imaginary usecase for such a system is a chat platform.
x messages for every chat (expires after a while if no users are active/connected)
y rows, which are cached by redis for a rather short period of timeSet of users subscribed to the room. New messages are immediately sent to users’ node/meteor instances which published to clients via DDP.List which is used as a mysql batching queue and then is (atomically with the previous operation) added to the room’s Set of messages
I haven’t had the chance to play with clustered redis too much, but I know there will definitely be some optimizations made. I’m hypothesizing how to isolate some of the data in an unclustered/not-replicated redis instance when it pertains only to a given user, or perhaps a room.
I’m just hoping that whatever is decided upon as the new Data Layer would be conducive to my (digital)dream. But as many other have suggested, regardless of where this ends up going… please make it somewhat agnostic from the top level! LiveQuery support won’t be a piece of cake to implement with any kind of DB, solution, but I’d like to be able to benefit from the abstraction by creating my own driver if need be.
What are your thoughts about the impact on Meteor of the new MongoDB 3.2 features, like support for joins ($lookup operator) and in-memory adapter, which is for realtime purposes?
Among others new features like partial indexes, BI conector, APM integration, schemas (document validation), data encryption, etc…
Maybe the oplog tailing gets better with in-memory and gets easier to build an ODM with smart joins inside the db with the $lookup operator.
What is the reason behind using redis to store messages which will not change so much?
For user status as typing/online counter etc which does not need to be stored - that is redis usecase.
But for messages I still feel normal DB would be enough.
@arunoda: In your blog post you asked what would be left of Meteor if the data layer were to be replaced. I (and many others) have been wondering the same.
MDG itself has admitted that React is a better UI layer. Consensus appears to be coalescing around Webpack as a better build system and that obsoletes Atmosphere. Your proposal to replace the data layer very likely removes the need for DDP as the RPC layer of the stack, except possibly as an implementation detail.
What is your opinion on this? What is left? Is it more than a curated list of NPM packages, a best practices guide and a hosting service?
Yes. Currently we use a meteor method to get data. It’s not deployed yet, that’s our Kadira UI.
For the server side, you can use how we normally do it in node.
By the way, I think adding GraphQL as a layer on top of Meteor’s Mongo API might be the way to bring this to Meteor now. In the future, backends could be swappable, so it doesn’t matter if the app uses Mongo or SQL, and the UI code won’t have to change as it will simply use the same GraphQL queries.
I don’t have a public example yet. But it simply executing a GraphQL queries inside a Meteor method. So it’d be something like this.
const {
// This function is used execute GraphQL queries
graphql
} = Meteor.npmRequire('graphql');
const Future = Npm.require('fibers/future');
Meteor.methods({
executeGraphQL: function(query, vars) {
const f = new Future();
const Schema = loadGraphQLSchema();
graphql(Schema, query, null, vars)
.then(result => f.return(result))
.catch(err => f.throw(err));
return f.wait();
}
});
You can learn more from our Learn GraphQL project.
Folks, I’ve been an avid MeteorJS learner and have been super excited for it. I’ve been using it for a few months now and by no means am I a MeteorJS master. I see this as an opportunity to share some ideas (which, treat them as abstract / conversation starters, please).
I’ve been a long time user of SilverStripe and have loved their ORM approach with built in class methods. You define the db, run /dev/build, the fields / relations get added to the db (kind of a rake migration task) Once defined there, a basic scaffolding with admin views is already in place, and you have the ability to edit the summary fields (columns in index tables), override / add custom fields to the CMS view with great ease. It’s offered by far the fastest building of user friendly admin interfaces for clients, which just work.
Now, obviously, doing the same in Meteor at once would be a stretch but the building blocks are there. We also need to figure out subscriptions and at the same time perhaps discuss abstraction from the specific database, which I feel is a great drawback. It also feels that architecture-wise locking down approaches and technology while other libraries keep popping up, Meteor needs to stay extremely lean and open to new libraries coming in (e.g. ReactJS, Angular - which I thought, opening their arms to it, MDG did a great job). Reactivity, done in a (seemingly) simple approach, is probably the most exciting quality of Meteor, to me, the rest are just layers.
I’m going to share some thoughts on what an ideal framework would do for me, based on my years of working with SilverStripe:
I imagine defining each object / document as a class, with pretty much everything you would need. It would control subscriptions, if the object needs to be reactive, a custom getter/setter (so e.g. you could fetch some stuff from a custom source)
E.g.
Car = { schema: { // db fields } adminFields: { override/customize admin fields } permissions: { canCreate() canEdit() canView() } subscription: { // is it reactive or not, additional conditions... } serverMethods: { // class methods only seen by server } niceTitle() // a custom getter/setter in the client, with filters if needed yaddayadda somePublicMethod() // eg. to use in the templates }
And then you would do:
myCar = Car.find() (like we’ve been doing it, with all collection methods being readily available)
and in a template, {{ myCar.Title }} etc.
Based on what I’m seeing, Meteor goes steps forward, got awesome buzz initially at rollout, implemented an approach which now seems a little isolated (MongoDB, no hooks for a solid (or multiple solid) CMS’s tightly integrated into the methodoloy) – then some other library comes up, which does a portion of the job of the framework better – and effectively rendering recent development work obsolete (e.g. now that MJS is moving to React, I am wondering what the awesome folks at ReactionCommerce will need to go through to bring it up to speed, instead of focusing on building features).
So I see two things could happen: a) a systemic, holistic, lean re-factoring happens which tightens what it’s been missing and it gets a ton of new followers and with it MeteorJS becomes the de-facto developer friendly reactive framework, for any JS / Node or former PHP afficionado, or Angular / React genius out there or b) stays isolated, in which case, I feel it will just slowly fall into oblivion. I feel the second one is unlikely, but for that to happen a systemic methodology discussion needs to take place – and above all, everything - or, nearly everything, needs to be built as abstract as possible to be replacable easily when better technologies come. I am sensing that somehow MeteorJS is still considered a “toy” - and I really don’t see a reason for it, apart from (my guess?) the fact that it’s missing the openness (out of the box) to integrate with enterprise-level services / databases etc.
I am currently working on a project where I would need offline storage for mobile apps, but GroundDB seems dead and I don’t know the progress of the new Data Layer implementation.
Any news about new Data Layer status ?
Check Apollo project by MDG using graphql. That is the new data layer
Yep right now it’s kind of in a preview mode, but we’re working on adding more features, better documentation, and great Meteor integration. So if you’re excited to experiment, jump in - if you’re looking for something really stable and nicely integrated, now isn’t the time yet :]